The full answer
A compacted topic (cleanup.policy=compact) keeps at least the latest value for each key instead of deleting messages by age. Older values of a key are removed in the background, so the topic stays about as large as the number of keys, and a new consumer that reads it from the start rebuilds the current state: every customer's latest address, every product's latest price. Use it for changelogs of keyed, changing data, such as a database table captured with change data capture or an application's state. To delete a key, write a message with that key and a null value, a tombstone; compaction removes the earlier values, and the tombstone itself is removed after delete.retention.ms (1 day by default).
1. Delete versus compact
delete (the default) | compact | |
|---|---|---|
| What is kept | Every message until it is older than the retention time or the log is over its size | The latest value of each key, for as long as the key exists |
| Good for | Events that stand alone: clicks, logs, payments | State that changes: customer records, settings, a table's changes |
| A new consumer from the start sees | The last 7 days, say | The current value of every key |
Both can be set (delete,compact): compacted, and also dropped after the retention time.
2. What a consumer sees
A consumer that keeps up with the head of the log sees every message. A consumer that starts from the beginning later may miss intermediate values of a key, but always ends with its latest one. Offsets do not change during compaction; some are simply gone.
3. Deleting a key
producer.send("customers", key=b"c-42", value=None) # tombstone
Consumers must handle a null value as "this key was deleted". The tombstone stays for delete.retention.ms, which is also the time a consumer reading from offset 0 has to finish its scan and still see the delete. A consumer that takes longer could miss it and keep a deleted customer.
4. Things to watch
- Every message needs a key: compaction works per key.
- Compaction is not instant: recent messages stay uncompacted for a while, so a topic can hold several values of a key at any moment.
- It is a history of the latest state, not an audit trail. To keep every change, use a normal topic with long retention, or land the changes in a table with history, such as an SCD type 2 table.
What interviewers want: the latest value per key, the changelog and state-rebuild use case, tombstones with a null value and delete.retention.ms, consumers handling nulls, and compaction not being an audit log.
Last updated 16 October 2026
Sources