What does exactly-once mean in Kafka? A service reads payments from one topic, writes enriched payments to another, and must never write a payment twice or lose one, even if it crashes midway.

By default Kafka gives at-least-once: a producer that never hears back resends, and a consumer that crashes before committing its offset reads the same messages again. Exactly-once for a read-process-write service needs 3 things: the idempotent producer, so a resend is dropped by the broker; a transactional producer that writes the output messages and the consumer's offsets in one transaction, so either both happen or neither; and consumers of the output set to isolation.level=read_committed, so they never see an aborted transaction's messages. It covers Kafka to Kafka. Writing to a database or an API needs that side to be idempotent too.

1. Where duplicates come from

2. The idempotent producer

The broker gives each producer an id and checks a sequence number on every batch, dropping a batch it already has. It is on by default when acks=all and the related settings allow it. It fixes the first row only, and only within one producer session.

3. Transactions

producer = Producer({"transactional.id": "payments-enricher-1", ...})
producer.init_transactions()
while True:
    records = consumer.poll()
    producer.begin_transaction()
    for r in records:
        producer.produce("payments-enriched", key=r.key(), value=enrich(r))
    producer.send_offsets_to_transaction(consumer.position(...), consumer.consumer_group_metadata())
    producer.commit_transaction()

The output messages and the input offsets commit together. A crash before commit_transaction aborts both, so the restart reprocesses the same input and the aborted output is never seen. The transactional.id survives restarts, which lets the broker fence off an old copy of the service that is still running. Turn auto-commit off: offsets go through the transaction.

4. The reading side

Messages of an aborted transaction stay in the log. A consumer with the default read_uncommitted returns them; read_committed returns only committed ones, holding back messages behind an open transaction.

5. Where it stops

The guarantee covers Kafka topics and offsets. If the service also writes to Postgres or calls a card API, a crash after that write repeats it. Make that write idempotent, such as an upsert by payment id, which the Kafka docs note is idempotent whenever updates are keyed by a primary key. Kafka also warns that many exactly-once claims do not survive a failure, so ask which failures a claim covers.

What interviewers want: at-least-once as the default, the idempotent producer's id and sequence numbers, transactions that commit output and offsets together, read_committed on the consumers, fencing by transactional.id, and the boundary at external systems.