Kafka and streaming interview questions for Data Engineer roles

Kafka orders messages within a partition, never across a topic. Give every event the customer's id as its key: the producer sends all records with the same key to the same partition (a hash of the key), and every consumer of that partition reads them in the order they were written. Two things break it: adding partitions, which moves keys to other partitions, and retries without idempotence, which can swap 2 batches. So size the partitions for the traffic expected in 1 to 2 years at the start, and keep the producer idempotent (the default).

1. Where the order lives

producer.send("customer-events", key=customer_id.encode(), value=event)

A record with no key is spread over partitions for throughput, so 2 events from one customer can land in different partitions and be read in either order.

2. Why adding partitions breaks it

The partition is the hash of the key modulo the number of partitions. Change the number and many keys map somewhere new. Kafka does not move existing data, so for a while a customer's old events sit in partition 3 and new ones in partition 7, read by different consumers at different speeds: an update can be processed before the event it follows.

Avoid it by choosing the count up front, from the throughput a partition takes from producers and gives a consumer, sized for the future traffic.

3. Why retries can reorder

If a batch fails and is retried while the next batch to the same partition succeeds, the second batch is written first. The idempotent producer (on by default, with acks=all, retries above 0 and at most 5 requests in flight) keeps the order. Turn idempotence off and keep several requests in flight, and order is no longer guaranteed.

4. Inside the consumer

Order holds as far as the consumer's own code keeps it. A consumer that hands records from one partition to a thread pool processes them out of order. Keep one key's records on one thread, for example by hashing the key to a worker.

What interviewers want: ordering per partition and per key, the key choosing the partition, why adding partitions moves keys, the idempotent producer keeping order through retries, and parallel processing inside the consumer as the last place order is lost.

acks says how many copies must have a message before the producer is told it succeeded; min.insync.replicas says how few in-sync copies the broker will accept for an acks=all write. With 3 copies, the usual safe setting is acks=all and min.insync.replicas=2: every acknowledged message is on at least 2 brokers, so losing any one loses nothing, and the topic keeps taking writes while one broker is down. With 2 of 3 down, only 1 copy is in sync, fewer than 2, so producers get NotEnoughReplicas errors: Kafka refuses writes rather than accept ones it could lose.

1. The 3 acks settings

acks=all is the default in current clients, and the idempotent producer needs it.

2. Why acks=all alone is not enough

The in-sync replica set (ISR) shrinks when followers fall behind. If it shrinks to just the leader, acks=all means "the leader has it", the same as acks=1. min.insync.replicas puts a floor under that: with a floor of 2, a write that only the leader could take is refused.

3. The cases with 3 copies and a floor of 2

AWS's MSK guidance is exactly this: 3 copies and min.insync.replicas of 2, because 1 copy takes partitions offline during an update and 2 copies can lose data.

4. What the producer should do when writes are refused

Retry with backoff within delivery.timeout.ms, and alert on the errors. Do not lower min.insync.replicas to 1 to get writes flowing: that trades an outage for silent data loss later.

5. Consumers see only safe messages

Whatever acks a producer used, a message is not visible to consumers until it is on all in-sync replicas and the floor is met, so a consumer never reads a message that a leader failure could take back.

What interviewers want: the 3 acks levels, the ISR shrinking, min.insync.replicas as the floor that makes acks=all mean something, replication 3 with a floor of 2 tolerating one broker down, and refusing writes over losing them.

Within a group, each partition goes to exactly one consumer, so 8 partitions keep at most 8 consumers busy; the other 4 wait. Adding consumers past the partition count adds nothing. The pauses are rebalances: a consumer that does not call poll() within max.poll.interval.ms is counted as failed, and its partitions are reassigned to the others, which stops their processing while the group settles. Fix the slow processing per poll (smaller batches with max.poll.records, or faster work), use a cooperative assignor so a rebalance moves only the partitions that change, and if one consumer per partition still cannot keep up, add partitions (planned, because keys move) rather than consumers.

1. Why 4 consumers are idle

Kafka tracks a group's progress as one offset per partition, which only works if one consumer owns each partition at a time. The parallelism of a group is therefore the partition count:

2. Why the group keeps pausing

The usual cause is processing that takes longer than max.poll.interval.ms (5 minutes by default) between polls: a large batch, a slow database call, a retry loop. The group coordinator decides the consumer has died and rebalances. The consumer then finishes, polls, rejoins, and the group rebalances again. Partitions that change owner are not processed until the rebalance completes.

3. The fixes, in order

1. Find where the time goes. kafka-consumer-groups.sh --describe --group my-group shows lag per partition and which consumer owns it. Lag on every partition means too little capacity; lag on one partition means a hot key.
2. Poll more often. Lower max.poll.records so each batch finishes well inside the interval, or raise max.poll.interval.ms if a batch is legitimately slow.
3. Rebalance less painfully. CooperativeStickyAssignor keeps as many existing assignments as possible and rebalances cooperatively, and a static group.instance.id with a longer session timeout lets a restarted consumer come back without a rebalance.
4. Then add partitions. If 8 consumers working flat out cannot keep up, the topic needs more partitions. Confluent's rule sizes them as the throughput divided by what one consumer reads from one partition. Plan it, because adding partitions remaps keys.

4. When lag is on one partition only

That is a hot key: one customer or one default value sending most of the traffic. More partitions do not help it, because a key always goes to one partition. Change the key, or split the hot key's work downstream.

What interviewers want: one consumer per partition per group, partitions as the ceiling on parallelism, rebalances caused by slow polls, cooperative rebalancing and static membership, reading lag per partition, and a hot key as the case more partitions cannot fix.

A compacted topic (cleanup.policy=compact) keeps at least the latest value for each key instead of deleting messages by age. Older values of a key are removed in the background, so the topic stays about as large as the number of keys, and a new consumer that reads it from the start rebuilds the current state: every customer's latest address, every product's latest price. Use it for changelogs of keyed, changing data, such as a database table captured with change data capture or an application's state. To delete a key, write a message with that key and a null value, a tombstone; compaction removes the earlier values, and the tombstone itself is removed after delete.retention.ms (1 day by default).

1. Delete versus compact

Both can be set (delete,compact): compacted, and also dropped after the retention time.

2. What a consumer sees

A consumer that keeps up with the head of the log sees every message. A consumer that starts from the beginning later may miss intermediate values of a key, but always ends with its latest one. Offsets do not change during compaction; some are simply gone.

3. Deleting a key

producer.send("customers", key=b"c-42", value=None)   # tombstone

Consumers must handle a null value as "this key was deleted". The tombstone stays for delete.retention.ms, which is also the time a consumer reading from offset 0 has to finish its scan and still see the delete. A consumer that takes longer could miss it and keep a deleted customer.

4. Things to watch

• Every message needs a key: compaction works per key.
• Compaction is not instant: recent messages stay uncompacted for a while, so a topic can hold several values of a key at any moment.
• It is a history of the latest state, not an audit trail. To keep every change, use a normal topic with long retention, or land the changes in a table with history, such as an SCD type 2 table.

What interviewers want: the latest value per key, the changelog and state-rebuild use case, tombstones with a null value and delete.retention.ms, consumers handling nulls, and compaction not being an audit log.

By default Kafka gives at-least-once: a producer that never hears back resends, and a consumer that crashes before committing its offset reads the same messages again. Exactly-once for a read-process-write service needs 3 things: the idempotent producer, so a resend is dropped by the broker; a transactional producer that writes the output messages and the consumer's offsets in one transaction, so either both happen or neither; and consumers of the output set to isolation.level=read_committed, so they never see an aborted transaction's messages. It covers Kafka to Kafka. Writing to a database or an API needs that side to be idempotent too.

1. Where duplicates come from

2. The idempotent producer

The broker gives each producer an id and checks a sequence number on every batch, dropping a batch it already has. It is on by default when acks=all and the related settings allow it. It fixes the first row only, and only within one producer session.

3. Transactions

producer = Producer({"transactional.id": "payments-enricher-1", ...})
producer.init_transactions()
while True:
    records = consumer.poll()
    producer.begin_transaction()
    for r in records:
        producer.produce("payments-enriched", key=r.key(), value=enrich(r))
    producer.send_offsets_to_transaction(consumer.position(...), consumer.consumer_group_metadata())
    producer.commit_transaction()

The output messages and the input offsets commit together. A crash before commit_transaction aborts both, so the restart reprocesses the same input and the aborted output is never seen. The transactional.id survives restarts, which lets the broker fence off an old copy of the service that is still running. Turn auto-commit off: offsets go through the transaction.

4. The reading side

Messages of an aborted transaction stay in the log. A consumer with the default read_uncommitted returns them; read_committed returns only committed ones, holding back messages behind an open transaction.

5. Where it stops

The guarantee covers Kafka topics and offsets. If the service also writes to Postgres or calls a card API, a crash after that write repeats it. Make that write idempotent, such as an upsert by payment id, which the Kafka docs note is idempotent whenever updates are keyed by a primary key. Kafka also warns that many exactly-once claims do not survive a failure, so ask which failures a claim covers.

What interviewers want: at-least-once as the default, the idempotent producer's id and sequence numbers, transactions that commit output and offsets together, read_committed on the consumers, fencing by transactional.id, and the boundary at external systems.