The full answer
The checkpoint stores the query's progress (offsets) and its state, and both assume the query keeps the same shape. Some code changes break that assumption: a different source, a different file sink path, or any change to a stateful operation (the keys of an aggregation, the columns of a dedup, a stream-stream join). Spark does not support those changes on an existing checkpoint, and the restart fails or behaves unpredictably. You either undo the change, or start a new checkpoint and decide how to rebuild what the old one held.
What is safe and what is not
| Change | With the same checkpoint |
|---|---|
| Add or remove a filter | Allowed |
| Change a projection, same output schema | Allowed |
Add or change maxOffsetsPerTrigger | Allowed |
| Change the Kafka topic you subscribe to | Generally not allowed |
| Add a second source, or change source type | Not allowed |
| Change a file sink's output path | Not allowed |
Add a column to a groupBy, change an aggregate | Not allowed |
| Change dedup columns, join keys or join type | Not allowed |
Change the state schema or timeout of flatMapGroupsWithState | Not allowed |
What to do now
- Can you roll the change back? If the change is not urgent, revert it, restart from the existing checkpoint, and plan the change properly.
- If the change must go in, start a new checkpoint. Point the query at a new checkpoint location and decide where it should start reading:
- Stateless query: start from the offsets the old query had reached (for Kafka, set
startingOffsetsfrom the last committed offsets), so nothing is skipped or read twice. - Stateful query: the old state is gone. Rebuild it by reprocessing from a point early enough to cover the state you need, for example the start of the open aggregation windows, and make the sink tolerate the replay.
- Stateless query: start from the offsets the old query had reached (for Kafka, set
- Make the sink tolerate replays.
foreachBatchgives at-least-once delivery by default. Use thebatchIdto skip a batch already written, or write with a MERGE on a key, so reprocessing does not create duplicates.
How to avoid it next time
- Treat a stateful query's keys and state schema as a contract. Changing them is a migration, planned like a table schema change.
- Version checkpoint paths with the query (
/checkpoints/orders_agg/v3), so a new version gets a clean location by design. - Keep sinks idempotent from the start, because every recovery plan depends on replaying data safely.
- If state must evolve, store it as bytes in a format that supports schema evolution, such as Avro.
Never do this
Do not delete the checkpoint folder of the running query and restart in place. The query would then start from its startingOffsets setting, which by default for Kafka is the latest offset, and silently skip everything in between.
What interviewers want: that the checkpoint holds offsets and state tied to the query's shape, which changes are not allowed, and a recovery plan that neither loses nor duplicates data.
Last updated 16 October 2026