The full answer
A stream falls behind when each micro-batch takes longer to process than the time it took for that much data to arrive. Compare the input rate with the processing rate in the query's progress reports, then find out what makes batches slow. The usual causes are state that keeps growing, a slow sink, too little parallelism, or a sudden surge of input.
1. Confirm it with the progress metrics
Each trigger reports its progress (query.lastProgress, or a StreamingQueryListener that sends it to your monitoring):
p = query.lastProgress
p["inputRowsPerSecond"], p["processedRowsPerSecond"]
p["durationMs"] # where the batch time went
p["stateOperators"] # numRowsTotal per stateful operator
If processedRowsPerSecond stays below inputRowsPerSecond, the backlog grows. For Kafka, also watch consumer lag on the topic.
2. Find out why batches are slow
| Signal | Likely cause |
|---|---|
numRowsTotal in state grows without end, batch time grows with it | State is never cleaned up: no watermark, or a dedup or join on keys that never expire |
| Batch time jumps around, executor GC time high | Large state held in JVM memory |
durationMs dominated by the write | Slow sink: a database, many small files, or a MERGE on a large table |
| One task per batch much slower than the rest | Skewed key, or too few input partitions |
| Batches suddenly huge after a restart or outage | Catching up on a backlog with no rate limit |
3. Fixes
- Bound the state. Add
withWatermarkto aggregations, deduplication and stream-stream joins, so Spark can drop state for windows that can no longer change. Without it, state grows for as long as the query runs. - Move large state off the heap. With millions of keys, switch to the RocksDB state store (
spark.sql.streaming.stateStore.providerClassset to theRocksDBStateStoreProviderclass). The default store keeps state in executor memory, where millions of keys cause long garbage collection pauses. - Fix the sink. Write in bulk inside
foreachBatchrather than row by row, and keep the target table free of small files. If the sink is a MERGE, check how much of the table each batch rewrites. - Cap the batch size.
maxOffsetsPerTriggerfor Kafka ormaxFilesPerTriggerfor files keeps catch-up batches a size the cluster can finish. Changing these rate limits is allowed between restarts. - Add parallelism. More Kafka partitions, more cores, or a repartition on a better-spread key.
4. Check the trigger
If a batch runs past the trigger interval, Spark starts the next one at once. That keeps the job busy but does not make it faster. Falling behind is a throughput problem, so a shorter interval will not help.
Prevent it
Alert on the gap between input and processed rates, on Kafka lag, and on state size, not only on query failures. A stream that is losing ground keeps running without errors until the backlog is hours long.
What interviewers want: reading the progress metrics, knowing that unbounded state makes a stream slower every hour, and the watermark and rate-limit fixes.
Last updated 16 October 2026
Sources