Structured Streaming Essentials
Test what you know about Spark Structured Streaming: output modes, watermarks and late data, checkpoints, triggers, sink guarantees, foreachBatch, what you can change between restarts, and streaming from Delta and Kafka.
Question 1 of 12
Question 1
What is the default output mode of a streaming query?
Not this time. Append is the default: only rows added to the result since the last trigger are written. It suits queries whose output rows never change once written.
Question 2
A streaming aggregation uses a watermark but runs in Complete mode. What happens to its state?
Not this time. Complete mode writes the whole result table every trigger, so it must keep every aggregate. The watermark cannot drop any state, and state keeps growing. Use Append or Update for the watermark to clean up.
Question 3
A query has withWatermark('eventTime', '2 hours'). What is guaranteed about an event 3 hours late?
Not this time. The guarantee runs one way only. Data less than 2 hours behind the latest event time seen is always processed. Data later than that is not guaranteed to be dropped: it may or may not be counted, and the later it is, the less likely.
Question 4
2 streaming queries write to different tables. Can they share one checkpoint location?
Not this time. No. The checkpoint holds a query's offsets, commits, state and its unique query ID. Each query must have its own checkpoint location, and queries should never share one.
Question 5
A job should start, process everything that has arrived since its last run, then stop. Which trigger fits?
Not this time. The available-now trigger processes all available data, in as many micro-batches as needed, then stops. It replaces the once trigger, which is deprecated.
Question 6
What does foreachBatch guarantee for the writes inside it?
Not this time. foreachBatch gives at-least-once writes: after a failure, a micro-batch can run again. Use the batchId it passes to skip a batch already written, or make the write idempotent, for exactly-once results.
Question 7
Which sink gives exactly-once output in the Spark guide, and which only at-least-once?
Not this time. The file sink is listed as exactly-once. The Kafka and foreach sinks are at-least-once, so a consumer of the Kafka topic may see a record twice after a failure.
Question 8
You restart a query from its checkpoint with one change. Which change does the Spark guide allow?
Not this time. Adding, removing or changing a rate limit such as maxOffsetsPerTrigger is allowed. Changing the number or type of input sources is not, changing subscribed topics is generally not, and adding or removing stateful operations is not.
Question 9
A stream reads a Delta table, and someone runs UPDATE on that table. What happens by default?
Not this time. A Delta streaming source expects appends. An upstream update or delete makes the stream fail with an error. skipChangeCommits lets it go on reading appends only, ignoring the changed files; a full refresh or the change data feed are the other routes.
Question 10
What does maxFilesPerTrigger control when streaming from a Delta table?
Not this time. It is the number of new files considered in each micro-batch, 1000 by default. maxBytesPerTrigger is a soft limit by size. Both keep batch size and latency steady.
Question 11
The same payment arrives twice with different event times. Which operator removes the copy within the watermark?
Not this time. dropDuplicatesWithinWatermark removes rows with the same unique id within the watermark delay, and does not need the event time to match. That suits a sender that stamps the time when it writes.
Question 12
Why does a stream-static join need no watermark?
Not this time. A stream-static join matches each new streaming row against a static DataFrame, so no state is kept between micro-batches. A stream-stream join is different: it buffers both sides as state, and watermarks bound it.
Sources