What is a watermark in stream processing?
In stream processing, a marker that says how far event time has got: data older than the latest event seen, less a set delay, is treated as too late, so windows can close and their state be freed.
intermediate · Also called Event-time watermark, Late data threshold, withWatermark
A watermark is how a streaming job decides that data for a past moment has stopped arriving. It is set as a delay, such as 10 minutes: once the job has seen an event from 21:20, it treats anything from before 21:10 as too late, closes the windows that ended before then, and frees the memory they held.
2 clocks in every stream
Every event carries 2 times. Event time is when it happened: when the card was tapped, when the sensor read the temperature. Processing time is when the stream got round to it. They are rarely the same, and the gap is not steady. A payment terminal that loses its signal sends 5 minutes of payments at once when it comes back. A phone in a tunnel sends its clicks an hour late.
A report about what happened, such as payments per card per 5 minutes, has to count by event time. Counting by processing time would put the terminal's 5 lost minutes into the minute it reconnected. But counting by event time raises a question the stream cannot answer for certain: has every payment from 21:00 to 21:05 arrived yet?
How the watermark answers it
The watermark is a guess written as a rule. The stream tracks the latest event time it has seen and subtracts the delay you set. That point is the watermark. A window whose end is behind it is closed: its count is final, its state is dropped from memory, and an event that arrives for it afterwards is not counted. In Apache Spark's Structured Streaming it is one line, withWatermark("event_time", "10 minutes"), before the aggregation.
The newest is N, paid at 21:21
Time the payment was made (event time), in 5-minute windows
Latest payment seen, 21:21Watermark, 21:11Closed window
21:00
Final, state dropped
21:05
Final, state dropped
21:10
…
Waiting
21:15
…
Waiting
21:20
…
Waiting
Counted
13
Dropped as too late
1
Windows in memory
3
| Payment | Paid | Came after | Watermark | Result |
|---|---|---|---|---|
| D | 21:04 | 21:06 | before 21:00 | Counted |
| G | 21:02 | 21:12 | 21:02 | Counted |
| I | 21:08 | 21:14 | 21:04 | Counted |
| K | 21:13 | 21:17 | 21:07 | Counted |
| M | 21:03:30 | 21:19 | 21:09 | Dropped |
N, paid at 21:21, is the newest payment yet, so the watermark moves to 21:11. Waiting 10 minutes, 13 of 14 payments are counted and 3 windows are still held in memory. In append mode only the closed windows have been written.
The payments are made up, and each one is its own micro-batch. In Structured Streaming the watermark is the latest event time seen less the delay, and a window's state is dropped once the watermark passes its end. Spark promises to count anything less late than the delay; later data may or may not be counted.
Choosing the delay
The delay trades lateness against memory and waiting. A short delay keeps few windows open and writes results soon, but drops more late data. A long delay counts more stragglers, but holds more state and, in append mode, writes each window's result only once the watermark passes its end. In update mode a count is written as it changes, so a dashboard sees it at once but must accept that it can still change. Complete mode rewrites everything and cannot drop state, so it grows for ever.
Pick the delay from the data, not from a default. Measure how late events actually arrive, for example the gap between the event time and the time the stream read it, and choose a delay above nearly all of them. Spark promises to count anything less late than the delay. Anything later may or may not be counted, so a rare very late event needs another path, such as a nightly batch that corrects the totals.
The same idea for duplicates and joins
A watermark also bounds state outside windows. Removing duplicate events by an id means remembering every id seen; with dropDuplicatesWithinWatermark, ids older than the watermark are forgotten, so the delay should be longer than the largest gap between 2 copies of the same event. A join between 2 streams needs a watermark on each side to know when a payment can no longer match a refund, and an outer join cannot run without one. When streams are combined, Spark uses the slowest stream's watermark by default, so one quiet source holds back the results of all of them.
Where it shows up
Engines that count by event time need one: Apache Flink uses the same name and the same idea. Watermarks sit in the cleaning step of the medallion architectureMedallion ArchitectureOrganising a lakehouse in 3 layers of rising quality: bronze for raw data, silver for cleaned and joined data, and gold for tables shaped for reports and AI., where late and out-of-order data is put right, and the dropped count belongs with the other data qualityData QualityHow far data can be trusted: complete, correct, on time and in the expected shape. Pipelines check it with rules and stop or flag rows that fail. checks: a sudden rise in late events usually means a source is struggling.
Sources