# What is a watermark in stream processing?

In stream processing, a marker that says how far event time has got: data older than the latest event seen, less a set delay, is treated as too late, so windows can close and their state be freed.

- Page: https://tecyfy.com/concepts/watermark
- Subject: Data engineering in the AI era (https://tecyfy.com/concepts)
- Level: intermediate
- Also called: Event-time watermark, Late data threshold, withWatermark

A watermark is how a streaming job decides that data for a past moment has stopped arriving. It is set as a delay, such as 10 minutes: once the job has seen an event from 21:20, it treats anything from before 21:10 as too late, closes the windows that ended before then, and frees the memory they held.

## 2 clocks in every stream

Every event carries 2 times. **Event time** is when it happened: when the card was tapped, when the sensor read the temperature. **Processing time** is when the stream got round to it. They are rarely the same, and the gap is not steady. A payment terminal that loses its signal sends 5 minutes of payments at once when it comes back. A phone in a tunnel sends its clicks an hour late.

A report about what happened, such as payments per card per 5 minutes, has to count by event time. Counting by processing time would put the terminal's 5 lost minutes into the minute it reconnected. But counting by event time raises a question the stream cannot answer for certain: has every payment from 21:00 to 21:05 arrived yet?

## How the watermark answers it

The watermark is a guess written as a rule. The stream tracks the latest event time it has seen and subtracts the delay you set. That point is the watermark. A window whose end is behind it is closed: its count is final, its state is dropped from memory, and an event that arrives for it afterwards is not counted. In Apache Spark's Structured Streaming it is one line, `withWatermark("event_time", "10 minutes")`, before the aggregation.

(An interactive demo goes here on the page: https://tecyfy.com/concepts/watermark)

## Choosing the delay

The delay trades lateness against memory and waiting. A short delay keeps few windows open and writes results soon, but drops more late data. A long delay counts more stragglers, but holds more state and, in append mode, writes each window's result only once the watermark passes its end. In update mode a count is written as it changes, so a dashboard sees it at once but must accept that it can still change. Complete mode rewrites everything and cannot drop state, so it grows for ever.

Pick the delay from the data, not from a default. Measure how late events actually arrive, for example the gap between the event time and the time the stream read it, and choose a delay above nearly all of them. Spark promises to count anything less late than the delay. Anything later may or may not be counted, so a rare very late event needs another path, such as a nightly batch that corrects the totals.

## The same idea for duplicates and joins

A watermark also bounds state outside windows. Removing duplicate events by an id means remembering every id seen; with `dropDuplicatesWithinWatermark`, ids older than the watermark are forgotten, so the delay should be longer than the largest gap between 2 copies of the same event. A join between 2 streams needs a watermark on each side to know when a payment can no longer match a refund, and an outer join cannot run without one. When streams are combined, Spark uses the slowest stream's watermark by default, so one quiet source holds back the results of all of them.

## Where it shows up

Engines that count by event time need one: Apache Flink uses the same name and the same idea. Watermarks sit in the cleaning step of the [medallion architecture](https://tecyfy.com/concepts/medallion-architecture), where late and out-of-order data is put right, and the dropped count belongs with the other [data quality](https://tecyfy.com/concepts/data-quality) checks: a sudden rise in late events usually means a source is struggling.

## Related

- [Data Pipeline](https://tecyfy.com/concepts/data-pipeline): A set of steps that moves data from where it is made to where it is used, cleaning and reshaping it on the way, run on a schedule or as data arrives.
- [Change Data Capture](https://tecyfy.com/concepts/change-data-capture): Copying only the rows that were added, changed or deleted in a source database, read from its change log, instead of copying the whole table again.
- [Medallion Architecture](https://tecyfy.com/concepts/medallion-architecture): Organising a lakehouse in 3 layers of rising quality: bronze for raw data, silver for cleaned and joined data, and gold for tables shaped for reports and AI.
- [Data Quality](https://tecyfy.com/concepts/data-quality): How far data can be trusted: complete, correct, on time and in the expected shape. Pipelines check it with rules and stop or flag rows that fail.

## Sources

- [Structured Streaming Programming Guide: APIs on DataFrames and Datasets](https://spark.apache.org/docs/latest/streaming/apis-on-dataframes-and-datasets.html)
- [Apply watermarks to control data processing thresholds](https://docs.databricks.com/aws/en/structured-streaming/watermarks)
- [Timely Stream Processing](https://nightlies.apache.org/flink/flink-docs-stable/docs/concepts/time/)