The full answer
The watermark assumes rows become visible in updated_at order, and they do not. In PostgreSQL, now() and CURRENT_TIMESTAMP return the time the transaction started, but other sessions only see the row when the transaction commits. A transaction that starts at 10:00 and commits at 10:20 writes updated_at = 10:00. If your 10:15 extract already moved the watermark past 10:00, that row is never picked up. Hard deletes are lost too, because a deleted row has no updated_at to find.
Walk through the failure
| Time | Source | Your extract |
|---|---|---|
| 10:00 | Transaction A starts, updates order 7 with updated_at = now() = 10:00 | |
| 10:05 | Transaction B updates order 9, updated_at = 10:05, commits | |
| 10:15 | Reads updated_at > 09:45, sees order 9 (A has not committed), sets watermark to 10:05 | |
| 10:20 | A commits | |
| 10:30 | Reads updated_at > 10:05, order 7 (10:00) is skipped for good |
Other causes of the same symptom:
- Code paths or bulk scripts that update rows without touching
updated_at. - Clock or time zone differences when the watermark is computed on a different machine.
- A watermark taken from the extract's own clock instead of the maximum value it actually read.
Fixes, from quick to thorough
- Overlap window. Read
updated_at > last_watermark - interval '1 hour'and upsert by primary key, so rereading rows does no harm. Choose the overlap longer than your longest transaction, and alert on transactions that run longer. - Set
updated_atwith a trigger, so every write path sets it, including scripts. - Track deletes: soft deletes with a
deleted_at, or a periodic full key comparison. - Use change data capture from the database log such as PostgreSQL's logical replication. It sends every change, including deletes, and applies them in the same order as the source, so the timestamp problem goes away.
Prove the gap is closed
Run a daily reconciliation: counts and a checksum of key columns per day or per key range, source against target. It catches every one of these causes, including ones you have not thought of.
What interviewers want: understanding that commit order differs from timestamp order, now() returning the transaction start, the overlap-and-upsert fix, deletes, CDC as the thorough answer, and reconciliation as the safety net.
Last updated 16 October 2026
Sources