What is an idempotent pipeline, and what does exactly-once really mean?

A pipeline step is idempotent when running it twice gives the same result as running it once. Loading Monday's orders, then loading them again after a crash, should leave one copy of Monday's orders, not 2. Exactly-once is the promise built on top of that: each record affects the result once, even though the machinery underneath may deliver or process it more than once. In practice, exactly-once is almost always at-least-once delivery plus an idempotent write.

Why a record arrives twice

Every pipeline fails sometimes: a machine restarts, a network call times out, a job is killed halfway. The safe response is to try again, and that is where duplicates come from. The Kafka documentation gives the classic case. A producer sends a message, the broker stores it, and the acknowledgement is lost on the way back. The producer cannot tell whether the message arrived, so it sends it again. Kafka names the 3 possible promises:

• At most once: never sent again, so a message can be lost.
• At least once: sent again until confirmed, so a message can arrive twice.
• Exactly once: each message processed once and only once.

Losing data is rarely acceptable, so most systems choose at least once and then make the duplicates harmless. Kafka itself warns to read the fine print of any exactly-once claim.

How each tool makes a retry safe

• Kafka producers can be idempotent: the broker gives each producer an id and drops a resent message by its sequence number. Transactions go further, writing to several partitions and the consumer's position all or nothing.
• Structured Streaming records which offsets each micro-batch read in its checkpoint, so after a failure it reprocesses exactly that batch. With a source that can be replayed, such as Kafka, and a sink that ignores a batch it already has, such as a Delta table, the result is exactly-once. A foreachBatch write is at-least-once unless you make it idempotent; on Delta, the txnAppId and txnVersion options, with the version set to the batch id, do that.
• Airflow retries failed tasks, so its best practices ask that a task give the same outcome every time it runs: write with an upsert rather than an INSERT, and read and write one data interval rather than "the latest data".

The 3 habits that make a step idempotent

1. Upsert by a key instead of appending. A MERGE on the order id writes the same row whether the order arrives once or 3 times. Kafka's docs note that any update keyed by a primary key is idempotent this way.
2. Overwrite a defined slice. A daily job that replaces the partition for its own day, chosen by the run's date and not by today's clock, can be rerun for last Tuesday and give last Tuesday's answer.
3. Never half-finish. Airflow's advice is to treat a task like a database transaction. Write to a table that commits all or nothing, as a Delta table does, so a crash leaves the old result rather than a partial one.

A WatermarkWatermarkIn stream processing, a marker that says how far event time has got: data older than the latest event seen, less a set delay, is treated as too late, so windows can close and their state be freed. plus deduplication by an event id handles the duplicates that come from the source itself, such as a payment sent twice by the client, which no retry setting can see.

Why it matters for AI

An AI agent that calls tools has the same problem in a new place. An agent retries a call that timed out, and if the tool sends an email or charges a card, the customer gets 2. The fix is the same: give each action a key, and make the tool return the earlier result when the same key comes back. A pipelineData PipelineA set of steps that moves data from where it is made to where it is used, cleaning and reshaping it on the way, run on a schedule or as data arrives. that feeds an assistant's answers also needs to be idempotent, because a doubled day of sales is a confident wrong answer.