The full answer
Delta Lake's transaction log provides the consistency: every write either commits completely or not at all, and readers always see a complete snapshot. Data Factory's job is to run the writes in a way that avoids conflicts and retries safely when one happens.
1. What Delta Lake guarantees
- Atomic commits. A write adds files and then commits one entry to the
_delta_log. If the job fails before committing, the files are ignored. - Snapshot reads. Readers see the last committed version, never half of a write.
- Optimistic concurrency. There are no locks. Each writer reads the current version, writes its files, and at commit time checks whether another commit changed the data it depended on. If so, its commit fails with a conflict exception (such as
ConcurrentAppendException) instead of corrupting the table.
2. How conflicts actually arise
- Conflicts are detected on the files a write read or rewrote, not on individual rows. Two
MERGEorUPDATEjobs touching different rows that sit in the same files can still conflict. - Appends that only add new files rarely conflict.
- On Azure Databricks, row-level concurrency (with deletion vectors and liquid clustering) detects conflicts per row, which removes many of these false conflicts.
- With the default WriteSerializable isolation, concurrent appends do not block other operations; Serializable is stricter.
3. How to run updates from ADF
- ADF writes Delta through mapping data flows (Delta as source and sink, with insert, update, upsert and delete), or by calling a Databricks job, notebook or SQL statement that runs
MERGE. - Avoid concurrent writers on the same data: serialise pipelines that update the same table (pipeline concurrency set to 1), or partition the work so each writer touches different partitions, and include the partition column in the
MERGEcondition so Delta knows the writes don't overlap. - Make writes idempotent: a
MERGEon a business key gives the same result if retried, unlike a blind append. - Retry on conflict: set a retry count and interval on the activity; a retried
MERGEreads the new version and usually succeeds.
4. Example
Two pipelines update orders: one sets status for orders 1–100, the other sets ship_date for orders 50–150. If the table is partitioned by order_date and both batches cover the same dates, the second commit can fail with a conflict; Delta never applies one update over the other blindly. The retried activity re-reads the table and applies its change on top. Partitioning the work or running the two updates in sequence avoids the conflict altogether.
5. Monitor
Use DESCRIBE HISTORY orders to see each commit and the operation that made it, and alert on repeated conflict failures, which signal that the pipelines' schedules or layout need changing.
Last updated 16 October 2026
Sources