The full answer
Delta uses optimistic concurrency: each job reads a snapshot, writes new files, and checks at commit time whether another job changed the data it read. If so, the later commit fails. ConcurrentAppendException means the other job added files in the part of the table your MERGE read. The fix is to make the two jobs touch different data, or let Delta detect conflicts per row instead of per file, and retry the operation that fails.
Why two MERGEs conflict even on different rows
Without row-level concurrency, conflicts are detected per file. Job A updates customer 17 and job B updates customer 942, but if both rows live in the same file, both jobs rewrite that file and one of them fails. On an unpartitioned table, a MERGE that scans the whole table "reads" every file, so almost any concurrent write conflicts with it.
Diagnosis
- Read the exception and the table history (
DESCRIBE HISTORY) for the two commits: which operations, which partitions, at what times. - Check the table: partitioned or not, deletion vectors on or off, liquid clustering, identity column.
- Check the MERGE condition: does it filter the target to the slice it needs, or does it scan everything?
Fixes, in order of preference
| Fix | When it applies |
|---|---|
| Row-level concurrency | Databricks Runtime 14.3 LTS or later, an unpartitioned table and deletion vectors enabled. Then two MERGEs only conflict when they change the same row. |
| Explicit filters in the MERGE condition | On a partitioned table, add AND t.date = '2026-10-15' so each job only reads its own partition. A condition on the source alone does not limit what is read from the target. |
| Partition by the column the jobs split on | If the jobs naturally split by region or date and row-level concurrency is not an option. Avoid high-cardinality partition columns. |
| Serialise the writers | Two jobs that really change the same rows will always conflict. Run them in sequence, or merge their sources into one job. |
Retry, but retry the right thing
Under the default isolation level, plain appends never conflict; the operation that read the changed data is the one that fails. So put retry with backoff around the MERGE, UPDATE or DELETE. A retry re-reads the latest snapshot, which is safe for an idempotent MERGE.
Other causes to rule out
- A schema change,
ALTER TABLEor table property change during the run fails every concurrent writer. Schedule them in a quiet window. - Tables with identity columns do not support concurrent transactions at all.
OPTIMIZE ... ZORDER BYcan conflict with MERGE even with row-level concurrency.
What interviewers want: optimistic concurrency at commit time, why file-level conflict detection hits different rows, row-level concurrency and its requirements, explicit filters on the target in the MERGE condition, and retrying the reading operation.
Last updated 16 October 2026
Sources