What are ACID transactions, and why do data tables need them?

A bank transfer is the classic example: take 100 from one account, add 100 to another. If the system crashes between the 2 steps, money has vanished. ACID is the name for the guarantees that stop that happening.

The 4 letters

• Atomicity. The change is all or nothing. Both updates land, or neither does.
• Consistency. A change moves the data from one valid state to another. Rules such as "balance cannot be negative" hold before and after.
• Isolation. Changes running at the same time do not see each other half-done. A report that runs during the transfer sees the money in one account or the other, never in neither.
• Durability. Once the system says "committed", the change survives a crash or power cut.

Databases such as PostgreSQL have offered this for decades. Data lakes did not: a folder of Parquet files in cloud storage has no idea that 3 files belong to one change, so a job that fails halfway leaves half a table, and a reader can see it.

How a lakehouse gets it

Open table formatsOpen Table FormatA published way of storing a table as plain data files plus a log of changes, so many engines can read and safely write it. Delta Lake and Apache Iceberg are the best known. such as Delta LakeDelta LakeAn open table format that turns Parquet files in cloud storage into a table with transactions, a version history, schema checks, and updates and deletes, so a data lake behaves like a database. and Apache Iceberg add a transaction log next to the data files. A writer first writes new data files, then commits one entry in the log that says "these files are now the table". Until that entry exists, readers do not see the new files; if the job fails, the files are simply never committed. That one log entry is the atomic step.

Isolation comes from snapshots: a reader picks a version of the log and reads exactly those files, while writers add newer versions. There are no locks. Instead, 2 writers that change the same data are caught at commit time, and the second one fails and retries. This is called optimistic concurrency.

One table, or several

For years these guarantees covered one table at a time. A pipeline that updated an orders table and an inventory table ran 2 separate transactions, and a failure between them left the tables disagreeing. Since October 2026, Databricks commits statements across up to 100 tables as one transaction, with BEGIN ATOMIC ... END, on Unity Catalog managed tables.

Why it matters for AI

An AI agent that writes data, booking, refunding or updating a record in several places, is an unreliable writer: it may fail or be stopped midway. ACID transactions are what make its half-finished work disappear instead of leaking into the next report. They also make retries safeIdempotent Pipelines and Exactly-OnceA pipeline step is idempotent when running it twice gives the same result as running it once. That is what makes retries safe, and it is how systems that deliver a message more than once still count it exactly once.: a step that failed left nothing behind, so running it again cannot double-count.