What is Delta Lake?

Cloud storage is cheap and holds anything, but a folder of files is not a table. Nobody knows which files belong to the current version, a failed job leaves half its files behind, and changing one row means rewriting files by hand. Delta Lake fixes that by adding one thing: a log.

Files plus a log

A Delta table is a folder. The data sits in ordinary Apache Parquet files. Next to them is a _delta_log folder of numbered JSON files, one per commit, each saying which data files were added and which removed. Every so often a checkpoint file sums up the log so readers do not replay thousands of entries.

orders/
  part-00000-....parquet
  part-00001-....parquet
  _delta_log/
    00000000000000000000.json
    00000000000000000001.json

The table is whatever the log says it is. That single rule gives Delta its features.

What the log makes possible

• Transactions. A write adds files, then commits one log entry. If it fails before the commit, the new files are ignored. Readers never see half a write. See ACID transactionsACID TransactionsThe 4 promises a database makes about a change: it happens completely or not at all, leaves valid data, does not see other changes half-done, and survives a crash once committed..
• Time travel. Older log versions still point at older files, so SELECT * FROM orders VERSION AS OF 12 reads the table as it was, for audits, debugging or reproducing a model's training data.
• Schema enforcement. The log stores the table's schema, so a write with a wrong column type is rejected instead of corrupting the table.
• Updates, deletes and MERGE. Changing rows means writing new files and marking old ones removed in one commit, which is what change data captureChange Data CaptureCopying only the rows that were added, changed or deleted in a source database, read from its change log, instead of copying the whole table again. and upserts need.
• Batch and streaming together. A stream can append to the same table a batch job reads.

Where it sits

Delta Lake is open source and one of the open table formatsOpen Table FormatA published way of storing a table as plain data files plus a log of changes, so many engines can read and safely write it. Delta Lake and Apache Iceberg are the best known., alongside Apache Iceberg and Apache Hudi. Databricks uses it as the default format for every table and adds features on top, such as liquid clustering and multi-table transactions through Unity Catalog. Other engines, Spark, Trino, DuckDB and more, read and write it too.

Small files pile up as a table takes many small writes, which slows reads; CompactionCompactionReplacing the older part of a long conversation with a summary, so an agent can keep working when its context fills up. rewrites them into fewer large ones, and VACUUM deletes files the log no longer needs once the time-travel window has passed.

Our Delta Lake vs Apache Iceberg comparison sets the 2 formats side by side, with the same job written in each.