# What is Delta Lake?

An open table format that turns Parquet files in cloud storage into a table with transactions, a version history, schema checks, and updates and deletes, so a data lake behaves like a database.

- Page: https://tecyfy.com/concepts/delta-lake
- Subject: Data engineering in the AI era (https://tecyfy.com/concepts)
- Level: beginner
- Also called: Delta table, Delta tables, Delta Lake table format

Cloud storage is cheap and holds anything, but a folder of files is not a table. Nobody knows which files belong to the current version, a failed job leaves half its files behind, and changing one row means rewriting files by hand. Delta Lake fixes that by adding one thing: a log.

## Files plus a log

A Delta table is a folder. The data sits in ordinary Apache Parquet files. Next to them is a `_delta_log` folder of numbered JSON files, one per commit, each saying which data files were added and which removed. Every so often a checkpoint file sums up the log so readers do not replay thousands of entries.

```
orders/
  part-00000-....parquet
  part-00001-....parquet
  _delta_log/
    00000000000000000000.json
    00000000000000000001.json
```

The table is whatever the log says it is. That single rule gives Delta its features.

## What the log makes possible

- **Transactions.** A write adds files, then commits one log entry. If it fails before the commit, the new files are ignored. Readers never see half a write. See [ACID transactions](https://tecyfy.com/concepts/acid-transactions).
- **Time travel.** Older log versions still point at older files, so `SELECT * FROM orders VERSION AS OF 12` reads the table as it was, for audits, debugging or reproducing a model's training data.
- **Schema enforcement.** The log stores the table's schema, so a write with a wrong column type is rejected instead of corrupting the table.
- **Updates, deletes and MERGE.** Changing rows means writing new files and marking old ones removed in one commit, which is what [change data capture](https://tecyfy.com/concepts/change-data-capture) and upserts need.
- **Batch and streaming together.** A stream can append to the same table a batch job reads.

## Where it sits

Delta Lake is open source and one of the [open table formats](https://tecyfy.com/concepts/open-table-format), alongside Apache Iceberg and Apache Hudi. Databricks uses it as the default format for every table and adds features on top, such as liquid clustering and multi-table transactions through Unity Catalog. Other engines, Spark, Trino, DuckDB and more, read and write it too.

Small files pile up as a table takes many small writes, which slows reads; [Compaction](https://tecyfy.com/concepts/compaction) rewrites them into fewer large ones, and `VACUUM` deletes files the log no longer needs once the time-travel window has passed.

Our [Delta Lake vs Apache Iceberg](/data-engineering/delta-lake-vs-apache-iceberg) comparison sets the 2 formats side by side, with the same job written in each.

## Related

- [Open Table Format](https://tecyfy.com/concepts/open-table-format): A published way of storing a table as plain data files plus a log of changes, so many engines can read and safely write it. Delta Lake and Apache Iceberg are the best known.
- [Lakehouse](https://tecyfy.com/concepts/lakehouse): One store for all of an organisation's data: cheap files in cloud storage, as in a data lake, with the tables, transactions and fast queries of a data warehouse on top.
- [ACID Transactions](https://tecyfy.com/concepts/acid-transactions): The 4 promises a database makes about a change: it happens completely or not at all, leaves valid data, does not see other changes half-done, and survives a crash once committed.
- [Data Lake](https://tecyfy.com/concepts/data-lake): Cheap storage that holds data of any kind in its raw form, usually as files in cloud object storage, to be cleaned and structured later.
- [Change Data Capture](https://tecyfy.com/concepts/change-data-capture): Copying only the rows that were added, changed or deleted in a source database, read from its change log, instead of copying the whole table again.
- [Compaction](https://tecyfy.com/concepts/compaction): Replacing the older part of a long conversation with a summary, so an agent can keep working when its context fills up.

## Sources

- [Welcome to the Delta Lake documentation](https://docs.delta.io/latest/delta-intro.html)
- [Delta Transaction Log Protocol](https://github.com/delta-io/delta/blob/master/PROTOCOL.md)
- [What are ACID guarantees on Databricks?](https://docs.databricks.com/aws/en/lakehouse/acid)