# Delta Lake vs Apache Iceberg: how the 2 table formats differ and which to choose

Delta Lake and Apache Iceberg are both open table formats - Parquet files plus a metadata layer that adds transactions, time travel, MERGE and schema changes to files in object storage. Delta Lake keeps an ordered log of commits next to the data and fits best where Spark and Databricks do most of the work. Iceberg keeps a tree of snapshots that a catalog points to, and has the widest choice of engines that can write the same table, including Snowflake, Trino, Flink and Athena. Today the choice is less final than it was - Delta tables can be read as Iceberg through UniForm, and Databricks and Snowflake both manage Iceberg tables.

- Page: https://tecyfy.com/data-engineering/delta-lake-vs-apache-iceberg
- Part of: Data engineering (https://tecyfy.com/data-engineering)

- **Delta Lake** (Linux Foundation project, started by Databricks, https://docs.delta.io/latest/): A table format whose transaction log, a folder of numbered commits beside the Parquet files, records every change to the table. Install: `pip install delta-spark`
- **Apache Iceberg** (Apache Software Foundation project, started at Netflix, https://iceberg.apache.org/docs/latest/): A table format whose metadata files list snapshots of the table, each pointing to manifests of data files, with a catalog holding the current one. Install: `spark.jars.packages=org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.12.0`

A data lake on its own is a folder of Parquet files. It has no idea which files make up the table right now, so 2 jobs writing at once can leave readers with half a change, and an update means rewriting files by hand. An [open table format](https://tecyfy.com/concepts/open-table-format) adds a layer of metadata on top of the files that fixes this: every change is a transaction, readers see one consistent version, and old versions can be read back. Delta Lake and Apache Iceberg are the 2 formats most [lakehouses](https://tecyfy.com/concepts/lakehouse) are built on, and both do the basics well. Where they differ is in how they record changes, who can write to them and how much they leave to you.

## Delta Lake: a log of commits beside the data

A Delta table is a folder. Next to the Parquet files sits `_delta_log`, and each write adds the next numbered commit to it: which files were added, which were removed and what the schema is. To read the table, an engine replays the log, helped by checkpoints written every so often so it does not start from commit 0. When 2 writers clash, they meet optimistic concurrency: each checks at commit time that nothing it read has changed, and one of them retries.

Because the version is a number, time travel is plain: `VERSION AS OF 3`. Deletes and updates rewrite the Parquet files that hold the changed rows, unless deletion vectors are on, in which case the rows are marked as removed and the files stay. Layout is handled by partition columns, by generated columns that derive a partition value from another column, or by liquid clustering, whose clustering columns can be changed later without rewriting what is already there.

## Iceberg: snapshots that a catalog points to

An Iceberg table is a tree. A metadata file lists the table's snapshots; each snapshot points to manifest lists, then manifests, then the data files, with statistics at each level so a query can skip files it does not need. A catalog holds one thing per table: the address of the current metadata file. A commit writes a new metadata file and swaps the catalog's pointer in one atomic step, and a writer that loses the race retries on top of the new state.

That catalog is the part people coming from Delta most often miss. You cannot point Iceberg at a folder and query it in production; you configure a catalog, such as a REST catalog, a Hive metastore or AWS Glue, and every engine that shares the table uses the same one. In return, Iceberg gives a few things Delta does not: hidden partitioning, where the table derives `day(ts)` itself and queries filter on `ts` with no partition column in sight, partition schemes that change in place, and branches and tags, named pointers to snapshots for audits or for testing a write before readers see it.

## Side by side

| | Delta Lake | Apache Iceberg |
| --- | --- | --- |
| **How it records changes** | A transaction log beside the data, one numbered commit per write, with checkpoints so readers need not replay every commit. | A tree of metadata files. Each write makes a new snapshot, and the catalog swaps its pointer to the new metadata file in one atomic step. |
| **What finds the table** | Its folder. A metastore such as Unity Catalog can name it, but the log in the folder is the truth. | A catalog, such as a REST catalog, Hive metastore, AWS Glue or a folder, which holds the address of the current metadata file. |
| **Time travel** | VERSION AS OF a version number, or TIMESTAMP AS OF a time. Up to 30 days of history by default, less once VACUUM has removed old files. | VERSION AS OF a snapshot id, or a branch or tag name, or TIMESTAMP AS OF a time, while the snapshot has not been expired. |
| **Updates and deletes** | Rewrites the files that hold changed rows, or with deletion vectors marks rows as removed and leaves the file in place. | Copy-on-write by default, which rewrites files. Merge-on-read writes delete files instead, and format version 3 adds deletion vectors. |
| **Partitions** | Partition columns, generated columns to derive them, or liquid clustering, whose columns can change without rewriting the data. | Hidden partitioning from a column and a transform such as day(ts), so queries need no partition filter, and the scheme can change in place. |
| **Upkeep** | OPTIMIZE to compact small files, and VACUUM, which you run yourself, to delete files older than 7 days that no version needs. | rewrite_data_files to compact, expire_snapshots to drop snapshots older than 5 days by default, and remove_orphan_files. |
| **Branches and tags** | No. History is one line of versions. | Yes. Named branches and tags point to snapshots and keep their own retention, for audits or testing a write before it goes live. |
| **Engines that write it** | Spark and Databricks first, with connectors for Flink, Trino, Hive and Rust (delta-rs). | Spark, Flink, Trino, Presto, Hive, Dremio, Snowflake, Amazon Athena, EMR and Redshift, Google BigQuery and others. |
| **Reading it as the other format** | UniForm writes Iceberg metadata after each commit, so Iceberg engines can read the table. They must not write to it. | Databricks manages Iceberg tables in Unity Catalog, and Snowflake manages them in its own catalog or reads them from another. |

## The same job on each

The same job on each format: create a customers table, upsert the day's changes with MERGE, read the table as it was before the merge, add a column, then compact and clean up old files. Each file runs on a laptop with PySpark; both were run first and print the same final table.

### Delta Lake: `customers_delta.py`

```python
from delta import configure_spark_with_delta_pip
from pyspark.sql import SparkSession

builder = (
    SparkSession.builder.appName("customers")
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
)
spark = configure_spark_with_delta_pip(builder).getOrCreate()

spark.sql("CREATE TABLE customers (id INT, email STRING, city STRING) USING DELTA")
spark.sql("INSERT INTO customers VALUES (1, 'ana@example.com', 'Lisbon'), (2, 'raj@example.com', 'Pune')")

# The day's changes: customer 2 moved, customer 3 is new.
spark.createDataFrame(
    [(2, "raj@example.com", "Mumbai"), (3, "li@example.com", "Shanghai")], "id INT, email STRING, city STRING"
).createOrReplaceTempView("changes")
spark.sql("""
    MERGE INTO customers AS t USING changes AS s ON t.id = s.id
    WHEN MATCHED THEN UPDATE SET *
    WHEN NOT MATCHED THEN INSERT *
""")

# Every write is a numbered version: read the table as it was before the merge.
spark.sql("SELECT * FROM customers VERSION AS OF 1 ORDER BY id").show()
spark.sql("DESCRIBE HISTORY customers").select("version", "operation").show()

# A new column, then the upkeep: compact small files, delete files older than the retention period.
spark.sql("ALTER TABLE customers ADD COLUMNS (plan STRING)")
spark.sql("OPTIMIZE customers")
spark.sql("VACUUM customers")
spark.sql("SELECT * FROM customers ORDER BY id").show()
```

The table is a folder with a _delta_log inside. Versions are numbered from 0, so version 1 is the table after the first insert. pip install delta-spark brings a matching Spark; configure_spark_with_delta_pip adds the Delta package.

### Apache Iceberg: `customers_iceberg.py`

```python
from pyspark.sql import SparkSession

spark = (
    SparkSession.builder.appName("customers")
    .config("spark.jars.packages", "org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.12.0")
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
    # A catalog named "local" that keeps its tables in a folder; in production, a REST, Glue or Hive catalog.
    .config("spark.sql.catalog.local", "org.apache.iceberg.spark.SparkCatalog")
    .config("spark.sql.catalog.local.type", "hadoop")
    .config("spark.sql.catalog.local.warehouse", "warehouse")
    .getOrCreate()
)

spark.sql("CREATE TABLE local.db.customers (id INT, email STRING, city STRING) USING iceberg")
spark.sql("INSERT INTO local.db.customers VALUES (1, 'ana@example.com', 'Lisbon'), (2, 'raj@example.com', 'Pune')")

# The day's changes: customer 2 moved, customer 3 is new.
spark.createDataFrame(
    [(2, "raj@example.com", "Mumbai"), (3, "li@example.com", "Shanghai")], "id INT, email STRING, city STRING"
).createOrReplaceTempView("changes")
spark.sql("""
    MERGE INTO local.db.customers AS t USING changes AS s ON t.id = s.id
    WHEN MATCHED THEN UPDATE SET *
    WHEN NOT MATCHED THEN INSERT *
""")

# Every write is a snapshot with its own id: read the table as it was before the merge.
first = spark.sql("SELECT snapshot_id FROM local.db.customers.snapshots ORDER BY committed_at").first()[0]
spark.sql(f"SELECT * FROM local.db.customers VERSION AS OF {first} ORDER BY id").show()
spark.sql("SELECT snapshot_id, operation FROM local.db.customers.snapshots").show()

# A new column, then the upkeep: compact small files, expire old snapshots (their files go with them).
spark.sql("ALTER TABLE local.db.customers ADD COLUMNS (plan STRING)")
spark.sql("CALL local.system.rewrite_data_files(table => 'db.customers')")
spark.sql("CALL local.system.expire_snapshots(table => 'db.customers', retain_last => 1)")
spark.sql("SELECT * FROM local.db.customers ORDER BY id").show()
```

Every table lives in a catalog, here one called local that keeps tables in a folder. A snapshot id is a long random number, so the file looks the first one up in the table's snapshots metadata table. Upkeep runs as stored procedures with CALL.

The 2 files differ in 3 places: Iceberg needs a catalog in its settings, its versions are snapshot ids rather than counters, and its upkeep runs as `CALL` procedures where Delta has `OPTIMIZE` and `VACUUM` commands. The `MERGE`, the column change and the result are the same. That is the honest picture for most day-to-day work: the SQL barely changes, and the decision is about engines and catalogs, not syntax.

## Which to choose

- **My data platform is Databricks:** Delta Lake. Delta is Databricks' default format, with liquid clustering, deletion vectors and predictive optimization built in. Turn on Iceberg reads if another engine needs to read the tables.
- **Several engines must write the same tables:** Apache Iceberg. Iceberg has the widest set of engines that write it, and a REST catalog lets Spark, Flink, Trino and Snowflake share one table without one vendor in the middle.
- **Snowflake and Spark on the same data:** Apache Iceberg. Snowflake creates and writes Iceberg tables in your own storage, and Spark reaches the same tables through a catalog.
- **Iceberg readers on tables Spark writes:** Delta Lake or Apache Iceberg. Either works. Delta with UniForm keeps Spark's Delta features and gives Iceberg readers read-only access; a plain Iceberg table lets every engine write too.
- **Partitions that change as the data grows:** Delta Lake or Apache Iceberg. Iceberg's partition evolution and Delta's liquid clustering both change the layout without rewriting the existing data.
- **Test a load before readers see it:** Apache Iceberg. Write to an Iceberg branch, check it, then fast-forward main to it. Delta has no branches.

## Using both

The old framing was a choice made once. Today the formats meet in 2 places, so many teams run both.

- **Delta tables read as Iceberg.** UniForm writes Iceberg metadata for a Delta table after each commit, pointing at the same Parquet files. Iceberg clients, such as BigQuery, can then read the table with no copy. It is read-only on the Iceberg side, and the metadata lands a moment after the Delta commit.
- **Iceberg tables in Databricks and Snowflake.** Databricks can manage Iceberg tables in Unity Catalog, which outside engines write through the Iceberg REST Catalog API, and Snowflake can create Iceberg tables in your own storage or read ones registered in another catalog.

## Things to watch

- **Clean-up removes history.** Delta's `VACUUM` deletes files older than 7 days that no current version needs, and Iceberg's `expire_snapshots` drops snapshots older than 5 days by default. Time travel beyond that, and streaming readers that fall behind, then fail. Set retention to the longest you need before you schedule either.
- **Small files pile up.** Frequent small writes, such as a stream committing every minute, leave many small files and slow reads. Schedule compaction with `OPTIMIZE` or `rewrite_data_files`, or let the platform do it.
- **Row-level changes have a read cost.** Deletion vectors and Iceberg's merge-on-read make updates cheap by leaving the cleanup for later, so reads do more work until the next compaction. Copy-on-write is the reverse.
- **Feature support differs by engine.** Not every engine supports every feature of a format version. Databricks does not support Iceberg branches, for example, so check the features you rely on in each engine you plan to use.

To see where a table format sits in a whole pipeline, from the app database through [Change Data Capture](https://tecyfy.com/concepts/change-data-capture) and the [medallion layers](https://tecyfy.com/concepts/medallion-architecture) to an AI answer, follow one order through [Data engineering, step by step](/data-engineering). To see what the same work costs on each platform, try the [Snowflake vs Databricks cost comparison](/data-engineering/snowflake-vs-databricks).

## Questions

### What is the difference between Delta Lake and Iceberg?

Both store a table as Parquet files plus metadata that makes changes atomic and keeps history. Delta Lake records changes in a log of numbered commits in the table's folder; Iceberg records them as snapshots in a tree of metadata files, with a catalog pointing to the current one. Iceberg adds hidden partitioning and branches, and more engines write it. Delta is the default on Databricks, with liquid clustering and deletion vectors.

### Is Iceberg better than Delta Lake?

Neither is better for every team. Iceberg fits when several engines, such as Spark, Trino, Flink and Snowflake, must write the same tables. Delta Lake fits when Spark or Databricks does most of the work. Both give transactions, time travel, MERGE and schema changes on Parquet files.

### Can Databricks use Iceberg tables?

Yes, in 2 ways. Iceberg reads, built on UniForm, write Iceberg metadata for Delta tables so Iceberg clients can read them, from Databricks Runtime 14.3 LTS. Managed Iceberg tables are Iceberg tables in Unity Catalog, which outside engines can write through the Iceberg REST Catalog API and whose compaction and snapshot expiry Unity Catalog runs.

### Can Snowflake read Delta Lake tables?

Yes, as an Iceberg table. Snowflake's Delta Direct creates an Iceberg table in Snowflake from a Delta table's files in object storage, and reads it there; Snowflake does not manage that table's files. A Delta table with UniForm turned on can also be read through its Iceberg metadata.

### What is UniForm in Delta Lake?

Universal Format writes Iceberg (or Hudi) metadata for a Delta table after each commit, using the same Parquet files, so Iceberg clients can read the table with no copy. It is read-only from the Iceberg side, and the metadata is written asynchronously, a moment after the Delta commit.

### Do Delta Lake and Iceberg store data the same way?

The data files are the same kind, Parquet, which is why UniForm can give one set of files 2 sets of metadata. They differ in the metadata - how changes are recorded, how partitions are described and how a reader finds the current version.

### Why does VACUUM or expire_snapshots break time travel?

Time travel reads old files. Delta's VACUUM deletes files no longer needed by versions inside the retention period, 7 days by default, and Iceberg's expire_snapshots drops snapshots older than 5 days by default and the files only they used. After either, earlier versions can no longer be read. Keep the retention as long as your oldest time-travel query or streaming reader needs.

## Sources

- [Universal Format (UniForm)](https://docs.delta.io/latest/delta-uniform.html)
- [Table batch reads and writes](https://docs.delta.io/latest/delta-batch.html)
- [Table utility commands](https://docs.delta.io/latest/delta-utility.html)
- [Table deletes, updates, and merges](https://docs.delta.io/latest/delta-update.html)
- [Optimizations](https://docs.delta.io/latest/optimizations-oss.html)
- [What are deletion vectors?](https://docs.delta.io/latest/delta-deletion-vectors.html)
- [Use liquid clustering for Delta tables](https://docs.delta.io/latest/delta-clustering.html)
- [Concurrency control](https://docs.delta.io/latest/concurrency-control.html)
- [Quickstart](https://docs.delta.io/latest/quick-start.html)
- [Integrations](https://delta.io/integrations)
- [Spark getting started](https://iceberg.apache.org/docs/latest/spark-getting-started/)
- [Spark configuration](https://iceberg.apache.org/docs/latest/spark-configuration/)
- [Spark writes](https://iceberg.apache.org/docs/latest/spark-writes/)
- [Spark queries](https://iceberg.apache.org/docs/latest/spark-queries/)
- [Spark procedures](https://iceberg.apache.org/docs/latest/spark-procedures/)
- [Configuration](https://iceberg.apache.org/docs/latest/configuration/)
- [Evolution](https://iceberg.apache.org/docs/latest/evolution/)
- [Partitioning](https://iceberg.apache.org/docs/latest/partitioning/)
- [Branching and tagging](https://iceberg.apache.org/docs/latest/branching/)
- [Reliability](https://iceberg.apache.org/docs/latest/reliability/)
- [Iceberg table spec](https://iceberg.apache.org/spec/)
- [Vendors and integrations](https://iceberg.apache.org/vendors/)
- [Apache Iceberg on Databricks](https://docs.databricks.com/aws/en/iceberg/)
- [Read Delta Lake tables with Iceberg clients](https://docs.databricks.com/aws/en/delta/uniform)
- [Apache Iceberg tables](https://docs.snowflake.com/en/user-guide/tables-iceberg)