Delta Lake vs Apache Iceberg: how the 2 table formats differ and which to choose

A data lake on its own is a folder of Parquet files. It has no idea which files make up the table right now, so 2 jobs writing at once can leave readers with half a change, and an update means rewriting files by hand. An open table formatOpen Table FormatA published way of storing a table as plain data files plus a log of changes, so many engines can read and safely write it. Delta Lake and Apache Iceberg are the best known. adds a layer of metadata on top of the files that fixes this: every change is a transaction, readers see one consistent version, and old versions can be read back. Delta Lake and Apache Iceberg are the 2 formats most lakehousesLakehouseOne store for all of an organisation's data: cheap files in cloud storage, as in a data lake, with the tables, transactions and fast queries of a data warehouse on top. are built on, and both do the basics well. Where they differ is in how they record changes, who can write to them and how much they leave to you.

Delta Lake: a log of commits beside the data

A Delta table is a folder. Next to the Parquet files sits _delta_log, and each write adds the next numbered commit to it: which files were added, which were removed and what the schema is. To read the table, an engine replays the log, helped by checkpoints written every so often so it does not start from commit 0. When 2 writers clash, they meet optimistic concurrency: each checks at commit time that nothing it read has changed, and one of them retries.

Because the version is a number, time travel is plain: VERSION AS OF 3. Deletes and updates rewrite the Parquet files that hold the changed rows, unless deletion vectors are on, in which case the rows are marked as removed and the files stay. Layout is handled by partition columns, by generated columns that derive a partition value from another column, or by liquid clustering, whose clustering columns can be changed later without rewriting what is already there.

Iceberg: snapshots that a catalog points to

An Iceberg table is a tree. A metadata file lists the table's snapshots; each snapshot points to manifest lists, then manifests, then the data files, with statistics at each level so a query can skip files it does not need. A catalog holds one thing per table: the address of the current metadata file. A commit writes a new metadata file and swaps the catalog's pointer in one atomic step, and a writer that loses the race retries on top of the new state.

That catalog is the part people coming from Delta most often miss. You cannot point Iceberg at a folder and query it in production; you configure a catalog, such as a REST catalog, a Hive metastore or AWS Glue, and every engine that shares the table uses the same one. In return, Iceberg gives a few things Delta does not: hidden partitioning, where the table derives day(ts) itself and queries filter on ts with no partition column in sight, partition schemes that change in place, and branches and tags, named pointers to snapshots for audits or for testing a write before readers see it.

Side by side

The same job on each

from delta import configure_spark_with_delta_pip
from pyspark.sql import SparkSession

builder = (
    SparkSession.builder.appName("customers")
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
)
spark = configure_spark_with_delta_pip(builder).getOrCreate()

spark.sql("CREATE TABLE customers (id INT, email STRING, city STRING) USING DELTA")
spark.sql("INSERT INTO customers VALUES (1, 'ana@example.com', 'Lisbon'), (2, 'raj@example.com', 'Pune')")

# The day's changes: customer 2 moved, customer 3 is new.
spark.createDataFrame(
    [(2, "raj@example.com", "Mumbai"), (3, "li@example.com", "Shanghai")], "id INT, email STRING, city STRING"
).createOrReplaceTempView("changes")
spark.sql("""
    MERGE INTO customers AS t USING changes AS s ON t.id = s.id
    WHEN MATCHED THEN UPDATE SET *
    WHEN NOT MATCHED THEN INSERT *
""")

# Every write is a numbered version: read the table as it was before the merge.
spark.sql("SELECT * FROM customers VERSION AS OF 1 ORDER BY id").show()
spark.sql("DESCRIBE HISTORY customers").select("version", "operation").show()

# A new column, then the upkeep: compact small files, delete files older than the retention period.
spark.sql("ALTER TABLE customers ADD COLUMNS (plan STRING)")
spark.sql("OPTIMIZE customers")
spark.sql("VACUUM customers")
spark.sql("SELECT * FROM customers ORDER BY id").show()

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder.appName("customers")
    .config("spark.jars.packages", "org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.12.0")
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
    # A catalog named "local" that keeps its tables in a folder; in production, a REST, Glue or Hive catalog.
    .config("spark.sql.catalog.local", "org.apache.iceberg.spark.SparkCatalog")
    .config("spark.sql.catalog.local.type", "hadoop")
    .config("spark.sql.catalog.local.warehouse", "warehouse")
    .getOrCreate()
)

spark.sql("CREATE TABLE local.db.customers (id INT, email STRING, city STRING) USING iceberg")
spark.sql("INSERT INTO local.db.customers VALUES (1, 'ana@example.com', 'Lisbon'), (2, 'raj@example.com', 'Pune')")

# The day's changes: customer 2 moved, customer 3 is new.
spark.createDataFrame(
    [(2, "raj@example.com", "Mumbai"), (3, "li@example.com", "Shanghai")], "id INT, email STRING, city STRING"
).createOrReplaceTempView("changes")
spark.sql("""
    MERGE INTO local.db.customers AS t USING changes AS s ON t.id = s.id
    WHEN MATCHED THEN UPDATE SET *
    WHEN NOT MATCHED THEN INSERT *
""")

# Every write is a snapshot with its own id: read the table as it was before the merge.
first = spark.sql("SELECT snapshot_id FROM local.db.customers.snapshots ORDER BY committed_at").first()[0]
spark.sql(f"SELECT * FROM local.db.customers VERSION AS OF {first} ORDER BY id").show()
spark.sql("SELECT snapshot_id, operation FROM local.db.customers.snapshots").show()

# A new column, then the upkeep: compact small files, expire old snapshots (their files go with them).
spark.sql("ALTER TABLE local.db.customers ADD COLUMNS (plan STRING)")
spark.sql("CALL local.system.rewrite_data_files(table => 'db.customers')")
spark.sql("CALL local.system.expire_snapshots(table => 'db.customers', retain_last => 1)")
spark.sql("SELECT * FROM local.db.customers ORDER BY id").show()

The 2 files differ in 3 places: Iceberg needs a catalog in its settings, its versions are snapshot ids rather than counters, and its upkeep runs as CALL procedures where Delta has OPTIMIZE and VACUUM commands. The MERGE, the column change and the result are the same. That is the honest picture for most day-to-day work: the SQL barely changes, and the decision is about engines and catalogs, not syntax.

Which to choose

Using both

The old framing was a choice made once. Today the formats meet in 2 places, so many teams run both.

• Delta tables read as Iceberg. UniForm writes Iceberg metadata for a Delta table after each commit, pointing at the same Parquet files. Iceberg clients, such as BigQuery, can then read the table with no copy. It is read-only on the Iceberg side, and the metadata lands a moment after the Delta commit.
• Iceberg tables in Databricks and Snowflake. Databricks can manage Iceberg tables in Unity Catalog, which outside engines write through the Iceberg REST Catalog API, and Snowflake can create Iceberg tables in your own storage or read ones registered in another catalog.

Things to watch

• Clean-up removes history. Delta's VACUUM deletes files older than 7 days that no current version needs, and Iceberg's expire_snapshots drops snapshots older than 5 days by default. Time travel beyond that, and streaming readers that fall behind, then fail. Set retention to the longest you need before you schedule either.
• Small files pile up. Frequent small writes, such as a stream committing every minute, leave many small files and slow reads. Schedule compaction with OPTIMIZE or rewrite_data_files, or let the platform do it.
• Row-level changes have a read cost. Deletion vectors and Iceberg's merge-on-read make updates cheap by leaving the cleanup for later, so reads do more work until the next compaction. Copy-on-write is the reverse.
• Feature support differs by engine. Not every engine supports every feature of a format version. Databricks does not support Iceberg branches, for example, so check the features you rely on in each engine you plan to use.

To see where a table format sits in a whole pipeline, from the app database through Change Data CaptureChange Data CaptureCopying only the rows that were added, changed or deleted in a source database, read from its change log, instead of copying the whole table again. and the medallion layersMedallion ArchitectureOrganising a lakehouse in 3 layers of rising quality: bronze for raw data, silver for cleaned and joined data, and gold for tables shaped for reports and AI. to an AI answer, follow one order through Data engineering, step by step. To see what the same work costs on each platform, try the Snowflake vs Databricks cost comparison.