What is the difference between Delta Lake and Apache Iceberg?

Both store data as Parquet files and add a metadata layer for transactions, time travel and schema changes. Delta keeps an ordered log of commits in the table's folder. Iceberg keeps a tree of metadata files, snapshots, manifest lists and manifests, and a catalog holds the address of the current one. Delta fits best where Spark and Databricks do most of the work; Iceberg has the widest choice of engines that write it.

What is UniForm in Delta Lake?

UniForm, which Databricks now calls Iceberg reads, makes a Delta table also write Iceberg metadata after each commit, pointing to the same Parquet files, so Iceberg engines can read the table without a copy. Iceberg clients can only read it. On Databricks the table must be in Unity Catalog, have column mapping on and deletion vectors off.

Can Databricks create Iceberg tables?

Yes. CREATE TABLE with USING iceberg in a Unity Catalog schema makes a managed Iceberg table, and a DataFrame can be saved with format("iceberg"). Unity Catalog runs snapshot expiration and file compaction on it, and external Iceberg engines can read and write it through Unity Catalog's Iceberg REST Catalog API. Without USING iceberg, Databricks creates a Delta table.

Apache Iceberg and UniForm: when a Delta table must also be an Iceberg table

Apache Iceberg is the other open table format: Parquet files plus metadata that adds transactions, time travel and schema changes, like Delta, but tracked through snapshots that a catalog points to, and written by the widest range of engines. On Databricks, keep tables in Delta unless another engine needs them. If it only reads, turn on Iceberg reads (UniForm) and the Delta table also publishes Iceberg metadata. If it must write too, create a managed Iceberg table in Unity Catalog with USING iceberg.

Module 4: Tables: Parquet, Delta Lake and Iceberg

At the end of this part you can:

Say what Apache Iceberg is and how its metadata differs from Delta's log
Choose between Delta, Delta with Iceberg reads and a managed Iceberg table on Databricks
Turn on Iceberg reads for a Delta table, and know what it rules out

Take this module into your AI

ChatGPT Claude

Delta Lake is not the only open table format. Apache Iceberg, an Apache project, solves the same problem the same way: Parquet files, plus metadata that turns a folder into a table with transactions, time travel and schema changes. You will meet it when a table has to be shared with an engine other than Spark.

How Iceberg differs

The difference is in how each finds the current version. Delta keeps a log of numbered commits in the table's folder, as part 1 showed, and the newest commit is the current version. Iceberg keeps a tree of metadata files: each snapshot points to a list of manifests, each manifest lists data files with the value ranges of their columns, and a catalog holds the address of the current metadata file. A write commits by moving that pointer in 1 step.

For the work in this module, the 2 behave alike. Both support MERGE, time travel and schema changes, both skip files by their value ranges, and both need compaction and a clean-up of old files. Iceberg adds hidden partitioning, where a partition is derived from a column with a transform such as day(ordered_at) and queries need no partition filter, and named branches and tags, which Delta does not have. Its strength is reach: Spark, Flink, Trino, Snowflake, Amazon Athena, Google BigQuery and others work with it, through a shared catalog.

The comparison of Delta Lake and Iceberg goes through every difference, with the same upsert, time travel and clean-up run on each.

ComparisonDelta Lake vs Apache IcebergDelta Lake and Apache Iceberg are both open table formats - Parquet files plus a metadata layer that adds transactions, time travel, MERGE and schema changes to…

3 choices on Databricks

On Databricks, a table is Delta unless you ask for something else. Iceberg matters only when another engine needs the table, and what it needs decides the choice.

Nobody outside Databricks reads it. Keep it Delta. Liquid clustering, deletion vectors and predictive optimization are built around it, and everything in this module applies.

Another engine reads it. Keep it Delta and turn on Iceberg reads, which the Delta project calls UniForm. After each commit, Databricks also writes Iceberg metadata that points to the same Parquet files, so an Iceberg engine reads the table through Unity Catalog with no copy. On the orders table, that is 1 statement on Databricks:

ALTER TABLE orders SET TBLPROPERTIES (
  'delta.columnMapping.mode' = 'name',
  'delta.enableIcebergCompatV2' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
)

It has conditions. The table must be in Unity Catalog and written by name, not by path. Iceberg clients can only read it, never write. Deletion vectors must be off, so DELETE, UPDATE and MERGE go back to rewriting files. The Iceberg metadata is written after the Delta commit, not with it, so an Iceberg reader can be a moment behind.

Another engine writes it too. Make it a managed Iceberg table: CREATE TABLE ... USING iceberg in a Unity Catalog schema, or .format("iceberg") before saveAsTable. External engines then read and write it through Unity Catalog's Iceberg REST Catalog API, and Unity Catalog expires old snapshots and compacts files. This is the choice when, for example, Flink and Databricks both load the same table.

Snowflake also creates and manages Iceberg tables in your own storage, the other road to 1 table that both platforms use.

Which this course uses

The course stays with Delta tables: they are the default on Databricks and on Free Edition, and every later module, from Unity Catalog to pipelines, builds on them. If your tables ever need an outside reader, the change is the 1 statement above, not a rewrite.

Part 6 puts the module together: a load of the day's orders that can run twice without harm.

Check yourself

2 questions, each in a new situation

1. Analysts query the shop's tables from Snowflake, read only. The pipeline writes them on Databricks with MERGE every hour. What is the simplest setup?
Not quite. Iceberg reads, also called UniForm, keep the table in Delta for the pipeline and add Iceberg metadata that points to the same Parquet files, so nothing is copied. Iceberg clients may only read such a table, which fits analysts who query it. The table must be registered in Unity Catalog, and deletion vectors must be off.
2. A team wants Apache Flink to write into the same table the Databricks pipeline writes. Which choice allows that?
Not quite. Iceberg reads give other engines read-only access; Databricks says writes from Iceberg clients are not supported. A managed Iceberg table is an Iceberg table first, and Unity Catalog's Iceberg REST Catalog API supports reads and writes from external Iceberg engines, while Unity Catalog runs its upkeep.

Questions

What is the difference between Delta Lake and Apache Iceberg?: Both store data as Parquet files and add a metadata layer for transactions, time travel and schema changes. Delta keeps an ordered log of commits in the table's folder. Iceberg keeps a tree of metadata files, snapshots, manifest lists and manifests, and a catalog holds the address of the current one. Delta fits best where Spark and Databricks do most of the work; Iceberg has the widest choice of engines that write it.
What is UniForm in Delta Lake?: UniForm, which Databricks now calls Iceberg reads, makes a Delta table also write Iceberg metadata after each commit, pointing to the same Parquet files, so Iceberg engines can read the table without a copy. Iceberg clients can only read it. On Databricks the table must be in Unity Catalog, have column mapping on and deletion vectors off.
Can Databricks create Iceberg tables?: Yes. CREATE TABLE with USING iceberg in a Unity Catalog schema makes a managed Iceberg table, and a DataFrame can be saved with format("iceberg"). Unity Catalog runs snapshot expiration and file compaction on it, and external Iceberg engines can read and write it through Unity Catalog's Iceberg REST Catalog API. Without USING iceberg, Databricks creates a Delta table.

Sources