What is an open table format?
A published way of storing a table as plain data files plus a log of changes, so many engines can read and safely write it. Delta Lake and Apache Iceberg are the best known.
intermediate · Also called Delta Lake, Apache Iceberg, Table format
A folder of Parquet files in cloud storage is not a table. It has no idea which files make up the current version. If a job is halfway through writing 20 new files when you read the folder, you get 11 of them. If 2 jobs write at once, the result is whatever happened to land. And there is no way to say "show me the table as it was this morning".
An open table format fixes this with a small idea: keep the data files as they are, and add a log next to them that says which files make up the table.
The log is the table
Each change to the table writes new data files, then records in the log which files were added and which no longer count. The table is whatever the log says it is.
The key step is how that record is written. In Apache Iceberg every change creates a new metadata file and replaces the old one in a single atomic swap. In Delta Lake every write adds a new entry to a transaction log. Either way, a change has fully happened or has not happened at all, so a reader always sees a complete, committed version, never a half-written one.
What you get from it
- Versions. Each write makes a new version, and older ones can still be read. Iceberg calls them snapshots. In Databricks SQL a query can name one:
SELECT * FROM orders VERSION AS OF 41. That is how you rerun yesterday's report on yesterday's data. - Safe changes to rows. Updates and deletes without rewriting the whole table. Iceberg's version 2 format added row-level deletes; version 3 added new types and row lineage.
- Schema rules. A Delta table on Databricks rejects a write whose columns do not exist in the table, unless you turn on schema evolution for that write. Iceberg tracks columns by an id rather than by name, so a column can be renamed without breaking files already written. Both matter when a source changes shape, which is schema driftSchema DriftWhen the shape of incoming data changes without warning, such as a renamed or new column, and breaks or silently corrupts the pipelines that read it..
- Part of the table
- Read by the query
- Still in storage, not part of it
Commit 4 in the log
DELETE 1003
− part-b.parquet
SELECT * FROM orders- 1001€42.50
- 1002€24.99
The table now is version 4: 2 orders from 1 file. The files that left the table are still in storage, which is why older versions can still be read. Pick one.
The orders and file names are made up; real data files have long random names, and real logs hold more detail. The query is Databricks SQL on a Delta table; Iceberg keeps versions the same way and calls them snapshots.
Why "open"
The data is in an open file format, and the log follows a published specification. So any engine that implements the spec can read the same table: Spark, a SQL warehouse, a Python library. You are not tied to the product that wrote it.
Delta Lake and Iceberg are the 2 best known. Delta is the default on Databricks, where every table is a Delta table unless you say otherwise. They do the same job in different ways, and Databricks can convert Iceberg tables to Delta.
Why it matters for AI
An open table format is what turns a data lakeData LakeCheap storage that holds data of any kind in its raw form, usually as files in cloud object storage, to be cleaned and structured later. into a LakehouseLakehouseOne store for all of an organisation's data: cheap files in cloud storage, as in a data lake, with the tables, transactions and fast queries of a data warehouse on top.. For AI work it adds one thing a folder of files never had: you can say exactly which version of the data a model was trained on, or an index was built from, and read that version again later. When an answer looks wrong, "what did the data say then?" has an answer.
Sources