# Delta Lake tables: the transaction log, time travel and RESTORE A Delta Lake table is a folder of Parquet files plus a transaction log, _delta_log, that lists which files make up each version. A write adds data files first and then commits 1 JSON file naming the files it added and removed, so readers see the whole change or none of it. DESCRIBE HISTORY lists the commits, VERSION AS OF reads an earlier version, and RESTORE makes it current again, for as long as its files have not been vacuumed. - Page: https://tecyfy.com/data-engineering/course/delta-lake-and-iceberg - Course: Data engineering course (https://tecyfy.com/data-engineering/course), Module 4: Tables: Parquet, Delta Lake and Iceberg, The transaction log and time travel - Parts of this module: The transaction log and time travel (https://tecyfy.com/data-engineering/course/delta-lake-and-iceberg); MERGE (https://tecyfy.com/data-engineering/course/delta-lake-and-iceberg/merge); Schema enforcement and evolution (https://tecyfy.com/data-engineering/course/delta-lake-and-iceberg/schema); OPTIMIZE, clustering and VACUUM (https://tecyfy.com/data-engineering/course/delta-lake-and-iceberg/optimize-and-vacuum); Iceberg and UniForm (https://tecyfy.com/data-engineering/course/delta-lake-and-iceberg/iceberg-and-uniform); Build it: an incremental load (https://tecyfy.com/data-engineering/course/delta-lake-and-iceberg/build-it) At the end of this part you can: - Say what a Delta table adds to a folder of Parquet files, and why a failed write cannot half-replace it - Read a table's transaction log and its history - Read an earlier version with time travel and put it back with RESTORE [Module 2](/data-engineering/course/pyspark-dataframes/build-it) wrote the shop's orders as a folder of Parquet files. A folder is fine until something goes wrong while it is being rewritten, or until 1 order changes and you need to update it. This module turns the folder into a table, first with Delta Lake, the format Databricks uses for every table, and then looks at Iceberg. ## Set up Delta Lake on your computer On Databricks there is nothing to install: every table is a Delta table unless you ask for another format. On your computer, add the `delta-spark` package to the PySpark you installed in [module 0](/data-engineering/course/start-here). This module was run on PySpark 4.2.0 with delta-spark 4.4.1: ```bash pip install delta-spark==4.4.1 ``` Save this as `delta_session.py`. Every script in the module starts its session with it: ```python """A local Spark session with Delta Lake. On Databricks, use the spark it gives you instead.""" from delta import configure_spark_with_delta_pip from pyspark.sql import SparkSession def delta_spark(app: str = "tables") -> SparkSession: builder = ( SparkSession.builder.master("local[4]").appName(app) .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") .enableHiveSupport() ) spark = configure_spark_with_delta_pip(builder).getOrCreate() spark.sparkContext.setLogLevel("ERROR") return spark ``` The 2 settings plug Delta into Spark's SQL and catalog. `enableHiveSupport()` keeps table names between runs, in a `metastore_db` folder, and the tables themselves go in `spark-warehouse`. The first run downloads Delta's Java files, so it needs the internet once. ## What a failed rewrite leaves behind Here are the 4 current orders from module 2, written twice: once as Parquet, once as a Delta table. Then each is rewritten by a job that fails halfway, on a bad row. Save this as `orders_table.py`: ```python import os from pyspark.sql import functions as F from delta_session import delta_spark spark = delta_spark() orders = spark.createDataFrame( [ (1001, "C01", "shipped", 25.50, "2026-10-02 08:00:00"), (1002, "C02", "cancelled", 12.00, "2026-10-03 09:00:00"), (1004, "C01", "placed", 40.00, "2026-10-02 12:45:00"), (1005, "C09", "placed", 8.75, "2026-10-03 14:10:00"), ], "order_id BIGINT, customer_id STRING, status STRING, amount DOUBLE, updated_at STRING", ).select( "order_id", "customer_id", "status", F.col("amount").cast("decimal(10,2)").alias("amount"), F.col("updated_at").cast("timestamp").alias("updated_at"), ) # A rewrite that fails halfway: 1 bad row stops the job. broken = orders.withColumn( "amount", F.when(F.col("order_id") == 1005, F.raise_error(F.lit("bad row"))).otherwise(F.col("amount")) ) orders.write.mode("overwrite").parquet("lake/orders_parquet") try: broken.write.mode("overwrite").parquet("lake/orders_parquet") except Exception as error: print("Parquet rewrite failed:", type(error).__name__) print("Parquet files left:", os.listdir("lake/orders_parquet")) orders.write.format("delta").mode("overwrite").saveAsTable("orders") try: broken.write.format("delta").mode("overwrite").saveAsTable("orders") except Exception as error: print("Delta rewrite failed:", type(error).__name__) print("Delta rows:", spark.table("orders").count()) ``` ```text Parquet rewrite failed: SparkRuntimeException Parquet files left: [] Delta rewrite failed: SparkRuntimeException Delta rows: 4 ``` The Parquet writer deleted the old files when the overwrite began, so the failed job left an empty folder, and every report that reads it fails until someone reruns the job. The Delta table still has its 4 rows. ## The transaction log The difference is a folder called `_delta_log` inside the table. Save this as `log.py`: ```python import json, os from delta_session import delta_spark spark = delta_spark() location = spark.sql("DESCRIBE DETAIL orders").first().location.removeprefix("file:") log = os.path.join(location, "_delta_log") print(sorted(name for name in os.listdir(log) if name.endswith(".json"))) with open(os.path.join(log, "00000000000000000000.json")) as commit: for line in commit: action, body = next(iter(json.loads(line).items())) print(action, body.get("path", "")) ``` ```text ['00000000000000000000.json'] commitInfo metaData protocol add part-00000-ea515a97-fdd1-4a96-ad79-b4352a6351d1-c000.snappy.parquet add part-00001-b958d1c4-4e2c-4366-acb4-9972ca9a7415-c000.snappy.parquet add part-00002-d2302cad-d0c1-4b6d-a26f-f5697f310047-c000.snappy.parquet add part-00003-98a8b7b2-961d-4c50-8320-5d04c5fd3be5-c000.snappy.parquet ``` Each commit is 1 JSON file, numbered from 0. This one says what ran (`commitInfo`), the schema (`metaData`), the reader and writer versions it needs (`protocol`), and 1 `add` for each Parquet file. A write puts its data files in the folder first and commits last, by creating the next numbered file. The failed rewrite wrote files too, but never got to its commit, so it has no number and no reader looks at its files. That is what a transaction means here: the whole change or none of it. The table's folder holds the 4 files version 0 lists, the 4 the failed job left, and the log. A reader never lists the folder; it reads the log. ## History and time travel Delta never edits a Parquet file. An update writes a new file with the changed rows and commits a `remove` for the old one. Save this as `history.py`: ```python from pyspark.sql import functions as F from delta_session import delta_spark spark = delta_spark() spark.sql("UPDATE orders SET status = 'shipped', updated_at = '2026-10-04 08:30:00' WHERE order_id = 1004") spark.sql("DELETE FROM orders WHERE status = 'cancelled'") metrics = F.col("operationMetrics") spark.sql("DESCRIBE HISTORY orders").select( "version", "operation", F.coalesce(metrics["numAddedFiles"], metrics["numFiles"]).alias("files_added"), metrics["numRemovedFiles"].alias("files_removed"), ).show() spark.sql("SELECT order_id, status FROM orders VERSION AS OF 0 ORDER BY order_id").show() ``` ```text +-------+--------------------+-----------+-------------+ |version| operation|files_added|files_removed| +-------+--------------------+-----------+-------------+ | 2| DELETE| 0| 1| | 1| UPDATE| 1| 1| | 0|CREATE OR REPLACE...| 4| 0| +-------+--------------------+-----------+-------------+ +--------+---------+ |order_id| status| +--------+---------+ | 1001| shipped| | 1002|cancelled| | 1004| placed| | 1005| placed| +--------+---------+ ``` Each order sits in a file of its own here, so the update of 1004 replaced 1 file and the delete removed 1. Version 0 still reads as it was, because its files are still on disk: `VERSION AS OF` replays the log only up to that commit. `TIMESTAMP AS OF` with a date and time picks the version that was current then. (An interactive demo goes here on the page: https://tecyfy.com/data-engineering/course/delta-lake-and-iceberg) ## Put a version back with RESTORE To undo the update and the delete, restore version 0. Save this as `restore.py`: ```python from delta_session import delta_spark spark = delta_spark() spark.sql("RESTORE TABLE orders TO VERSION AS OF 0").select( "num_removed_files", "num_restored_files" ).show() ``` ```text +-----------------+------------------+ |num_removed_files|num_restored_files| +-----------------+------------------+ | 1| 2| +-----------------+------------------+ ``` RESTORE is a new commit, version 3, that removes the shipped 1004 file and adds back the 2 files version 0 had. The update and the delete stay in the history, so you can undo the restore too. On Databricks, the same SQL works on a table in your catalog and schema, and Databricks manages where its files live, so you read the log through `DESCRIBE HISTORY` rather than its JSON files. History is kept for 30 days by default and the files of old versions for 7, until VACUUM removes them, which [part 4](/data-engineering/course/delta-lake-and-iceberg/optimize-and-vacuum) covers. Databricks says not to use history as a backup. [Part 2](/data-engineering/course/delta-lake-and-iceberg/merge) updates the table from a batch of changes with MERGE. ## Check yourself 1. A nightly job rewrites the orders table and fails halfway through. With a folder of Parquet files, the folder is left empty. Why does the same failure leave a Delta table with all its rows? A. Delta keeps a backup copy of every file it replaces B. Delta writes the new data files first and only then commits a log entry that swaps them in, so a job that fails before its commit changes no version of the table C. Delta locks the table so the job cannot start until the old files are copied D. Delta writes all its files in 1 task, which cannot fail halfway 2. You ran DELETE on the orders table by mistake 2 hours ago. Since then, 1 more MERGE has run. What does RESTORE TABLE orders TO VERSION AS OF n, with n the version before the DELETE, do? A. It removes the DELETE and the MERGE from the history, as if they never happened B. It fails, because a later commit exists C. It makes a new commit whose files are those of version n, so the deleted rows come back and the MERGE's changes are undone too; the history keeps every commit D. It brings back only the deleted rows and keeps the MERGE's changes ### Answers 1. B. The Parquet writer deletes the old files when an overwrite starts, so a failure leaves nothing. A Delta write puts its new files beside the old ones and then commits 1 JSON file to _delta_log listing what it added and removed. Readers only follow the log, so files from a job that never committed are invisible, and the old version is still the current one. 2. C. RESTORE is a commit like any other. It adds and removes files until the table matches version n, so everything after n is undone, including the MERGE. The DELETE and the MERGE stay in the history, and you can still read or restore them. To keep the MERGE, read the deleted rows from version n with VERSION AS OF and insert them instead. ## Questions ### What is the Delta Lake transaction log? It is the _delta_log folder inside a Delta table, with 1 JSON file per commit, numbered from 00000000000000000000.json. Each commit lists actions, chiefly add and remove for data files, and the table at version n is every file added and not removed in commits 0 to n. Checkpoint files summarise the log so readers do not replay every commit. ### How do I see the history of a Delta table? Run DESCRIBE HISTORY with the table's name. It returns 1 row per commit, newest first, with the version, time, operation, its parameters and metrics such as the files and rows it added and removed. History is kept for 30 days by default, the table property delta.logRetentionDuration. ### How do I query an older version of a Delta table? Add VERSION AS OF and a version number, or TIMESTAMP AS OF and a time, after the table name, as in SELECT * FROM orders VERSION AS OF 3. From PySpark, spark.read.option("versionAsOf", 3).table("orders") does the same. It works while the version's data files are still there; VACUUM deletes them after 7 days by default. ### Is time travel a backup? No. Databricks says not to use table history as a long-term backup. VACUUM deletes the data files old versions need, history is cut after 30 days by default, and dropping the table takes its history with it. Use time travel to undo recent mistakes and to reproduce what a job read, and keep backups by other means. ## Sources - [Quickstart](https://docs.delta.io/latest/quick-start.html) - [Delta transaction log protocol](https://github.com/delta-io/delta/blob/master/PROTOCOL.md) - [Table batch reads and writes](https://docs.delta.io/latest/delta-batch.html) - [Table utility commands](https://docs.delta.io/latest/delta-utility.html) - [What is Delta Lake in Databricks?](https://docs.databricks.com/aws/en/delta/) - [Work with table history](https://docs.databricks.com/aws/en/delta/history)