Deletion Vectors in Delta Lake and Databricks: Faster DELETE, UPDATE and MERGE
How deletion vectors let Delta Lake mark rows as deleted instead of rewriting whole files, what that changed in a measured 100-row DELETE on 20 million rows, how to turn them on, which readers can still read the table, and how to apply the changes for good.
Tecyfy · Published · Updated · 7 min read
Deletion vectors let a Delta table delete or change a few rows without rewriting the files those rows are in. Instead of copying a whole Parquet file minus one row, Delta writes a small bitmap that says which rows of the file no longer count, and every reader skips those rows. DELETE, UPDATE and MERGE all use them. The changes are applied to the data files later, when a file is rewritten: sometimes by OPTIMIZE, and for every marked file on purpose with REORG TABLE ... APPLY (PURGE).
(The feature is sometimes searched for as "vector deletion". It is unrelated to vector databases and embeddings.)
The problem they solve
A Delta table never changes a Parquet file in place. Without deletion vectors, deleting one row means reading the whole file it lives in, writing a new file with every other row, and recording the swap in the transaction log. This is called copy-on-write. A 100-row correction spread across a large table rewrites most of the table.
With deletion vectors, Delta instead records, for each touched file, the positions of the deleted rows. These are stored as a compressed bitmap (a RoaringBitmap), either inside the transaction log or in a small file it points to. An UPDATE marks the old version of a row as deleted and writes only the new version to a new file. This is often called merge-on-read: writing is cheap, and readers do a little extra work to skip the marked rows.
What it changed in a real run
We ran the same DELETE with and without deletion vectors on Apache Spark 4.2.0 and Delta Lake 4.4.1, on one Mac with 4 cores (local[4]), 3 runs each. The table had 20 million rows in 8 Parquet files (390 MB), and the DELETE removed 100 rows spread across every file.
| Without deletion vectors | With deletion vectors | |
|---|---|---|
| Time (3 runs) | 4.7 to 6.7 s | 1.6 to 2.5 s |
| Data files rewritten | 8 (all of them) | 0 |
| Bytes written | 390 MB | One 1,161-byte deletion vector file |
| Rows copied to delete 100 | 19,999,900 | 0 |
The time saved is modest here because local disk is fast and the table is small. The bytes are the real story: without deletion vectors, removing 0.0005% of the rows rewrote 100% of the data. On cloud storage, with tables in the terabytes, that rewrite is what makes small deletes slow and expensive.
The cost moves rather than disappears. Applying the changes afterwards with REORG TABLE ... APPLY (PURGE) took 9.4 s, because it does the rewrite that was skipped. The difference is that you choose when it runs, instead of paying for it on every small change.
Turning them on
Many Databricks workspaces enable deletion vectors on new tables automatically. A workspace setting does this for tables created from a SQL warehouse or Databricks Runtime 14.3 LTS and above; the default depends on the region. To set it yourself:
-- A new table
CREATE TABLE sales.orders (
order_id BIGINT,
customer_id BIGINT,
amount DECIMAL(12, 2)
)
TBLPROPERTIES ('delta.enableDeletionVectors' = true);
-- An existing table
ALTER TABLE sales.orders
SET TBLPROPERTIES ('delta.enableDeletionVectors' = true);
Check with SHOW TBLPROPERTIES sales.orders. After a change, DESCRIBE HISTORY shows numDeletionVectorsAdded in the operation's metrics; a large numCopiedRows on a small delete means they were not used.
Check your readers first
Enabling deletion vectors upgrades the table's protocol. Any client that does not understand deletion vectors can no longer read the table.
| Client | Can read | Can write with them |
|---|---|---|
| Databricks with Photon | Runtime 12.2 LTS+ | DELETE, UPDATE, MERGE from 12.2 LTS |
| Databricks without Photon | Runtime 12.2 LTS+ | DELETE from 12.2 LTS, UPDATE from 14.1, MERGE from 14.3 LTS |
| Open-source Spark with Delta Lake | Delta 2.3.0+ | DELETE from 2.4.0, UPDATE from 3.0.0, MERGE from 3.1.0 |
| Delta Sharing recipients | Runtime 14.1+, or delta-sharing-spark 3.1+ | No |
Before switching on a shared table, list every engine that reads it: older Spark jobs, external query engines, a tool reading the files directly. Iceberg v2 readers cannot read them either; Iceberg v3 can.
If you need older readers back, Databricks Runtime 14.1 and above can remove the feature, which first rewrites the affected files:
ALTER TABLE sales.orders DROP FEATURE deletionVectors;
Applying the changes for good
Rows marked by a deletion vector are a soft delete: the data is still in the old Parquet file. Three things rewrite the files and apply the marks:
OPTIMIZEapplies them to the files it compacts.- Auto compaction does the same as part of writes.
REORG TABLE ... APPLY (PURGE)rewrites every file that has a deletion vector.
Compaction may leave some files untouched, so it does not guarantee every mark is applied. When a delete must be physical, for example a request to erase someone's personal data, purge and then vacuum:
REORG TABLE customers APPLY (PURGE);
-- then, once the retention period has passed since the purge:
VACUUM customers;
The purge writes new files without the rows; only VACUUM deletes the old files that still contain them. Our VACUUM LITE guide covers retention. On very large tables, setting spark.databricks.delta.reorg.purgeMode to rows makes the purge faster, because the default checks every file's footer.
When they help, and when they do not
They help most when changes are small and scattered: a MERGE of yesterday's changes into a large table, privacy deletions, corrections to a few records, change data feeds applied continuously. They also allow row-level concurrency (Databricks Runtime 14.2 and above), so two writers changing different rows of the same file no longer conflict.
They help less when a change rewrites most of a file anyway, or the table is rarely changed. Each read also has to apply the bitmaps, so a table with many unapplied deletion vectors should be compacted regularly. On Unity Catalog managed tables, predictive optimization runs OPTIMIZE for you.
Tecyfy Takeaway
- Deletion vectors mark deleted rows in a small bitmap instead of rewriting the Parquet file;
DELETE,UPDATEandMERGEuse them. - In our run, a 100-row delete on 20 million rows wrote 1 KB instead of rewriting 390 MB, and ran about 3 times faster even on a fast local disk.
- They upgrade the table protocol: check that every reader supports them before turning them on.
- The data is only physically gone after
REORG TABLE ... APPLY (PURGE)and thenVACUUM; compaction applies some marks but does not guarantee all of them.
Questions
- What are deletion vectors in Delta Lake?
- Deletion vectors let a Delta table delete or change rows without rewriting the files they are in. Delta writes a small bitmap marking which rows of a file no longer count, and readers skip them. DELETE, UPDATE and MERGE all use them.
- How do I enable deletion vectors on a Delta table?
- Set the table property: ALTER TABLE name SET TBLPROPERTIES ('delta.enableDeletionVectors' = true), or put it in CREATE TABLE. Many Databricks workspaces turn them on for new tables automatically.
- Do deletion vectors break older readers?
- Yes. Turning them on upgrades the table protocol, so any client that does not support them can no longer read the table: Databricks needs Runtime 12.2 LTS or later and open-source Delta 2.3.0 or later. Check every engine that reads the table first.
- Are rows deleted with deletion vectors really gone?
- Not at first: it is a soft delete and the data stays in the old Parquet file. Run REORG TABLE name APPLY (PURGE) to rewrite the files, then VACUUM after the retention period to delete the old ones, for example for a personal data erasure request.
- When do deletion vectors help most?
- When changes are small and scattered, such as a daily MERGE into a large table, privacy deletions or a few corrections. In Tecyfy's test, a 100-row delete on 20 million rows wrote 1 KB instead of rewriting 390 MB.
Sources