Databricks variant shredding is now the default: what changes for your tables

From 18 November 2026, Databricks adds the variantShredding table feature to every new table with a VARIANT column created with CREATE TABLE on Databricks Runtime 17.3 or above. Shredding stores the fields that occur often inside a VARIANT value as their own typed columns in the Parquet files, so a query that needs 2 fields reads 2 columns instead of decoding every whole value. The catch is compatibility: shredding is a table feature that readers must understand, and Databricks warns that clusters on Runtime 15.4 can't read or write these tables at all.

If you keep JSON events, API payloads or logs in VARIANT columns, this changes how your new tables are written. It does not touch your existing tables.

What shredding stores

A VARIANT column without shredding is one binary value per row. To read raw:amount, Databricks has to read that value and walk it to find the field, for every row, whatever else the value holds. Databricks has no min and max statistics for an unshredded VARIANT either, so a filter on a field inside it can't skip files.

Shredding splits the value. Databricks' implementation follows the idea of the open Apache Parquet variant shredding specification: each shredded field gets a typed_value column of a real type (a string, an integer, a timestamp), and a binary value keeps everything that was not shredded. A value that does not fit the column's type, such as "n/a" where the column holds numbers, also stays in the binary value, so nothing is lost. Readers put the 2 parts back together, and your SQL does not change.

Switch it on and off below and pick a query to see which parts each one has to read.

Try itWhat a query reads from a VARIANT column

OffOn

Query

·Revenue from purchases·Count signups·Refund reasons

SELECT sum(raw:amount::decimal(10,2)) FROM events WHERE raw:event_type::string = 'purchase'

event_type	user_id	amount	ts	value (the rest)
signup	101	null	2026-11-02 09:14	null
purchase	102	49.9	2026-11-02 09:20	null
purchase	103	null	2026-11-02 09:31	{"amount":"n/a"}
refund	102	-49.9	2026-11-02 10:02	{"reason":"damaged"}
purchase	104	12.5	2026-11-02 10:45	null

18% read, with fallback

The spec gives the 2 gains. A query reads only the columns for the fields it asks for, and each typed column carries its own statistics, so a filter on a field like event_type can skip data. Databricks adds that the typed columns compress better than the binary blob. For statistics and data skipping on shredded fields, Databricks recommends Runtime 18.1 or above.

Which fields are shredded is not something you list. Databricks' page says it stores the commonly occurring fields as columns, so the rare field, like reason in the widget, stays in the binary value and reads the old way.

What changes on 18 November

The 2 statements left out on purpose, CREATE OR REPLACE TABLE and ALTER TABLE, are left out, in Databricks' words, to avoid changing the shredding behaviour of existing tables and workloads. A job that rebuilds its table every night with CREATE OR REPLACE keeps writing unshredded data.

Who can still read the table

This is where the change can break something. Delta tables list the features a client must understand, and variant shredding is a reader and writer feature, at reader version 3 and writer version 7. A client that does not support it can't read the table, not just the shredded fields.

Databricks says 2 things about versions, and they are worth reading together:

• The announcement calls this a breaking change for Runtime 15.4, whose clusters can't read from or write to shredded tables, and tells you to move them to 16.4 or above.
• The shredding page says Runtime 17.3 or above is required to read and write shredded VARIANT tables.

The safe reading is 17.3 for every job, notebook and SQL warehouse that touches a new VARIANT table, and 18.1 if you want the data skipping. 15.4 is the line that fails outright, and it matters because VARIANT needs Runtime 15.4 LTS or above, so teams that adopted VARIANT early may still run 15.4.

Look beyond Databricks clusters too. Anything else that reads the Delta files directly, an engine reading the table through its storage path, needs to support the feature as well. Before 18 November, list the readers of the tables your pipelines create, not only the writers.

Opt in, opt out, or rewrite

For an existing table, shredding is a property:

-- Shred future writes on an existing table
ALTER TABLE events SET TBLPROPERTIES ('delta.enableVariantShredding' = 'true');

-- Stop shredding future writes
ALTER TABLE events SET TBLPROPERTIES ('delta.enableVariantShredding' = 'false');

For an Iceberg table the property is iceberg.enableVariantShredding.

Turning the property on does not convert what is already in the table: it applies to data written afterwards. To rewrite existing data in the shredded layout:

REORG TABLE events APPLY (SHRED VARIANT);

Setting the property to false stops new shredded writes but leaves the feature on the table. To take it off completely, drop the feature, which rewrites the shredded data in place in the unshredded format and sets the property to false:

ALTER TABLE events DROP FEATURE "variantShredding";

Where shredding does not help

• Writes cost more. Databricks lists write overhead as a limitation. A table written far more than it is read may not gain.
• Arrays and maps are not shredded. Shredding applies to top-level VARIANT columns and VARIANT fields inside structs. A VARIANT stored inside an array or a map stays a binary value.
• Rare and mixed fields read as before. A field that appears in a few rows, or holds different types from row to row, ends up in the binary value often, as the reason field and the "n/a" amount do in the widget.
• VARIANT rules stay the same. A VARIANT column still can't be a partition or clustering key, or go in GROUP BY, ORDER BY or DISTINCT.

Queries do not change

Shredding is a storage layout, so the way you query VARIANT stays as it was: the colon path, a cast, or variant_get with a path and a type.

SELECT
  raw:event_type::string           AS event_type,
  variant_get(raw, '$.amount', 'decimal(10,2)') AS amount
FROM events
WHERE raw:event_type::string = 'purchase';

Field names in these paths are case sensitive, as before. What changes is what this query reads from storage: with event_type and amount shredded, the 2 typed columns instead of every full event. Our benchmark of VARIANT against JSON strings on 2 million events shows how VARIANT itself compares to from_json and string functions; shredding builds on that.

A checklist before 18 November

1. Find the jobs that create tables with VARIANT columns using CREATE TABLE. From 18 November those tables get the feature.
2. List every cluster, warehouse and outside engine that reads or writes them, and move anything on 15.4 to 17.3 or above.
3. For a table an old reader must keep reading, create it with CREATE OR REPLACE TABLE, or drop the feature with ALTER TABLE ... DROP FEATURE "variantShredding". Setting the property to false only stops shredded writes; the feature stays on the table.
4. For a large existing VARIANT table that is read far more than written, set the property and run REORG TABLE ... APPLY (SHRED VARIANT).

VARIANT tables are Delta LakeDelta LakeAn open table format that turns Parquet files in cloud storage into a table with transactions, a version history, schema checks, and updates and deletes, so a data lake behaves like a database. tables like the rest, and the table feature list is how Delta keeps old readers from misreading new data. The data engineering guide follows where a record goes from there.