Spark VARIANT vs JSON Strings: A Benchmark on 2 Million Events

VARIANT is a column type added in Apache Spark 4.0 for semi-structured data such as JSON. You parse a JSON string once with parse_json, store the result as a compact binary value, and then read fields from it with variant_get without parsing text again and without declaring a schema. In our test on 2 million events, reading three fields from a stored VARIANT column took 0.23 seconds, against 0.84 to 0.89 seconds for the JSON-string functions.

That is a real gain, but not the "10x over UDFs" often quoted, and parsing to VARIANT on every query is slower than the alternatives. This guide shows how VARIANT works, the measured numbers, and when it is the right choice.

The three old ways to read JSON in Spark

A column of JSON strings can be read in three ways:

from pyspark.sql import functions as F

# 1. get_json_object: one path at a time, no schema
raw.select(F.get_json_object("raw", "$.user.country").alias("country"))

# 2. from_json: parse into a struct with a schema you declare
schema = "user STRUCT<country: STRING>, event STRING, amount DOUBLE"
raw.select(F.from_json("raw", schema).alias("p")).select("p.user.country")

# 3. a Python UDF with json.loads

All three parse the text every time the query runs. from_json is fast but needs a schema, and any field you leave out of it is lost. get_json_object needs no schema but reads one path per call. A Python UDF sends every row to a Python process and back.

How VARIANT works

parse_json turns a JSON string into a VARIANT value: a binary encoding that keeps the types (numbers stay numbers) and lets Spark jump to a field without scanning the text. You store that column, and later queries read fields by path:

events = raw.select(F.parse_json("raw").alias("v"))
events.write.parquet("events_variant")

v = spark.read.parquet("events_variant")
v.select(
    F.variant_get("v", "$.user.country", "string").alias("country"),
    F.variant_get("v", "$.amount", "double").alias("amount"),
)

In SQL the same is variant_get(v, '$.user.country', 'string'), and Spark 4.2 also accepts the shorter v:user.country::string. Dot notation such as v.user.country does not work on a VARIANT column: Spark answers INVALID_EXTRACT_BASE_FIELD_TYPE, because dots only reach into structs, arrays and maps.

Different rows can hold different fields. A path that is missing in a row gives NULL, so events whose shape changes over time can share one column.

The benchmark

We generated 2 million e-commerce events as JSON, each with a nested user, an event type, an amount, a list of items and some metadata, and stored them as Parquet twice: once as JSON strings, once as VARIANT. Each query read three fields, kept purchases and summed revenue by country. It ran on PySpark 4.2.0 with 4 local cores, three times each; all methods returned identical totals.

Reading from stored VARIANT was about 3.5 to 4 times faster than either string function. Parsing to VARIANT inside the query was the slowest, because it pays for building the binary value and then reads from it: the gain comes from parsing once and storing the result.

Storage went down too. The VARIANT Parquet files took 56.6 MB against 110.7 MB for the same events as JSON strings.

Python UDFs. A UDF with json.loads was the slowest of all, but by less than often claimed. On one core over the same 2 million rows, it took 3.9 seconds against 2.5 seconds for get_json_object, about 1.5 times as long. The large gap is between any text parsing and reading stored VARIANT, not between UDFs and built-in functions. Your numbers will differ with the size of the documents and how many fields you read; measure on your own data.

Reading VARIANT safely

-- a value of the wrong type fails the query
SELECT variant_get(parse_json('{"a":"x"}'), '$.a', 'int');
-- [INVALID_VARIANT_CAST] The variant value `"x"` cannot be cast into `"INT"`.

-- try_variant_get returns NULL instead
SELECT try_variant_get(parse_json('{"a":"x"}'), '$.a', 'int');   -- NULL

-- array elements by position
SELECT variant_get(parse_json('{"items":[{"sku":"S1"},{"sku":"S2"}]}'), '$.items[1].sku', 'string');   -- S2

Other functions worth knowing, all tested on Spark 4.2:

• try_parse_json returns NULL for invalid JSON, where parse_json fails the query.
• schema_of_variant describes a value's shape, for example OBJECT<a: BIGINT, b: ARRAY<BIGINT>, c: OBJECT<d: STRING>>, which helps when you decide which fields to promote to columns.
• variant_explode turns an object or array into one row per key or element.
• to_json turns a VARIANT back into a JSON string.

When to use VARIANT

Use it for semi-structured data you store and query many times, whose fields vary or change: event streams, API responses, logs, data from many sources. Parse it once when you load it, store the VARIANT column, and read fields as needed.

Use typed columns instead for fields every query reads, such as an event time or customer id. Pull them out with variant_get when you load the data and store them as ordinary columns; they compress better and filter faster. A common pattern keeps a few typed columns next to one VARIANT column holding the rest.

Keep from_json when the schema is fixed and known: parsing straight into typed columns is as fast as the string functions get, and you get a struct to work with.

Avoid parsing to VARIANT inside every query of a JSON string column. It is the slowest option in our test.

On Databricks, VARIANT is also available in Delta tables, with the same functions and the : path syntax.

Tecyfy Takeaway

VARIANT pays off when you parse JSON once and store it. Reading three fields from stored VARIANT was about four times faster than from JSON strings in our test, and the files were half the size. Use variant_get or try_variant_get, not dot notation. Promote the fields every query uses to typed columns, and keep from_json for data whose schema never changes.