Apache Spark 4.2 features: what's new and what breaks

Apache Spark 4.2.0 came out on 14 July 2026. Its main additions are geospatial types, change data capture built into Spark, Auto CDC in declarative pipelines, metric views, a nearest-neighbour join, and Python UDFs that run on Arrow by default. Most PySpark code runs on it unchanged. The things to check before upgrading are PyArrow 18, the Arrow default for UDFs and a few SQL results that now differ.

Databricks ships it in Databricks Runtime 19. If you are on Spark 4.0, what's new in Spark 4.0 covers the step before this one.

The new features at a glance

Geospatial types, on by default

Spark now has GEOMETRY and GEOGRAPHY types with ST_* functions, reads and writes them as WKB, WKT and Parquet, and has a full registry of coordinate systems. Until now this needed Apache Sedona or a library of UDFs. Delivery zones, store catchments and "orders within 5 km" become plain SQL.

Change data capture inside Spark

Change data captureChange Data CaptureCopying only the rows that were added, changed or deleted in a source database, read from its change log, instead of copying the whole table again. is now part of Spark itself. A table's changes between 2 versions come back as a DataFrame:

changes = (
    spark.read
    .option("startingVersion", "10")
    .option("endingVersion", "20")
    .changes("shop.orders")
)

SQL has a matching CHANGES clause, and a streaming read keeps up with new changes. There is a catch in the docs: it works for Data Source V2 tables whose catalog implements the change log, so it depends on your table format supporting it.

The other half is Auto CDC, which applies a change feed to a target table in Spark Declarative Pipelines. In 4.2 it keeps a SCD type 1Slowly Changing DimensionsThe ways a warehouse handles a change to a descriptive record, such as a customer's address: type 1 overwrites it, type 2 adds a new dated version, and type 3 keeps the previous value in an extra column. table from Python only: SQL and type 2 are marked for Spark 4.3. Spark 4.2 Auto CDC works through it with late and deleted rows.

Metric views come to open-source Spark

CREATE VIEW ... WITH METRICS defines measures in YAML once, so revenue per customer is computed correctly whatever it is grouped by. Databricks has had metric views in Unity Catalog for a while; 4.2 brings the same idea to Spark SQL. Databricks metric views shows an example and the limits.

Python is faster by default

Plain Python UDFs and UDTFs now run Arrow-optimized, and data moves between Python and the JVM as Arrow, with no change to your code. It is the biggest change most people will notice, in speed, and the one most likely to surface a type difference. The settings to switch each back are in the questions below.

The rest worth knowing

• NEAREST BY is a new join that keeps the top K nearest rows of the other table, with new vector similarity, norm and average functions beside it. Matching products to embeddings no longer needs a cross join and a window.
• Real-time mode processes streams with millisecond latency, in 4.2 for stateless PySpark queries without Python UDFs.
• SQL: QUALIFY filters on a window function directly; INSERT INTO ... WITH SCHEMA EVOLUTION adds new columns; INSERT INTO ... REPLACE ON replaces matching rows; time_bucket groups timestamps; SET PATH and CURRENT_PATH() control name lookup, off unless you turn on spark.sql.path.enabled.
• Data Source V2 connectors can commit several operations as one transaction.
• Platform: Java 25 support, a new Spark UI with dark mode and side-by-side plans, and R 3.x dropped.

What breaks, or gives a different answer

Mark what your jobs use:

Try itWhat changes for your code in Spark 4.2?

My code uses

·None of these·PyArrow older than 18·PyPy·Plain Python UDFs·Pandas UDFs on integer columns with nulls·createDataFrame from a NumPy array·pandas API on Spark drop()·GROUP BY GROUPING SETS (())·NATURAL JOIN on mixed-case names·A schema named builtin or session·Observation.get in error handling

1. PySpark

   PyArrow older than 18

   Not used

2. PySpark

   PyPy

   Not used

3. PySpark

   Plain Python UDFs

   Python UDFs and UDTFs now run Arrow-optimized by default, and data moves between Python and the JVM as Arrow.

   Fix: Usually faster with no change. If results differ, set spark.sql.execution.pythonUDF.arrow.enabled to false and test.

   Results can differ

4. PySpark

   Pandas UDFs on integer columns with nulls

   A nullable integer column with nulls arrives as pandas Int64 (or Int8, Int16, Int32), not float64.

   Fix: Change UDF code that expected NaN and floats.

   Results can differ

5. PySpark

   createDataFrame from a NumPy array

   Not used

6. PySpark

   pandas API on Spark drop()

   Not used

7. SQL

   GROUP BY GROUPING SETS (())

   An empty grouping set is now a grand total: 1 row over empty input instead of none.

   Fix: Expect the extra row, or set spark.sql.analyzer.lowerEmptyGroupingSetToGlobalAggregate.enabled to false.

   Results can differ

8. SQL

   NATURAL JOIN on mixed-case names

   Not used

9. SQL

   A schema named builtin or session

   Not used

10. SQL

    Observation.get in error handling

    Not used

2 more changes affect everyone a little. Shuffle outputs now carry order-independent checksums by default: when a retried stage gives different data, Spark reruns the stages that depend on it, and fails the job if it cannot. And duplicate CTE names that differ only in case, such as a and A, are now an error.

Should you upgrade now?

On Databricks, try it on Runtime 19 with a copy of your heaviest jobs. On your own clusters, upgrade PyArrow first, run the test suite on 4.2, and compare a day's output of every job that uses Python UDFs or pandas UDFs. Our data engineering course runs every example on PySpark 4.2.0, so it is a quick way to see the new version working.