The full answer
Traditional query optimisation in Spark plans the whole query once, before it runs, using estimates: catalog statistics, file sizes and rules. Adaptive query execution (AQE) re-optimises the remaining plan while the query runs, using the actual sizes of the data produced by each completed stage. Since Spark 3.2, AQE is on by default.
Traditional, static optimisation
- Catalyst applies rules (predicate pushdown, column pruning, constant folding) and, where statistics exist, cost-based choices such as join order and join strategy.
- Every decision is fixed before execution: the number of shuffle partitions, whether a join is broadcast, how partitions are sized.
- Estimates are often wrong: statistics are missing or stale, filters are more or less selective than assumed, and sizes after joins and user-defined functions are guesses.
Adaptive query execution AQE splits the query at shuffle and broadcast exchanges into query stages. When a stage finishes, Spark knows the real size of every output partition, and re-optimises what is left:
- Coalescing shuffle partitions: merges many small post-shuffle partitions, so
spark.sql.shuffle.partitionsdoesn't have to be tuned for each query. - Switching join strategies: converts a sort-merge join to a broadcast hash join if one side turns out to be small, reading shuffle files locally to save network traffic.
- Handling skew: splits oversized partitions in sort-merge joins into smaller tasks.
- Choosing shuffled hash joins when partitions are small enough (configurable).
| Static optimisation | AQE | |
|---|---|---|
| When decisions are made | Before execution | Before execution, then again after each stage |
| Information used | Statistics and estimates | Actual sizes of shuffle and broadcast output |
| Partition count | Fixed (spark.sql.shuffle.partitions) | Coalesced and split at runtime |
| Join strategy | Fixed at planning time | Can change mid-query |
| Skew | Manual fixes (salting, hints) | Automatic in sort-merge joins |
| Scope | Every query | Queries with shuffles or broadcasts; not streaming queries |
They work together
AQE doesn't replace Catalyst: the static optimiser still produces the initial plan, and AQE adjusts it. Good statistics (ANALYZE TABLE, file-level statistics in table formats) still help the initial plan; AQE corrects what estimates got wrong.
Seeing it
explain() shows AdaptiveSparkPlan isFinalPlan=false before execution; after the query runs, the SQL tab of the Spark UI shows the final plan, with nodes such as AQEShuffleRead (coalesced or skewed partitions) and any changed join strategies.
Last updated 16 October 2026
Sources