Skip to the page
Tecyfy

PySpark interview question · hard

How does Adaptive Query Execution (AQE) differ from traditional query optimization in Spark?

The short answerTraditional query optimisation in Spark plans the whole query once, before it runs, using estimates: catalog statistics, file sizes and rules.

The full answer

Traditional query optimisation in Spark plans the whole query once, before it runs, using estimates: catalog statistics, file sizes and rules. Adaptive query execution (AQE) re-optimises the remaining plan while the query runs, using the actual sizes of the data produced by each completed stage. Since Spark 3.2, AQE is on by default.

Traditional, static optimisation

  • Catalyst applies rules (predicate pushdown, column pruning, constant folding) and, where statistics exist, cost-based choices such as join order and join strategy.
  • Every decision is fixed before execution: the number of shuffle partitions, whether a join is broadcast, how partitions are sized.
  • Estimates are often wrong: statistics are missing or stale, filters are more or less selective than assumed, and sizes after joins and user-defined functions are guesses.

Adaptive query execution AQE splits the query at shuffle and broadcast exchanges into query stages. When a stage finishes, Spark knows the real size of every output partition, and re-optimises what is left:

  • Coalescing shuffle partitions: merges many small post-shuffle partitions, so spark.sql.shuffle.partitions doesn't have to be tuned for each query.
  • Switching join strategies: converts a sort-merge join to a broadcast hash join if one side turns out to be small, reading shuffle files locally to save network traffic.
  • Handling skew: splits oversized partitions in sort-merge joins into smaller tasks.
  • Choosing shuffled hash joins when partitions are small enough (configurable).
Static optimisationAQE
When decisions are madeBefore executionBefore execution, then again after each stage
Information usedStatistics and estimatesActual sizes of shuffle and broadcast output
Partition countFixed (spark.sql.shuffle.partitions)Coalesced and split at runtime
Join strategyFixed at planning timeCan change mid-query
SkewManual fixes (salting, hints)Automatic in sort-merge joins
ScopeEvery queryQueries with shuffles or broadcasts; not streaming queries

They work together AQE doesn't replace Catalyst: the static optimiser still produces the initial plan, and AQE adjusts it. Good statistics (ANALYZE TABLE, file-level statistics in table formats) still help the initial plan; AQE corrects what estimates got wrong.

Seeing it explain() shows AdaptiveSparkPlan isFinalPlan=false before execution; after the query runs, the SQL tab of the Spark UI shows the final plan, with nodes such as AQEShuffleRead (coalesced or skewed partitions) and any changed join strategies.

Last updated 16 October 2026

Sources

Where each fact comes from

  1. 1Performance tuning
  2. 2Spark configuration