Skip to the page
Tecyfy

PySpark interview question · hard

What are some limitations of AQE? Are there cases where it might not be beneficial or could even hurt performance?

The full answer

Yes. AQE is on by default because it helps most batch queries, but it only acts at shuffle and broadcast boundaries, only fixes certain problems, and in some cases its choices cost time.

Where AQE can't help

  • No exchange, no adaptation: a query that reads, filters and writes without a shuffle has nothing for AQE to re-optimise. The first stage, including how files are split into partitions, is planned statically.
  • Streaming queries: AQE doesn't apply to Structured Streaming.
  • Skew outside sort-merge joins: skew join handling covers sort-merge joins (and rebalancing). Skewed aggregations, skewed window functions and skew in broadcast or shuffled hash joins still need manual fixes such as salting.
  • Some join types: skew can only be split on the side that the join type allows, for example not the right side of a left outer join.
  • Bad upstream plans: AQE adjusts the plan after stages run; it won't rescue a cartesian product or a UDF that blocks pushdown.
  • RDD code: AQE works on DataFrames and SQL only.

Where it can hurt or surprise

  • Late broadcast conversion: switching to a broadcast join after both sides have already been shuffled is less efficient than planning a broadcast from the start. Broadcast hints or good statistics avoid the wasted shuffle.
  • Too much coalescing or too little: by default (parallelismFirst true) coalescing ignores the 64 MB target and keeps parallelism high, which can create many small tasks on a busy shared cluster. Setting it to false packs partitions to the target, which can reduce parallelism for CPU-heavy work on small data.
  • Output layout changes: coalescing changes the number of output partitions, so the number and size of written files can vary from run to run.
  • Forced skew handling: forceOptimizeSkewedJoin may add an extra shuffle.
  • Re-planning overhead: each stage boundary adds a little planning time, which matters only for very short queries.
  • Broadcast memory: a converted broadcast join can put pressure on driver and executor memory if the threshold has been raised too high.

How to manage it

  • Keep AQE on, and tune rather than disable it: initialPartitionNum, advisoryPartitionSizeInBytes, parallelismFirst, and the skew thresholds.
  • Disable individual rules with spark.sql.adaptive.optimizer.excludedRules if one consistently hurts a workload.
  • Use hints where you know better (BROADCAST, REPARTITION, REBALANCE).
  • Keep statistics current so the initial plan is good, and compare the initial and final plans in the Spark UI.
  • Handle known skew in aggregations yourself with salting, or with a two-stage aggregation.

Bottom line AQE's downsides are small and mostly about predictability; turning it off globally is rarely right. The real limitation is that it only fixes partition sizing, join strategy and join skew at shuffle boundaries, so plan design still matters.

Last updated 16 October 2026

Sources

Where each fact comes from

  1. 1Performance tuning
  2. 2Spark configuration