The full answer
Compare a slow run with a good run before changing anything. A job slows down for one of a few reasons: there is more input, the data has become uneven (skew), the query plan changed, tasks are spilling to disk, or the cluster got fewer resources. The Spark UI shows which one, stage by stage.
1. Find the stage that grew
Open both runs in the Spark UI (or the history server) and line up their stages. Usually one stage accounts for nearly all the extra time. Everything that follows is about that stage.
2. Read the task summary for that stage
The stage page shows task metrics as a distribution. Compare the median task with the slowest:
| What you see | What it means |
|---|---|
| Max task duration far above the median, one task reading most of the shuffle data | Skew: one key holds a large share of the rows |
| Shuffle spill (disk) present where it was zero before | Partitions too large for execution memory |
| GC time a large share of task time | Memory pressure on the executors |
| All tasks slower, evenly | More data, slower storage, or fewer cores |
| Many more tasks, each tiny | Too many small input files or partitions |
3. Check the input
Did the row count or the number of files jump? A source that now delivers 50,000 small files instead of 500 large ones slows the read even when the bytes are the same. A full reload upstream, or a broken filter that stopped pruning partitions, shows up as a much larger "input size" on the read stage.
4. Compare the query plans
In the SQL tab, open the plan of both runs. Look for:
- A join that was a broadcast hash join and is now a sort-merge join. The small side may have grown past the broadcast threshold (10 MB by default for
spark.sql.autoBroadcastJoinThreshold). - A filter that no longer reaches the scan, so a whole table is read.
- Row counts per operator: a join whose output is far larger than its inputs points at duplicate keys.
5. Check the cluster
Fewer executors than usual (a capped autoscaling pool, spot instances lost), a different instance type, or another job sharing the cluster can all make an unchanged job slower.
Fixes, matched to the cause
- Skew: make sure Adaptive Query Execution's skew join handling is on (AQE has been on by default since Spark 3.2.0). It splits a partition that is both 5 times the median and larger than 256 MB, by default. For skew it does not catch, such as in aggregations, salt the hot key or handle it separately.
- Spill: more, smaller partitions, or more memory per core.
- Plan change: fix the statistics or filter that caused it. Use a join hint only when you know the small side stays small.
- Small files: compact the input, or fix the writer upstream.
Prevent the next one
Record each run's duration, input rows and bytes, and output rows. Alert when duration or input moves far from the recent normal. Most slowdowns are visible in those numbers days before anyone complains.
What interviewers want: a method, not a list of settings. Compare against a good run, find the stage, read the task distribution, then fix the cause you found. "I would increase the cluster size" without a diagnosis is the answer they are screening out.
Last updated 16 October 2026
Sources