The full answer
Data skew means a few keys or partitions hold far more data than the rest, so a few tasks run much longer than the others and the stage waits for them. Start by confirming and locating it, then use adaptive query execution's skew handling, broadcast joins, salting or separate handling of the hot keys.
1. Detect it
- Spark UI: in the slow stage, compare the maximum task duration and shuffle read size with the median. A maximum many times the median means skew.
- Find the hot keys:
df.groupBy("key").count().orderBy(F.desc("count")).show(20). - Common causes: NULL or default keys ("unknown", 0), a few very large customers, a single date with a backlog.
2. Let adaptive query execution handle skewed joins
AQE is on by default. In sort-merge and shuffled hash joins it splits a partition that is larger than 5 times the median and over 256 MB (spark.sql.adaptive.skewJoin.*) into smaller tasks, replicating the matching rows from the other side. This fixes many skewed joins with no code change. Check the plan for skew=true in the SQL tab.
3. Broadcast the small side If one side of the join is small, broadcasting it avoids the shuffle, and with it the skew:
from pyspark.sql.functions import broadcast
result = facts.join(broadcast(dim), "customer_id")
4. Deal with NULL and junk keys Rows with NULL keys never match in an inner join, so filter them out before the join; or, for an outer join, give each a random key so they spread out.
5. Salt the hot keys (aggregations) Add a random salt, aggregate on key plus salt, then aggregate again on the key:
from pyspark.sql import functions as F
SALT = 16
partial = (df.withColumn("salt", (F.rand() * SALT).cast("int"))
.groupBy("key", "salt").agg(F.sum("value").alias("partial_sum")))
result = partial.groupBy("key").agg(F.sum("partial_sum").alias("total"))
This works for sums, counts, minimums and maximums; for averages, carry sums and counts and divide at the end.
6. Salt joins when AQE isn't enough Salt the large side randomly and replicate the small side once per salt value:
facts_salted = facts.withColumn("salt", (F.rand() * SALT).cast("int"))
dim_salted = dim.crossJoin(spark.range(SALT).withColumnRenamed("id", "salt"))
joined = facts_salted.join(dim_salted, ["key", "salt"]).drop("salt")
It multiplies the small side by the salt count, so salt only the hot keys when the other side is large.
7. Split hot keys out
Process the few hot keys separately (often with a broadcast join), process the rest normally, and union the results.
8. Fix it at the source Pre-aggregate data upstream, choose partition columns with even distribution, and avoid writing tables partitioned by a skewed column.
Re-check the stage's task-time distribution after each change: skew fixes add complexity, so keep only what makes a measurable difference.
Last updated 16 October 2026
Sources