What is data skewness in Spark, and how can it be addressed effectively?

Data skew is an uneven distribution of data across partitions, usually because a few key values have far more rows than the rest. The tasks that process those partitions run much longer, or run out of memory, while the others finish quickly, so the whole stage waits for a few stragglers.

Where it comes from

• Joins and aggregations on keys with hot values: a few large customers, a default value such as "unknown", or NULL keys.
• Time-based data with spikes, such as a sales day.
• Input files of very different sizes, or a repartition on a skewed column.

How to spot it

• In the Spark UI's stage details, the maximum task duration and shuffle read size are far above the median.
• Most tasks finish and the stage hangs on the last few.
• Out-of-memory errors or heavy spill on only some tasks.
• To find hot keys: df.groupBy("key").count().orderBy(F.desc("count")).show().

How to address it

1. Adaptive query execution (on by default) splits skewed partitions in sort-merge and shuffled hash joins: a partition over 5 times the median and over 256 MB is split, and the matching rows on the other side are replicated.
2. Broadcast joins when one side is small: no shuffle, so no skew.
3. Handle NULL and default keys separately: filter them out before inner joins, or spread them with random values in outer joins.
4. Salting: add a random number to hot keys so their rows spread over several partitions, then combine the partial results (two-step aggregation), or replicate the matching rows of the other side for joins.
5. Isolate hot keys: process the few biggest keys in a separate path, often with a broadcast join, and union the results.
6. Better partitioning when writing: avoid partitioning tables by skewed columns; in Delta, compaction and clustering keep files evenly sized.

Check the fix Compare task duration distributions before and after. A good fix brings the maximum close to the median without adding much total work; salting, in particular, adds extra steps, so use it only where skew is real.