Executors keep failing with "Container killed for exceeding memory limits". What do you check, and how do you fix it?

First work out which memory ran out, because there are two kinds. If the JVM heap ran out, the executor log shows a Java OutOfMemoryError. If the container was killed by YARN or Kubernetes for going over its limit, the memory that grew was outside the heap: usually Python worker processes, or native buffers. The fixes are different, so raising spark.executor.memory for every failure often does nothing.

How an executor's memory is split

The cluster manager gives each executor a container sized as the sum of:

• spark.executor.memory: the JVM heap, where Spark's execution and storage memory live.
• spark.executor.memoryOverhead: memory outside the heap. By default 10% of executor memory (40% for non-JVM jobs on Kubernetes).
• spark.executor.pyspark.memory, if set, and off-heap memory, if enabled.

If spark.executor.pyspark.memory is not set, the Python workers share the overhead allowance and Spark does not limit how much they use. A PySpark job with Python UDFs or pandas code is the usual way a container gets killed while the heap looks fine.

Diagnosis

1. Read the executor's last log lines. "Container killed ... exceeding memory limits" means the container limit. java.lang.OutOfMemoryError: Java heap space means the heap.
2. Find the stage that was running. Is it a wide stage (join, groupBy, window) or a stage with a Python UDF or applyInPandas?
3. Look at that stage's task distribution. One task with far more shuffle read than the rest means skew: a single partition is too big, and no setting fixes that alone.

Fixes for the container limit (outside the heap)

• Raise spark.executor.memoryOverhead, or set spark.executor.pyspark.memory so Python gets its own budget.
• Cut what Python holds: process smaller batches, avoid building large Python lists, and replace Python UDFs with built-in functions where you can.
• With applyInPandas or another grouped map, all rows of a group are loaded into memory at once. A large or skewed group needs splitting.

Fixes for the heap

• Raise parallelism so each task handles less data: more shuffle partitions, or let AQE size them. It is the simplest fix when one task's working set is too large.
• Fix skew: AQE skew join handling, salting a hot key, or treating the hot key separately.
• Fewer cores per executor. Concurrent tasks share the heap, so 8 cores with 16 GB gives each task about half as much as 4 cores with 16 GB.
• Stop caching data you do not reuse. Cached blocks compete with execution memory.

What not to do

Doubling memory until the error stops hides skew and makes every run cost more. It also tends to fail again when the data grows.

What interviewers want: the difference between heap and overhead, that PySpark's Python workers live outside the heap, and checking for skew before adding memory.