How is memory divided inside a Spark executor?

A Spark job runs as many executor processes, each a Java virtual machine with a fixed amount of memory. Most "Spark ran out of memory" problems are really "one part of this memory ran out", and knowing the parts tells you which setting, or which line of code, to change.

The parts

Take an executor with spark.executor.memory set to 8 GiB:

1. Reserved: 300 MiB. Kept for Spark's own needs.
2. Unified memory: 60% of the rest, about 4.6 GiB. Shared by execution (shuffles, joins, sorts, aggregations) and storage (cached DataFrames). Each starts with half. Execution can push cached data out when it needs room; cached data can never push out a running join. This split is spark.memory.fraction (0.6) and spark.memory.storageFraction (0.5).
3. User memory: the other 40%, about 3.1 GiB. Your own objects, data built inside UDFs, Spark's metadata.
4. Overhead, outside the heap: 10%, at least 384 MiB. Thread stacks, network buffers, native libraries. Set by spark.executor.memoryOverhead.
5. Python workers. In PySpark, Python processes run beside the JVM and use memory outside the heap, unless spark.executor.pyspark.memory gives them their own limit.

The cluster manager, YARN or Kubernetes, gives the executor a container sized heap plus overhead. If the container's total goes over, the container is killed, even if the heap was half empty.

Cores share it

An executor runs as many tasks at once as it has cores, and they share the unified memory. 4 cores on 8 GiB give each running task roughly a quarter of the execution pool. So fewer cores per executor, or more executors with less memory each, changes how much each task can hold, without any change to the total.

What fills it

The usual culprit is one task's working set: a shuffle partition far bigger than the rest because of skewShuffle and Data SkewA shuffle moves rows between machines so rows with the same key meet, for a join or group by; skew is when one key has far more rows, so one task does most of the work., too few partitions for a large shuffle, groupByKey gathering every value for a key, or a broadcast table larger than expected. Spark's own tuning guide puts more parallelism, smaller tasks, first among fixes, before more memory. PartitioningData Partitioning and ClusteringTwo ways to lay out a large table so a query reads only the files it needs: partitioning splits it by a column such as the date, clustering keeps similar values in the same files. the input well prevents much of it.

When execution memory runs short, Spark spills to disk instead of failing, which is slow but survivable. A Spark UI that shows heavy spill on a stage is the warning before an out-of-memory error.

Our article Spark out of memory matches each error message to its fix, and the Spark sizing calculator picks executor sizes for a workload.