A nightly PySpark job started failing: executors are lost and the logs mention running out of memory. Walk me through how you find the cause before changing any setting.

First establish which memory ran out, from the exact error, then which task, from the Spark UI. java.lang.OutOfMemoryError: Java heap space in an executor log means the heap filled, usually because one task held too much data. "Container killed by YARN for exceeding memory limits", or exit code 137 on Kubernetes, means the container's total went over, usually memory outside the heap such as Python workers. The Stages tab then shows whether every task is big (too few partitions) or one is huge (skew). Only then pick the fix, and raising spark.executor.memory is rarely the first one.

1. Read the error

2. Find the task in the Spark UI

• All tasks of the stage high in peak memory and spill: partitions are too big. Raise spark.sql.shuffle.partitions above the default 200, or repartition() before the heavy step.
• One task 10 times the median: skew on a key. Check the key's distribution; salt it, or rely on adaptive skew-join handling, which is on by default for joins.
• The failure at a join with a small table: a broadcast that is not small any more. Lower spark.sql.autoBroadcastJoinThreshold or drop the hint.

3. Check what changed

A job that worked last month and fails now usually met more data or a new skewed key. Compare input sizes and the top keys between a good and a bad run.

4. Look at the code

# Gathers every value for a key in one task's memory
rdd.groupByKey().mapValues(sum)

# Combines before the shuffle
rdd.reduceByKey(lambda a, b: a + b)
# or, better, a DataFrame aggregation
df.groupBy("key").agg(F.sum("value"))

Python UDFs that build large lists also count against memory outside the heap; spark.executor.pyspark.memory gives them a limit that fails clearly.

5. Then, and only then, size

Raise overhead or executor memory last, keeping memory plus overhead under the machine's container limit, and note that more memory per executor means fewer executors. See Spark Executor Memory.

What interviewers want: the 3 error families and which memory each means, the Spark UI to separate big partitions from skew, a check of what changed in the data, code fixes like reduceByKey and broadcast limits, PySpark memory outside the heap, and memory increases last.