The full answer
The driver plans the work and collects results. It does not process the data, so a driver out of memory almost always means data was pulled to it on purpose or by accident: collect(), toPandas(), a broadcast that turned out larger than expected, or plain Python code working on large data. Find which line did it, then keep that work on the executors.
The usual causes
| Cause | How it looks |
|---|---|
collect(), toPandas() or take(n) with a large n | Fails on the line that brings results back |
| A broadcast join whose small side is not small | Error naming the broadcast table and spark.driver.maxResultSize |
| Python loops or pandas over data that should be in Spark | Driver memory climbs while executors sit idle |
| Many notebooks or jobs sharing one cluster's driver | Fails at random times, often on an interactive cluster |
Fixes, one per cause
- Results pulled to the driver.
collect()andtoPandas()load every row into the driver's memory. Write the result to a table or files instead, and read back only what a person needs to see. If you need a few rows, uselimit()first. - An oversized broadcast. Spark estimates table sizes when it plans a broadcast, and the estimate can be wrong. Compute statistics on the table (
ANALYZE TABLE), remove abroadcast()hint that no longer fits, or ask for a sort-merge join with aMERGEhint. - Python on the driver. A loop over rows, or pandas code on a large frame, runs on the driver alone, however large the cluster is. Rewrite it with DataFrame operations so the executors share the work.
- A shared driver. Run scheduled jobs on their own job clusters, not on the interactive cluster everyone uses.
About spark.driver.maxResultSize
It caps the total size of results of one action (1 GB by default) and aborts the job above it. That error is a guard doing its job. Raising the cap without raising driver memory swaps a clear error for a crash. Raise it only when you know the result is meant to be that size.
When more memory really is the answer
If the driver still runs short once the code keeps the data on the executors, give it more: raise spark.driver.memory or pick a larger driver node. In client mode the setting has to be given when the application starts (--driver-memory), not from inside it, because the driver is already running by then.
What interviewers want: knowing that the driver does not process data, naming collect(), toPandas() and broadcasts as the main suspects, and fixing the code before resizing the driver.
Last updated 16 October 2026