How to read the Spark UI
Open the longest job, then its slowest stage, then compare the Max and Median columns of that stage's Summary Metrics. Most of the rest of the page can wait.
Run on Spark 4.2.0
The job runs, but one stage takes most of the time, and in it one task runs long after the rest.
Spark Jobs
Scheduling Mode: FIFOCompleted Jobs: 5Completed Jobs (5)
| Job Id ▾ | Description | Submitted | Duration | Stages: Succeeded/Total | Tasks (for all stages): Succeeded/Total |
|---|---|---|---|---|---|
| → | Join orders to customers | 2026/10/23 05:04:55 | 11 s | 1/1 (2 skipped) | 16/16 (8 skipped) |
| 2Join orders to customers | 2026/10/23 05:04:55 | 0.6 s | 1/1 | 4/4 | |
| Join orders to customers | 2026/10/23 05:04:55 | 0.6 s | 1/1 | 4/4 | |
| Skip | parquet at DirectMethodHandleAccessor.java:104 | 2026/10/23 05:04:54 | 16 ms | 1/1 | 1/1 |
| parquet at DirectMethodHandleAccessor.java:104 | 2026/10/23 05:04:54 | 0.2 s | 1/1 | 1/1 |
Each job ran on Spark 4.2 on one machine with 4 cores (local[4]), so times are seconds, not minutes; the pages are drawn from its event log, as the history server draws them. The host name is hidden.
What does data skew look like in the Spark UI?
One task in the stage runs many times longer than the median task and reads far more shuffle data, because one key holds most of the rows and every row of a key goes to the same task.
8 in 10 orders belong to customer 0. A join sends every row with the same key to the same partition, so one task got 8 in 10 of the rows while the other 15 had little to do. That task also spilled to disk: its rows did not fit in its share of memory.
The fix: Let adaptive execution split the skewed partition. It is on by default since Spark 3.2, but it only splits a partition over 256 MB and 5 times the median, and not into pieces under the 64 MB target size. This sample's big partition was about 110 MB, so the run lowered both to 16 MB. Where adaptive execution cannot split, as in a group by or a window on the hot key, add a random salt to the key.
The job that ran: join orders to customers
# 10 million orders; 8 in 10 belong to customer 0, the rest spread over 999 others.
spark.range(10_000_000).select(
F.when(F.col("id") % 10 < 8, 0).otherwise(F.col("id") % 1000).alias("customer_id"),
F.col("id").alias("order_id"),
(F.rand(7) * 100).alias("amount"),
).write.parquet(f"{path}/orders")
spark.range(1000).select(
F.col("id").alias("customer_id"), F.concat(F.lit("name-"), F.col("id")).alias("name")
).write.parquet(f"{path}/customers")
orders = spark.read.parquet(f"{path}/orders")
customers = spark.read.parquet(f"{path}/customers")
orders.join(customers, "customer_id").write.format("noop").mode("overwrite").save()What does spill to disk look like in the Spark UI?
Spill (Memory) and Spill (Disk) in the stage's Summary Metrics are above 0: each task's data was too big for its share of memory, so Spark wrote part of it to disk and read it back.
A sort, a join or a group by holds its partition's rows in memory. When a partition is bigger than the task's share of memory, Spark writes sorted runs to disk and merges them back, so the task does the work twice.
The fix: Use more, smaller shuffle partitions, so each task's share fits in memory. Adaptive execution then merges any that come out too small. More memory per executor works too, at a price. The shuffle partition calculator works out the number for your data.
The job that ran: sort orders by amount
# 12 million orders, each with a 64-character reference, written once as Parquet.
spark.range(12_000_000).select(
F.col("id").alias("order_id"),
(F.rand(7) * 100).alias("amount"),
F.lpad(F.col("id").cast("string"), 64, "0").alias("reference"),
).write.parquet(f"{path}/orders")
orders = spark.read.parquet(f"{path}/orders")
orders.orderBy("amount").write.format("noop").mode("overwrite").save()Questions
- What is the Spark UI?
- The Spark UI is the web page every Spark application serves while it runs, at port 4040 of the driver by default. It shows each job, its stages and every task, with how long they took and how much data they read, shuffled and spilled. The history server shows the same pages for an application that has finished, read from its event log.
- How do I open the Spark UI on Databricks?
- Open Compute, select the cluster that ran the job, then the Spark UI tab. From a notebook, the job's link under the cell opens the same pages. Serverless compute has no Spark UI; it shows the query profile instead.
- What is a stage, and what is a task, in the Spark UI?
- A job is split into stages at each shuffle, where rows move between machines. A stage is split into tasks, one for each partition of its data, and each task runs on one core. A stage ends only when its last task ends, so one slow task holds the whole stage.
- What does spill mean in the Spark UI?
- Spill (Memory) and Spill (Disk) above 0 mean a task's data did not fit in its share of memory, so Spark wrote part of it to disk and read it back. The usual fix is more, smaller partitions, so each task holds less.
- Why does a stage take long when most tasks are fast?
- A stage waits for its slowest task. If one task reads much more data than the others, because of skew, or runs on a slower machine, the stage takes as long as that task. Compare Max with the Median in Summary Metrics to see it.
- Where is the physical plan in the Spark UI?
- In the SQL / DataFrame tab: open the query, and Plan Visualization draws the physical plan as boxes with rows and sizes, while Plan Details holds it as text, the Final Plan and the Initial Plan when adaptive execution is on. The Spark UI shows only the physical plan; for the logical plans, run df.explain("extended").
- How do I check a Spark setting took effect?
- Open the Environment tab and read Spark Properties. A setting you set is listed with its value; one that is not listed runs at its default. In the SQL tab, an Exchange's number of partitions shows the partition count the query really used.
Sources