Skip to the page
Tecyfy

PySpark interview question · easy

What is the difference between `ORDER BY`, `SORT BY`, and `CLUSTER BY` in Spark?

The short answerORDER BY produces a total order of the whole result.

The full answer

ORDER BY produces a total order of the whole result. SORT BY only sorts rows within each partition, so the overall output may be out of order. CLUSTER BY col is shorthand for DISTRIBUTE BY col SORT BY col: it repartitions rows so equal values go to the same partition, then sorts within each partition.

-- 1. Total order: one global sort
SELECT * FROM sales ORDER BY amount DESC;

-- 2. Sorted within each partition only
SELECT * FROM sales SORT BY amount DESC;

-- 3. Same keys together, sorted within partitions
SELECT * FROM sales CLUSTER BY customer_id;
-- same as
SELECT * FROM sales DISTRIBUTE BY customer_id SORT BY customer_id;
ORDER BYSORT BYDISTRIBUTE BYCLUSTER BY
ShuffleYes: range partitioning across the clusterNoYes: hash partitioning by the expressionsYes: hash partitioning
SortGlobalWithin each partitionNoneWithin each partition
Output fully orderedYesOnly with one partitionNoNo
DirectionsASC or DESC per columnASC or DESC per columnAscending only
DataFrame APIorderBy, sortsortWithinPartitionsrepartition(col)repartition(col).sortWithinPartitions(col)

How ORDER BY works Spark samples the data to choose range boundaries, shuffles rows into ordered ranges, and sorts each range, so partition 1 holds the smallest values. It is the most expensive of the three; with LIMIT, Spark uses a cheaper top-N sort.

When to use each

  • ORDER BY for final results that people read or that must be ordered, preferably after aggregation, when data is small.
  • SORT BY when only per-partition order matters, such as preparing sorted output files where each file is sorted.
  • DISTRIBUTE BY or CLUSTER BY to group all rows with the same key in one partition before per-key processing, or so each written file contains whole keys, which improves compression and data skipping.

Pitfalls

  • Don't rely on SORT BY or CLUSTER BY for an ordered report: the output is only ordered within partitions.
  • Ordering isn't preserved by later shuffles; sort as the last step.
  • CLUSTER BY in a SELECT is unrelated to CLUSTER BY in CREATE TABLE, which (for Delta and some other table formats) defines liquid clustering, a storage layout.

Last updated 16 October 2026

Sources

Where each fact comes from

  1. 1ORDER BY clause
  2. 2SORT BY clause
  3. 3CLUSTER BY clause