The full answer
I treat partitioning on two levels: how data is split into Spark partitions while a job runs, which decides parallelism and memory per task, and how tables are laid out on storage, which decides what later queries can skip. The goal for both is evenly sized pieces of a sensible size, aligned with how the data is filtered and joined.
1. In-memory partitions during processing
Size: aim for roughly 100–200 MB per partition after a shuffle and around 128 MB per input split. Too few partitions leave cores idle and risk out-of-memory errors; too many add scheduling overhead and tiny files.
Input: Spark splits files according to spark.sql.files.maxPartitionBytes (128 MB by default) and packs small files together. Splittable formats (Parquet, ORC, uncompressed or bzip2 CSV) allow this; a single gzip file is read by one task.
Shuffles: spark.sql.shuffle.partitions (200 by default) sets the number of partitions after joins and aggregations. Adaptive query execution then merges small partitions toward a 64 MB target, so for most jobs I set a generous starting number for large data and let AQE coalesce.
Explicit changes:
repartition(n, "key")to spread data evenly or co-locate by key before heavy processing (it shuffles).coalesce(n)to reduce partitions without a shuffle, for example before writing fewer files.repartition("date")before a partitioned write so each task writes to few directories.
Parallelism check: the number of tasks per stage should be at least two to three times the total number of cores.
2. On-disk table layout
Partition columns: partition large tables by a column that most queries filter on and that has low cardinality, typically a date or month. Each partition should hold at least about 1 GB; partitioning by customer or by hour on modest data creates millions of small files.
File sizes: keep files in the range of about 128 MB to 1 GB. Compact small files (OPTIMIZE in Delta), and avoid writing many tiny files from too many partitions.
Beyond partitioning: for filters and joins on other columns, use clustering and data skipping: Z-ordering or liquid clustering in Delta, sort orders in Iceberg, or bucketing in Hive-style tables to avoid shuffles in repeated joins. Databricks now recommends liquid clustering, which replaces partitioning and Z-ordering, for all new tables.
3. Skew Check that partitions are even: one huge partition (a hot key or a very busy date) slows the whole stage. Use AQE skew handling, salting or separate processing for hot keys.
4. Verify
- In the Spark UI, task durations and input sizes within a stage should be similar.
df.rdd.getNumPartitions()(classic Spark) shows the current number.- After writing, check file counts and sizes per partition directory.
5. Example
(events
.repartition("event_date") # one shuffle, each task writes one date
.write.mode("overwrite")
.partitionBy("event_date")
.option("maxRecordsPerFile", 5_000_000)
.saveAsTable("silver.events"))
Partitioning is a design decision, so I revisit it when data volumes or query patterns change.
Last updated 16 October 2026
Sources