How does Spark manage Dynamic Partition Pruning (DPP) internally during query execution?

Internally, DPP is an optimizer rule that inserts a runtime partition filter on the large table, built from the join key values of the filtered small table, and a planner step that decides how to compute those values, normally by reusing the broadcast built for the join.

1. Logical optimisation The rule PartitionPruning looks for joins where one side is a partitioned table scanned with the join key as a partition column, and the other side has a filter. It adds a DynamicPruningSubquery predicate to the partitioned side, meaning "partition key IN (join keys of the filtered other side)".

2. Is it worth it? If the filtering side won't be broadcast, Spark would have to run that side twice, once for pruning and once for the join. It only does that when its statistics suggest the filter removes enough data; by default it prefers to apply DPP only when it can reuse a broadcast (spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly, an internal setting that defaults to true).

3. Physical planning

• If the join is a broadcast hash join with the filtered side as the build side, the predicate becomes a DynamicPruningExpression whose values come from a SubqueryBroadcast reusing that same broadcast exchange.
• If no broadcast can be reused (and the subquery isn't worth running), the predicate is replaced with true: pruning is off, and the plan shows dynamicpruningexpression(true).
• With adaptive query execution, the final decision is made when the join strategy is known at runtime.

4. Execution

1. The broadcast side runs first: the dimension table is scanned, filtered and broadcast.
2. The SubqueryBroadcast extracts the distinct join keys from it.
3. The fact table's file index applies them as partition filters, listing and reading only matching partition directories.
4. The join then runs on the reduced data, using the same broadcast hash table.

What you see in a plan

FileScan parquet sales[amount, date_id]
  PartitionFilters: [isnotnull(date_id), dynamicpruningexpression(date_id IN dynamicpruning#16)]
  ...
  +- SubqueryBroadcast dynamicpruning#16, [date_id]
     +- BroadcastExchange ...
        +- Filter (quarter = Q1)
           +- FileScan parquet dates[...]

Related runtime filters For non-partition columns, Spark can inject a Bloom filter or a semi-join filter into the other side of a shuffle join (spark.sql.optimizer.runtime.bloomFilter.enabled), which reduces rows rather than skipping partitions.