Skip to the page
Tecyfy

PySpark interview question · hard

Is `wholeTextFiles` better than `CombineFileInputFormat` for handling many small files in PySpark?

The short answerNeither is generally "better"; they do different jobs.

The full answer

Neither is generally "better"; they do different jobs. wholeTextFiles reads each file as a single record, a (path, whole content) pair, which suits many small files that must each be processed as one unit, such as small JSON or XML documents. CombineFileInputFormat is a Hadoop input format that packs many small files into fewer input splits while still reading them record by record. In current Spark, you rarely need either directly, because the DataFrame readers already combine small files into partitions.

wholeTextFiles

rdd = spark.sparkContext.wholeTextFiles("/data/small_docs/", minPartitions=64)
# RDD of (path, content)
  • Good when each file is one logical record and the file name matters.
  • Spark already groups many files into each partition (its input format for this extends CombineFileInputFormat), so it doesn't create one task per file.
  • Each file must fit in memory, and it's an RDD API: not available on Spark Connect or serverless compute.
  • The DataFrame equivalents are spark.read.text(path, wholetext=True) (one row per file) and the binaryFile format, which give file paths and contents as columns.

CombineFileInputFormat

  • A Hadoop API (with subclasses such as CombineTextInputFormat) that combines blocks of many files into splits of a target size, so one task reads many small files line by line.
  • Used from Spark through newAPIHadoopFile with configuration such as the maximum split size. It means writing low-level Hadoop configuration and usually Scala or Java classes.
  • Useful in old Hadoop-format pipelines; little reason to use it in new code.
wholeTextFilesCombineFileInputFormat
Record unitOne whole fileOne line (or record)
Typical useSmall documents processed as a wholeLine-oriented data spread over many small files
MemoryWhole file in memoryStreams records
API levelSpark RDDHadoop input format
Modern replacementread.text(wholetext=True), binaryFileDataFrame readers with built-in file packing

What Spark does for you now For Parquet, ORC, JSON, CSV and text, DataFrame readers pack small files into partitions up to spark.sql.files.maxPartitionBytes (128 MB), counting spark.sql.files.openCostInBytes (4 MB) per file as the cost of opening it. So 100,000 files of 10 KB don't become 100,000 tasks.

The real fix Many small files are slow mainly because of listing and opening costs on object storage. The lasting solution is to compact them: write fewer, larger files (aim for roughly 128 MB to 1 GB), or store data in a table format whose compaction (OPTIMIZE in Delta Lake) rewrites small files into large ones. See the next question for strategies.

Last updated 16 October 2026