The full answer
Neither is generally "better"; they do different jobs. wholeTextFiles reads each file as a single record, a (path, whole content) pair, which suits many small files that must each be processed as one unit, such as small JSON or XML documents. CombineFileInputFormat is a Hadoop input format that packs many small files into fewer input splits while still reading them record by record. In current Spark, you rarely need either directly, because the DataFrame readers already combine small files into partitions.
wholeTextFiles
rdd = spark.sparkContext.wholeTextFiles("/data/small_docs/", minPartitions=64)
# RDD of (path, content)
- Good when each file is one logical record and the file name matters.
- Spark already groups many files into each partition (its input format for this extends
CombineFileInputFormat), so it doesn't create one task per file. - Each file must fit in memory, and it's an RDD API: not available on Spark Connect or serverless compute.
- The DataFrame equivalents are
spark.read.text(path, wholetext=True)(one row per file) and thebinaryFileformat, which give file paths and contents as columns.
CombineFileInputFormat
- A Hadoop API (with subclasses such as
CombineTextInputFormat) that combines blocks of many files into splits of a target size, so one task reads many small files line by line. - Used from Spark through
newAPIHadoopFilewith configuration such as the maximum split size. It means writing low-level Hadoop configuration and usually Scala or Java classes. - Useful in old Hadoop-format pipelines; little reason to use it in new code.
wholeTextFiles | CombineFileInputFormat | |
|---|---|---|
| Record unit | One whole file | One line (or record) |
| Typical use | Small documents processed as a whole | Line-oriented data spread over many small files |
| Memory | Whole file in memory | Streams records |
| API level | Spark RDD | Hadoop input format |
| Modern replacement | read.text(wholetext=True), binaryFile | DataFrame readers with built-in file packing |
What Spark does for you now
For Parquet, ORC, JSON, CSV and text, DataFrame readers pack small files into partitions up to spark.sql.files.maxPartitionBytes (128 MB), counting spark.sql.files.openCostInBytes (4 MB) per file as the cost of opening it. So 100,000 files of 10 KB don't become 100,000 tasks.
The real fix
Many small files are slow mainly because of listing and opening costs on object storage. The lasting solution is to compact them: write fewer, larger files (aim for roughly 128 MB to 1 GB), or store data in a table format whose compaction (OPTIMIZE in Delta Lake) rewrites small files into large ones. See the next question for strategies.
Last updated 16 October 2026
Sources