How does Spark determine the schema of a CSV file, and what are the downsides of schema inference?

With inferSchema enabled, Spark reads the CSV data (all rows, or a sample set by samplingRatio) before the real read, and for each column picks the narrowest type that fits every sampled value: integer, long, decimal or double, boolean, date, timestamp, otherwise string. Without inferSchema, every column is a string. Column names come from the header, or _c0, _c1 and so on without one.

Downsides

1. An extra pass over the data Inference reads the files before processing them, which can double the read time on large data.

2. Wrong types

• Codes with leading zeros become numbers: zip code 01234 is read as the integer 1234, and the zero is gone.
• Money becomes double, which introduces rounding; financial data needs decimal.
• Dates in non-default formats stay strings unless you set dateFormat or timestampFormat.
• Large IDs can be read as numbers when they should be text.

3. Sampling can miss values With samplingRatio below 1, a column may look like integers in the sample and contain text elsewhere. Under ANSI mode (the default in Spark 4), or with mode set to FAILFAST, the job then fails, or in the default PERMISSIVE mode, the offending values become NULL.

4. Types change between runs A new file with one text value in a numeric column changes the inferred type, so downstream code and table writes break unpredictably.

5. Weak handling of bad data With inference there is no agreed contract to check rows against, so malformed records are harder to detect.

Better practice

schema = "order_id INT, order_date DATE, amount DECIMAL(10,2), zip STRING"
df = (spark.read
        .option("header", True)
        .option("mode", "PERMISSIVE")
        .schema(schema)
        .csv("/data/orders/"))

• Define the schema explicitly in pipelines; it is also faster.
• Infer once while exploring, then copy the result (df.schema.simpleString()) into the code and correct it.
• For changing sources, tools such as Databricks Auto Loader store an inferred schema and evolve it in a controlled way, with unexpected values kept in a rescued data column.