Auto Loader's schema inference. Does it handle schema drift automatically for all data formats?

No, and the reason is not the file format alone. Auto Loader infers and evolves schemas for JSON, CSV, XML, Avro and Parquet, but by default it only adds new columns: it does not change an existing column's type, and it stops the stream once each time it adds a column. Text and binary files have a fixed schema, so there is nothing to evolve.

How it works for each format

• JSON, CSV and XML have no types in the file, so Auto Loader infers every column as a string, including nested JSON fields, unless you set cloudFiles.inferColumnTypes to true. Strings make type changes impossible, which is why it is the safe default.
• Avro and Parquet carry their own schema. If two files give a column different types, Auto Loader picks the wider type; a value that cannot fit goes to the rescued data column.
• Text and binaryFile always have the same fixed columns.

The schema evolution modes (cloudFiles.schemaEvolutionMode):

Example

(spark.readStream.format("cloudFiles")
   .option("cloudFiles.format", "json")
   .option("cloudFiles.schemaLocation", "/Volumes/main/raw/_schemas/events")
   .option("cloudFiles.inferColumnTypes", "true")
   .option("cloudFiles.schemaHints", "amount DECIMAL(10,2)")
   .load("/Volumes/main/raw/events/"))

schemaLocation is where Auto Loader keeps the schema and its history. Schema hints fix the types of the columns you care about while the rest is inferred.

In practice: run the stream as a Lakeflow Job or pipeline with retries, so the planned stop on a new column restarts by itself; keep the _rescued_data column in bronze and alert when it fills; and decide in silver which new columns to use.