The full answer
Let the new column in on purpose: enable schema evolution for that write (mergeSchema, or automatic schema evolution for MERGE), or use Auto Loader's schema evolution, and check the new data's quality before it reaches downstream tables.
Why the write fails. Delta Lake enforces the table's schema on write. A DataFrame with a column the table does not have is rejected, which protects the table from accidental changes.
1. Evolve the schema on append
(new_data.write
.option("mergeSchema", "true")
.mode("append")
.saveAsTable("sales.orders"))
New columns are added to the end of the schema; existing rows read them as NULL. Incompatible changes, such as a string arriving where the table has an integer, still fail. (Databricks can also widen some types, such as INT to BIGINT, when type widening is enabled on the table.)
For MERGE INTO, use MERGE WITH SCHEMA EVOLUTION INTO ... (or enable automatic schema evolution) so new source columns are added.
2. Let Auto Loader handle it at ingestion
(spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("cloudFiles.schemaLocation", "/Volumes/main/raw/_schemas/orders")
.load("/Volumes/main/raw/orders/"))
Auto Loader stores the inferred schema in schemaLocation. By default (addNewColumns), when a new column appears the stream stops once, records the new schema, and continues after a restart (jobs can restart automatically). Values that do not fit the schema go into a _rescued_data column instead of being lost. Other modes are rescue (never change the schema, keep new fields in _rescued_data) and failOnNewColumns.
3. Keep quality in check
- Land new columns in bronze with evolution on, and decide in silver whether to use them.
- Add expectations or
CHECKconstraints for the columns that matter. - Alert on schema changes, for example by watching
DESCRIBE HISTORYfor metadata changes or the Auto Loader schema log.
When not to evolve: for curated silver and gold tables that others depend on, keep an explicit schema and change it deliberately with ALTER TABLE ... ADD COLUMN, so a source change cannot silently reshape a table people report from.
Last updated 16 October 2026
Sources