What is schema drift?

On Monday the orders arrive with a column called amount. On Tuesday the app team ships a release, and the column is now called total_amount. The pipeline still runs. It still asks for amount, which new rows no longer have, so it reads empty values and writes them on. The revenue report drops by a third and nothing has failed.

That is schema drift: the shape of incoming data changing without warning.

The forms it takes

• A new column. Usually harmless, sometimes important: the new column is where the discount now lives.
• A removed or renamed column. The dangerous one, because the pipeline may read empty values without failing.
• A changed type. A number arrives as text, or a whole number becomes a decimal.
• A changed meaning. The same column, the same type, but prices are now in cents. No schema check catches this one, only a quality ruleData QualityHow far data can be trusted: complete, correct, on time and in the expected shape. Pipelines check it with rules and stop or flag rows that fail..

The 3 responses: stop, absorb or set aside

When a pipeline meets a new shape, it can do 3 things, and good tools let you pick per table.

Stop. Refuse the data until a person decides. A Delta table on Databricks enforces its schema on write: a write with a column the table does not have fails, unless schema evolution was turned on for that write. Iceberg protects itself in another way: it tracks columns by id rather than by name, so renaming a column in the table does not break files already written.

Absorb. Accept the new shape and change the table. Auto Loader's default mode adds new columns to the schema, and its stream stops once with an error so the change is visible; on restart it carries on with the new columns.

Set aside. Keep the data flowing but put anything unexpected to one side. In Auto Loader's rescue mode the schema never changes; unexpected columns go into a rescued data column as a JSON blob, with the file each row came from. Nothing is lost, and nothing breaks.

Where to use which

A common split follows the medallionMedallion ArchitectureOrganising a lakehouse in 3 layers of rising quality: bronze for raw data, silver for cleaned and joined data, and gold for tables shaped for reports and AI. layers. In bronze, absorb or rescue: the job of bronze is to keep everything that arrived. In silver and gold, be strict: those tables have readers who depend on their columns, so a change there should stop and be looked at.

Better still is to hear about the change before it arrives, through a data contractData ContractAn agreement between the team that produces data and the teams that use it, written down and checked, that fixes its columns, meaning and freshness. with the team that owns the source.

Why it matters for AI

AI systems are built on the column names as they were. A text-to-SQLText-to-SQLA model turning a question in plain words into a database query, so people can ask the data directly. It is only as right as its knowledge of the tables and their meaning. assistant has instructions and example queries that name amount. A semantic layerSemantic LayerOne shared definition of business terms and metrics, such as revenue or active customer, that reports, dashboards and AI assistants all read instead of writing their own. defines revenue on amount. When the column is renamed upstream and the change flows through, those pieces do not fail loudly. They return empty or partial answers, and the assistant explains them as fact.