The source team added fields to the JSON files your Auto Loader stream reads. Overnight the stream failed, and in another pipeline the new field never appeared in the table. Why the different behaviour, and how should you set it up?

The two streams used different schema evolution modes, most likely without anyone choosing them. Auto Loader's default, when you let it infer the schema, is to add the new column and then stop the stream with an UnknownFieldException, so the next start picks up the new schema. When you give it a schema, the default changes to ignoring new columns, so the field silently never arrives. Choose the mode on purpose for each stream and plan for the restart.

The modes (cloudFiles.schemaEvolutionMode)

How to set it up

• Run the stream as a job with retries. With addNewColumns, the failure is expected: the first run records the new schema in the schema location, the retry continues. A stream started by hand from a notebook just stays down.
• Always keep a rescued data column. It catches new columns, type mismatches and fields whose case differs from the schema, as JSON with the source file path, instead of dropping them. With an explicit schema, set rescuedDataColumn.
• Use schema hints, not a full schema, for the columns you care about. JSON is inferred as strings; hints such as amount decimal(18,2), event_ts timestamp fix the types you depend on while still letting new columns evolve. A full schema forces none or rescue.
• Alert on rescued data. A daily count of rows with a non-null _rescued_data tells you the source changed before a user does.

Bronze and silver are different decisions

Bronze should keep everything, so addNewColumns or rescue with job retries fits. Silver tables that downstream reports depend on should change on purpose: read the new column from bronze when you have decided what it means, and add it to silver with a reviewed change.

What interviewers want: knowing the modes and that the default depends on whether a schema was given, that the addNewColumns failure is designed to be restarted by a job, the rescued data column, schema hints, and a bronze-to-silver split for schema decisions.