What is a production readiness checklist for a data pipeline?

A list of what the pipeline must prove before people rely on it, each item with a way to check it. It covers correctness (contracts, quality rules, checks that compare tables), reruns and backfills, alerts and timeouts, access to personal data, tested and automated deployment, cost per run and a budget, and documentation. Each item is proved by running something, not by reading the code.

How do I test that a data pipeline handles failures?

Break it on purpose in a development copy, 1 failure at a time, and watch what happens. Drop a file with the wrong columns, load a day twice, skip a day, add a bad row, remove a sync, and check that each is stopped or found by the guard meant for it, and that someone is told. A failure nobody notices is worse than one that stops the job.

Data pipeline production readiness checklist: what to prove before go-live

Before a pipeline goes live, prove each promise it makes rather than trusting it: that bad data is stopped and reported, that any day can be rerun without duplicates, that a failure or a slow run alerts someone, that access rules hold for every table, that only tested code reaches production, that its cost is known and budgeted, and that the data feeding AI is current. Then break it on purpose, 1 failure at a time, and check that the right guard catches each one.

Module 14: Final project

At the end of this part you can:

Review a pipeline against a production readiness checklist, with a way to prove each item
Break a pipeline on purpose and check that each failure is caught where it should be
Set up alerts that reach someone for a failed or a slow run, including after retries

Take this module into your AI

ChatGPT Claude

Part 1 wrote the shop's brief and laid out the pipeline from raw files to a dashboard and an AI answer. A pipeline that runs is not yet one people can rely on. This part is the review before the owner does: each promise in the brief, and how to prove it. It needs the pieces from the earlier modules, each rebuilt by its own "Build it" part, and works best on the development copy that module 11 deploys.

Prove it, don't read it

Every item below is checked by running something and reading the result, not by reading the code and agreeing with it. Code that looks right and data that is right are different things, which is why each module ended with a check of its own.

Correct

A bad file is stopped before bronze. The contract check fails on a file with a missing column or a wrong type; module 11.
A file's columns are read by its header, or it fails. A file with 2 columns swapped either lands correctly or stops the load, never lands in the wrong columns; module 6.
Bronze holds every file, once. The load's own check compares the table's rows with the files'; module 6.
Bad rows are kept, with the reason. Rows that break a rule go to a quarantine table with the rules they broke, and a gate stops the run when too many do; module 7.
Tables agree with each other. Silver, gold and the serving tables add up to the same sales; module 7's check and module 8's.
A valid file that is still wrong is noticed. Each run's row count and totals are compared with the week before; module 11.

Safe to run again

Every task replaces its day. Loading a day twice leaves the same rows; module 10.
Any month can be backfilled, and a missing day found and filled alone; module 10.
A failed run can be repaired from the failed task, without rerunning what passed; module 10.

Watched

A failure reaches a person. The job notifies on failure, to an address or a channel someone reads; module 10.
A failure that a retry hides is still seen. Job-level notifications are not sent when a failed task is retried, so the tasks with retries get their own failure notification.
A slow run is noticed. An expected duration with a warning, and a timeout so a hanging task cannot run all day; Databricks warns that a hanging task can stop the whole job and run up costs.
A step that did not run fails the job. The serving check fails if the app's copy was not synced; module 8.

Secure

Analysts read gold and nothing else. Grants by group, checked by the catalog script; module 5.
No full email outside the mask. The catalog script fails on any gold view that shows one; module 5.
Production runs as a service principal, not as a person, and holds no stored token; module 11.
Secrets are not in the repository. Connection settings come from secrets or the deployment's variables.

Shipped from Git

Every change is tested: unit tests of the logic in CI on each pull request; module 11.
Production runs only merged code, deployed by the workflow, never from a notebook; module 11.
Development and production are separate, and only production's schedule runs; module 11.

Affordable

The compute fits the job: serverless in standard mode, or classic for a reason written down; module 12.
The cost of a run and of a month is known, from the billing tables, and a budget alerts above it; module 12.

Ready for AI

The help index follows the articles: a change to 1 article re-embeds 1 chunk, and a removed section leaves the index; module 13.
Retrieval is measured on a set of test questions, and the measure is kept for the next change; module 13.
AI Functions process each row once, so a nightly run does not pay again for old rows; module 13.

Break it on purpose

A guard that has never fired is a guess. On the development copy, cause each failure below, 1 at a time, and check that it is caught where the table says. Put everything back before the next.

Do this	It should be caught by	Built in
Drop a file with a column missing	The contract check, before the load	Module 11
Drop a file with 2 columns swapped	The header check, or Auto Loader reading by header	Module 6
Add an order with a negative amount	The quarantine table, and the gate if there are many	Module 7
Write a negative amount to silver directly, around the pipeline	The Delta constraint, which refuses the write	Module 11
Load the same day twice	Nothing: the day's sales are unchanged	Module 10
Skip a day's run	The month check, then a backfill of that day alone	Module 10
Add a gold view that selects the email	The catalog script, which stops	Module 5
Make a task fail once, then pass	The task's own failure notification	This part
Change 1 help article	The change feed: 1 update, 1 chunk re-embedded	Module 13

The fifth row is the one teams skip, and the one that matters most: a backfill is only safe if a second load of a day changes nothing.

Write down what happened for each drill, with the date: it becomes part of the design document in part 3, and the proof that requirement 3, 4 and the others hold.

What the checklist leaves out

The list is for this pipeline, a daily batch job for 1 shop. A stream adds its own items, such as a lag alert and a checkpoint that must not be lost, from module 9. A pipeline with more teams adds ownership: a name beside every table, and someone to answer for each contract. Add the items your pipeline needs, and keep each one provable.

InterviewSystem design interview questions for Data Engineer rolesSystem design questions for Data Engineer roles, with answers, from easy to hard.

Check yourself

2 questions, each in a new situation

1. The shop's job has an email on failure at the job level. load_day fails at 02:10, its retry succeeds at 02:15, and the run completes. Who hears about the failure?
Not quite. Databricks does not send job-level notifications when failed tasks are retried, and a run that ends well is a success. That is often what you want at 02:00, but a task that fails every night and passes on its retry is a problem growing quietly. A failure notification on the task reports each failed attempt; the run timeline from module 12 shows the pattern.
2. A drill loads 2026-10-01 twice. The second run leaves sales for that day at twice the first. What does that say?
Not quite. Loading the same day twice must leave the same rows, which is what makes a rerun or a backfill safe. A total that doubles means a write added the day's rows again instead of replacing them, the mistake module 10 removed with a write that replaces exactly the day being loaded.

Questions

What is a production readiness checklist for a data pipeline?: A list of what the pipeline must prove before people rely on it, each item with a way to check it. It covers correctness (contracts, quality rules, checks that compare tables), reruns and backfills, alerts and timeouts, access to personal data, tested and automated deployment, cost per run and a budget, and documentation. Each item is proved by running something, not by reading the code.
How do I test that a data pipeline handles failures?: Break it on purpose in a development copy, 1 failure at a time, and watch what happens. Drop a file with the wrong columns, load a day twice, skip a day, add a bad row, remove a sync, and check that each is stopped or found by the guard meant for it, and that someone is told. A failure nobody notices is worse than one that stops the job.

Sources