Build it: run the data pipeline end to end and write its design document

Part 2 listed what the shop's pipeline must prove, and the drills that break it on purpose. This part runs the whole pipeline, as the owner will rely on it, and writes it up. It needs the repository from part 1: module 11's bundle with the shop's job, module 8's serving tasks, module 13's help_index job, and module 5's catalog. Each of those modules' "Build it" parts rebuilds its piece.

Build it: the alerts the review asked for

The review found 2 gaps in module 11's job: a failure that a retry hides reaches nobody, and nothing notices a run that is slow or hangs. Both are settings on the job in databricks.yml. On load_day, the task with retries, add a notification of its own:

        - task_key: load_day
          # ... as in module 11, then:
          email_notifications:
            on_failure:
              - <your email>

On the job itself, beside schedule, add a failure notification, a duration warning and a timeout:

      email_notifications:
        on_failure:
          - <your email>
        on_duration_warning_threshold_exceeded:
          - <your email>
      health:
        rules:
          - metric: RUN_DURATION_SECONDS
            op: GREATER_THAN
            value: 1800        # warn after 30 minutes
      timeout_seconds: 10800   # stop after 3 hours

Choose the numbers from the job's own runs. Module 12's timeline query gives the usual run time: the warning goes well above it, and the timeout well above that, but early enough to rerun before 07:00. Use an address or a channel that someone reads in the morning; Databricks can also send to Slack, Microsoft Teams or PagerDuty once an admin sets them up as destinations.

Deploy and run 1 day

From the repository, on a branch, validate and deploy the development copy:

databricks bundle validate -t dev
databricks bundle deploy -t dev
databricks bundle run -t dev shop_daily --params run_date=2026-10-01
databricks bundle run -t dev help_index

Then check each output against the brief, not against the code:

• The dashboard: the day's sales by country match the gold table, and gold matches silver; module 8's checks pass.
• The app: the customer lookup has the day's orders for a customer you know, and check_serving.py passes.
• The mask: run a query on silver.customers as a user outside pii_readers, and the emails come back masked.
• The assistant: run module 13's answer query for 2 or 3 questions, and read each answer against its sources.

Backfill a week

Load the days before it, the way a new pipeline fills its history:

for day in 2026-09-24 2026-09-25 2026-09-26 2026-09-27 2026-09-28 2026-09-29 2026-09-30; do
  databricks bundle run -t dev shop_daily --params run_date=$day
done

Then run module 10's month check, which lists any day with no load. Run 1 day of the week a second time and check that its sales did not change. That is the drill part 2 called the one that matters most, and now it has passed on the real job.

Run the drills

Work through part 2's table on the development copy, 1 row at a time, and record each in a table of your own: what you did, what caught it, and the date. A drill that is not caught is not a failure of the exercise; it is the most useful thing it can find, and the fix goes in before production.

When every row passes, merge the branch. Module 11's workflow tests it and deploys production, and only production's schedule runs.

Write the design document

The repository's README.md is the design document. Write it for the next engineer, who knows Databricks but not the shop:

# Shop orders pipeline

## What it is for
The 7 requirements from the brief, as they are written in part 1.

## How data moves
The 11 steps, 1 line each: what it does, where it runs, which table it writes.

## Decisions
Each decision, its reason, and what would change it.

## Tables
Each table with its grain (1 row per what), its key, its owner, and who may read it.

## Checks
Each check, what it stops, and what happens when it fails.

## Running it
The schedule; how to rerun a day, backfill a month and repair a failed run; who is alerted.

## Cost
Cost per run and per month at list price, the budget, and when it was last measured.

## Drills
The table of drills with their results and dates.

## Known limits
What the pipeline does not do, such as data sooner than daily.

Every section is something the course built; the document only says where. Change it in the same pull request as the code it describes, or it is out of date the day after.

Explain it in 5 minutes

The same document is the answer to "tell me about a project" in a data engineering system design interview. In about 5 minutes: who it is for and 2 of the requirements; the flow, 1 sentence a step; 1 decision with its reason and its cost; 1 failure and how the pipeline catches it, with the drill that proved it; and what it costs a month. Then stop, and let the interviewer pick where to dig. They usually pick the failure.

What you built

Across 15 modules, 1 shop's orders went from a line in a file to:

• A star schema with customer history, cleaned in PySpark, stored in Delta tables that take updates, keep versions and skip files.
• A catalog where analysts read gold, personal data stays masked, and lineage shows where every column came from.
• An incremental load of every file once, silver and gold built with expectations and a quality gate, and a stream from Kafka for when minutes matter.
• A dashboard, Genie, a metric view, a share and an app copy, each checked.
• A job that runs every night, retries safely, backfills a month and repairs a failed day.
• Tests, contracts, quality metrics and a bundle deployed from GitHub with no stored token.
• The cost of every run, and a budget.
• Help articles chunked and indexed, models called from SQL, and answers with their sources.

The AI engineering course starts where this one stops, building the applications that read these tables. For a certificate to show for the course, the Databricks Data Engineer Associate guide covers what the exam asks.

InterviewSystem design interview questions for Data Engineer rolesSystem design questions for Data Engineer roles, with answers, from easy to hard.