# Build it: run the data pipeline end to end and write its design document

To finish an end-to-end data pipeline project, deploy it from Git to a development copy, run 1 day and then a backfill of several, and check every output: the dashboard's numbers, the app's copy and an AI answer with its sources. Run the failure drills and record what caught each. Then write a design document of the requirements, the steps, the decisions with their reasons, the checks, the operations and the cost, which is also how you explain the project in an interview.

- Page: https://tecyfy.com/data-engineering/course/capstone/build-it
- Course: Data engineering course (https://tecyfy.com/data-engineering/course), Module 14: Final project, Build it: run it end to end and write it up
- Parts of this module: The brief and the pipeline (https://tecyfy.com/data-engineering/course/capstone); The production readiness review (https://tecyfy.com/data-engineering/course/capstone/review); Build it: run it end to end and write it up (https://tecyfy.com/data-engineering/course/capstone/build-it)

At the end of this part you can:

- Add task notifications, a duration warning and a timeout to a Databricks job
- Run a whole pipeline from Git in development, for 1 day and a backfill, and check each output
- Write a design document for a pipeline and explain it in 5 minutes

[Part 2](/data-engineering/course/capstone/review) listed what the shop's pipeline must prove, and the drills that break it on purpose. This part runs the whole pipeline, as the owner will rely on it, and writes it up. It needs the repository from [part 1](/data-engineering/course/capstone): module 11's bundle with the shop's job, module 8's serving tasks, module 13's `help_index` job, and module 5's catalog. Each of those modules' "Build it" parts rebuilds its piece.

## Build it: the alerts the review asked for

The review found 2 gaps in module 11's job: a failure that a retry hides reaches nobody, and nothing notices a run that is slow or hangs. Both are settings on the job in `databricks.yml`. On `load_day`, the task with retries, add a notification of its own:

```yaml
        - task_key: load_day
          # ... as in module 11, then:
          email_notifications:
            on_failure:
              - <your email>
```

On the job itself, beside `schedule`, add a failure notification, a duration warning and a timeout:

```yaml
      email_notifications:
        on_failure:
          - <your email>
        on_duration_warning_threshold_exceeded:
          - <your email>
      health:
        rules:
          - metric: RUN_DURATION_SECONDS
            op: GREATER_THAN
            value: 1800        # warn after 30 minutes
      timeout_seconds: 10800   # stop after 3 hours
```

Choose the numbers from the job's own runs. [Module 12's](/data-engineering/course/performance-and-cost/find-the-slow-step) timeline query gives the usual run time: the warning goes well above it, and the timeout well above that, but early enough to rerun before 07:00. Use an address or a channel that someone reads in the morning; Databricks can also send to Slack, Microsoft Teams or PagerDuty once an admin sets them up as destinations.

## Deploy and run 1 day

From the repository, on a branch, validate and deploy the development copy:

```bash
databricks bundle validate -t dev
databricks bundle deploy -t dev
databricks bundle run -t dev shop_daily --params run_date=2026-10-01
databricks bundle run -t dev help_index
```

Then check each output against the brief, not against the code:

- **The dashboard**: the day's sales by country match the gold table, and gold matches silver; [module 8's](/data-engineering/course/serving-data/build-it) checks pass.
- **The app**: the customer lookup has the day's orders for a customer you know, and `check_serving.py` passes.
- **The mask**: run a query on `silver.customers` as a user outside `pii_readers`, and the emails come back masked.
- **The assistant**: run [module 13's](/data-engineering/course/data-for-ai/build-it) answer query for 2 or 3 questions, and read each answer against its sources.

## Backfill a week

Load the days before it, the way a new pipeline fills its history:

```bash
for day in 2026-09-24 2026-09-25 2026-09-26 2026-09-27 2026-09-28 2026-09-29 2026-09-30; do
  databricks bundle run -t dev shop_daily --params run_date=$day
done
```

Then run [module 10's](/data-engineering/course/orchestration/build-it) month check, which lists any day with no load. Run 1 day of the week a second time and check that its sales did not change. That is the drill part 2 called the one that matters most, and now it has passed on the real job.

## Run the drills

Work through part 2's table on the development copy, 1 row at a time, and record each in a table of your own: what you did, what caught it, and the date. A drill that is not caught is not a failure of the exercise; it is the most useful thing it can find, and the fix goes in before production.

When every row passes, merge the branch. [Module 11's](/data-engineering/course/testing-and-ci-cd/build-it) workflow tests it and deploys production, and only production's schedule runs.

## Write the design document

The repository's `README.md` is the design document. Write it for the next engineer, who knows Databricks but not the shop:

```markdown
# Shop orders pipeline

## What it is for
The 7 requirements from the brief, as they are written in part 1.

## How data moves
The 11 steps, 1 line each: what it does, where it runs, which table it writes.

## Decisions
Each decision, its reason, and what would change it.

## Tables
Each table with its grain (1 row per what), its key, its owner, and who may read it.

## Checks
Each check, what it stops, and what happens when it fails.

## Running it
The schedule; how to rerun a day, backfill a month and repair a failed run; who is alerted.

## Cost
Cost per run and per month at list price, the budget, and when it was last measured.

## Drills
The table of drills with their results and dates.

## Known limits
What the pipeline does not do, such as data sooner than daily.
```

Every section is something the course built; the document only says where. Change it in the same pull request as the code it describes, or it is out of date the day after.

## Explain it in 5 minutes

The same document is the answer to "tell me about a project" in a [data engineering system design interview](/interview-questions-and-answers/data-engineer/system-design). In about 5 minutes: who it is for and 2 of the requirements; the flow, 1 sentence a step; 1 decision with its reason and its cost; 1 failure and how the pipeline catches it, with the drill that proved it; and what it costs a month. Then stop, and let the interviewer pick where to dig. They usually pick the failure.

## What you built

Across 15 modules, 1 shop's orders went from a line in a file to:

- A star schema with customer history, cleaned in PySpark, stored in Delta tables that take updates, keep versions and skip files.
- A catalog where analysts read gold, personal data stays masked, and lineage shows where every column came from.
- An incremental load of every file once, silver and gold built with expectations and a quality gate, and a stream from Kafka for when minutes matter.
- A dashboard, Genie, a metric view, a share and an app copy, each checked.
- A job that runs every night, retries safely, backfills a month and repairs a failed day.
- Tests, contracts, quality metrics and a bundle deployed from GitHub with no stored token.
- The cost of every run, and a budget.
- Help articles chunked and indexed, models called from SQL, and answers with their sources.

The [AI engineering course](/ai-engineering) starts where this one stops, building the applications that read these tables. For a certificate to show for the course, the [Databricks Data Engineer Associate guide](/blog/master-databricks-certified-data-engineer-associate-exam-ultimate-guide) covers what the exam asks.

- Interview: [System design interview questions for Data Engineer roles](https://tecyfy.com/interview-questions-and-answers/data-engineer/system-design). System design questions for Data Engineer roles, with answers, from easy to hard.

## Check yourself

1. In an interview, you are asked what happens if the shop's job runs twice for the same day. Which answer shows the most?

   A. It is fine, Delta Lake is ACID
   B. Retries are turned off, so it cannot happen
   C. Every task replaces the day it loads, so a second run leaves the same rows, and the drill of loading a day twice proved it on the development copy
   D. The dashboard removes duplicates

### Answers

1. C. ACID transactions keep each write whole, but a whole append of the same rows is still a duplicate. The answer that shows the most says how the pipeline makes a rerun safe, a write that replaces its day, and how you know it works, a drill you ran. Interviewers ask about failure because it shows whether the design was thought through.

## Questions

### What goes in a data pipeline design document?

The purpose and the requirements as testable lines; the steps from source to use, with where each runs; the decisions, each with its reason and what would change it; the tables with their grain and owner; the checks and what each stops; how to run, rerun, backfill and repair; who is alerted and how; who can read what; what it costs a run and a month; and the known limits. Keep it in the repository beside the code, and change it in the same pull request as the code.

### How do I explain a data engineering project in an interview?

In about 5 minutes, in this order. Who it is for and what they needed, as 2 or 3 requirements. The flow from source to use, 1 sentence a step. 1 decision and why, with what you gave up. 1 failure and how the pipeline catches it, with how you proved it. What it costs. Then let the interviewer choose where to go deeper.

## Sources

- [Add notifications on a job](https://docs.databricks.com/aws/en/jobs/notifications)
- [Best practices for reliability](https://docs.databricks.com/aws/en/lakehouse-architecture/reliability/best-practices)
- [Bundle commands](https://docs.databricks.com/aws/en/dev-tools/cli/bundle-commands)