# Build it: CI/CD for a Databricks pipeline with GitHub Actions, pytest and bundles

A CI/CD workflow for a Databricks pipeline runs the unit tests on every pull request, deploys the bundle to a development target when they pass, and deploys it to production only when the change is merged to main. In GitHub Actions, install Java, Python, PySpark and pytest to run the tests, use databricks/setup-cli for the Databricks CLI, and sign in with workload identity federation: permissions id-token write, DATABRICKS_AUTH_TYPE github-oidc and the service principal's client ID, with no token stored in GitHub.

- Page: https://tecyfy.com/data-engineering/course/testing-and-ci-cd/build-it
- Course: Data engineering course (https://tecyfy.com/data-engineering/course), Module 11: Testing, CI/CD and bundles, Build it: tests and deploys in CI/CD
- Parts of this module: Unit tests for PySpark with pytest (https://tecyfy.com/data-engineering/course/testing-and-ci-cd); Data contracts, a check before the load (https://tecyfy.com/data-engineering/course/testing-and-ci-cd/data-contracts); Data quality over time (https://tecyfy.com/data-engineering/course/testing-and-ci-cd/data-quality); Declarative Automation Bundles, dev and prod (https://tecyfy.com/data-engineering/course/testing-and-ci-cd/bundles); Build it: tests and deploys in CI/CD (https://tecyfy.com/data-engineering/course/testing-and-ci-cd/build-it)

At the end of this part you can:

- Run PySpark unit tests in GitHub Actions on every pull request
- Deploy a bundle to development on a pull request and to production on a merge, only after the tests pass
- Sign GitHub Actions in to Databricks with workload identity federation instead of a stored token

[Part 4](/data-engineering/course/testing-and-ci-cd/bundles) put the shop's job, its code and its tests in 1 bundle, and deployed it from a laptop. This part has GitHub do it on every change, in the order that keeps a broken change out of production: tests, then a development deploy for the pull request, then production when it is merged. This part uses part 4's `shop` folder, as a GitHub repository. To rebuild from nothing, start from part 1.

## Build it: the pipeline for every change

The rule the workflow enforces is simple: nothing deploys unless the tests pass, and nothing reaches production unless it was reviewed and merged.

```text
pull request ──► test ──► deploy to dev
merge to main ──► test ──► deploy to prod
```

A failed test stops both. The tests are part 1's, and they need no workspace, so they run on GitHub's own machines in minutes and cost nothing on Databricks.

## Let GitHub sign in without a token

The deploy jobs need to sign in to Databricks. The old way was a personal access token saved as a GitHub secret: a long-lived password that can leak and has to be rotated. **Workload identity federation** replaces it. GitHub signs a short-lived token for each workflow run that says which repository and environment it is from, and Databricks exchanges it for its own short-lived token, if a **federation policy** on a service principal trusts that repository and environment.

Set it up once:

1. In the Databricks account, create a service principal for each target, such as `shop-dev` and `shop-prod`, and give each the rights its target needs: production's needs to write `silver.orders_daily` and read the landing volume. Put the production one's application ID in part 4's `run_as`.
2. Create a federation policy on each service principal for your GitHub organization, the repository and the entity type **Environment**, which Databricks recommends, with the environment `dev` or `prod`. The policy's subject must match the `sub` claim GitHub sends exactly, such as `repo:<org>/<repo>:environment:prod`. Repositories created, renamed or moved since 15 July 2026 use a form with numeric IDs; GitHub's OpenID Connect page shows how to find yours.
3. In the repository's settings, create 2 environments, `dev` and `prod`. In each, add a variable `DATABRICKS_HOST` with the workspace URL and a secret `DATABRICKS_CLIENT_ID` with that target's service principal application ID. On `prod`, add a rule that a reviewer approves each deployment, if your team wants a person to press the button.

No token is stored anywhere. A fork of the repository, or another repository, cannot sign in, because its tokens do not match the policy.

## The workflow

Save this as `.github/workflows/shop.yml`:

```yaml
name: shop

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

# Workload identity federation needs id-token: write.
permissions:
  id-token: write
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: "17"
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install . pyspark==4.2.0 pytest
      - run: python -m pytest tests

  deploy_dev:
    if: github.event_name == 'pull_request'
    needs: test
    runs-on: ubuntu-latest
    environment: dev
    env:
      DATABRICKS_AUTH_TYPE: github-oidc
      DATABRICKS_HOST: ${{ vars.DATABRICKS_HOST }}
      DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CLIENT_ID }}
    steps:
      - uses: actions/checkout@v4
      - uses: databricks/setup-cli@main
      - run: pip install uv
      - run: databricks bundle validate -t dev
      - run: databricks bundle deploy -t dev

  deploy_prod:
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    needs: test
    runs-on: ubuntu-latest
    environment: prod
    concurrency: prod
    env:
      DATABRICKS_AUTH_TYPE: github-oidc
      DATABRICKS_HOST: ${{ vars.DATABRICKS_HOST }}
      DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CLIENT_ID }}
    steps:
      - uses: actions/checkout@v4
      - uses: databricks/setup-cli@main
      - run: pip install uv
      - run: databricks bundle validate -t prod
      - run: databricks bundle deploy -t prod
```

What each part does:

- **`on`** starts the workflow for a pull request into `main` and for a push to `main`, which is what a merge is.
- **`test`** installs Java 17 for Spark, Python, the `shop` package from `pyproject.toml`, PySpark and pytest, then runs part 1's tests. Installing `shop` with `pip install .` is what lets the tests import it from `src/`.
- **`needs: test`** makes each deploy wait for the tests, and skips it if they fail.
- **`if`** picks the deploy: development for a pull request, production for a push to `main`.
- **`environment`** names the GitHub environment, which gives the job that environment's host and client ID, and is what the federation policy trusts.
- **`pip install uv`** is there because the bundle's wheel is built with `uv build` on the runner during the deploy.
- **`concurrency: prod`** lets only 1 production deploy run at a time, and production mode's own deployment lock guards the workspace as well.

## What happens on a change

Open a pull request that changes `orders_for_day`. The `test` job runs first. If a test fails, the pull request shows the failed test and its rows, and neither deploy runs. If the tests pass, `deploy_dev` validates the bundle and deploys it to the dev workspace as the `shop-dev` service principal, prefixed with its name and with the schedule paused, so a reviewer can run that copy with `databricks bundle run -t dev shop_daily --params run_date=2026-09-16` and read the result.

Merge the pull request, and the push to `main` runs the tests again on the merged code, then `deploy_prod` deploys the same bundle to production: the job named `shop_daily`, running as the production service principal, with its schedule on at 02:00 UTC. Nobody copied a script or clicked through the workspace, and the repository's history says who changed what and who approved it.

A deploy changes only what the bundle describes. Data is not rolled back by a deploy; if a bad version loaded wrong rows, revert the commit, let the workflow deploy the fixed version and rerun the affected days with [module 10's backfill](/data-engineering/course/orchestration/backfill), which replaces each day it loads.

## What you built

- Unit tests of the load's logic with pytest and `assertDataFrameEqual`, on rows made up for each case.
- A data contract checked before every load, and Delta constraints that refuse bad rows from any path.
- Quality metrics on every run, compared with the week before, to catch a valid file that is still wrong.
- The job, its code and its contract as 1 bundle with development and production targets.
- A workflow that tests every change, deploys it to development for review and to production when it is merged, with no stored token.

Module 12 turns to what the pipeline costs: serverless or classic compute, finding the slow step, and the bill for a month before it runs.

- Tool: [Databricks cost calculator](https://tecyfy.com/data-engineering/databricks-cost). DBUs and machines for jobs, all-purpose and pipeline clusters, serverless and SQL warehouses, on AWS, Azure and Google Cloud.

## Check yourself

1. A pull request changes orders_for_day, and 1 unit test fails in the workflow. What happens to the 2 deploy jobs?

   A. Both run, because deploys do not depend on tests
   B. Neither runs; both need the test job, and production also waits for a merge to main
   C. The development deploy runs and the production deploy waits
   D. The production deploy runs, because main has not changed

2. Why does the workflow sign in with workload identity federation rather than a Databricks token saved as a GitHub secret?

   A. GitHub Actions cannot store secrets
   B. Tokens do not work with bundles
   C. There is no long-lived secret to leak or rotate; GitHub gives each run a short-lived token that Databricks trusts only for this repository and environment
   D. It makes the deploy faster

### Answers

1. B. Both deploy jobs say needs test, so GitHub Actions skips them when the test job fails. The production job also runs only on a push to main, which a pull request is not. A broken change reaches no workspace, and the pull request shows the failed test.
2. C. With federation, Databricks trusts tokens that GitHub signs for 1 repository and environment, and exchanges each for a short-lived Databricks token. Nothing secret is stored, so nothing can leak from the repository's settings, and nothing needs rotating.

## Questions

### How do I set up CI/CD for Databricks with GitHub Actions?

Keep the project as a bundle with its tests. Add a workflow that runs the tests on every pull request, then runs databricks bundle deploy -t dev, and a job that runs databricks bundle deploy -t prod on a push to main. Install the CLI with the databricks/setup-cli action and sign in with workload identity federation: give the workflow permissions id-token: write, set DATABRICKS_AUTH_TYPE to github-oidc, DATABRICKS_HOST and DATABRICKS_CLIENT_ID, and create a federation policy for the service principal that names your repository and environment.

### Can PySpark tests run in GitHub Actions without Databricks?

Yes. Unit tests that build their own small DataFrames need only Java, Python and the pyspark package on the runner, and start a local Spark session. They do not touch a workspace, so they cost nothing on Databricks and run on every pull request in about the time the packages take to install.

## Sources

- [GitHub Actions](https://docs.databricks.com/aws/en/dev-tools/ci-cd/github)
- [Enable workload identity federation for GitHub Actions](https://docs.databricks.com/aws/en/dev-tools/auth/provider-github)
- [CI/CD on Databricks](https://docs.databricks.com/aws/en/dev-tools/ci-cd/)
- [Declarative Automation Bundles deployment modes](https://docs.databricks.com/aws/en/dev-tools/bundles/deployment-modes)
- [Testing PySpark](https://spark.apache.org/docs/latest/api/python/getting_started/testing_pyspark.html)