Build it: CI/CD for a Databricks pipeline with GitHub Actions, pytest and bundles

Part 4 put the shop's job, its code and its tests in 1 bundle, and deployed it from a laptop. This part has GitHub do it on every change, in the order that keeps a broken change out of production: tests, then a development deploy for the pull request, then production when it is merged. This part uses part 4's shop folder, as a GitHub repository. To rebuild from nothing, start from part 1.

Build it: the pipeline for every change

The rule the workflow enforces is simple: nothing deploys unless the tests pass, and nothing reaches production unless it was reviewed and merged.

pull request ──► test ──► deploy to dev
merge to main ──► test ──► deploy to prod

A failed test stops both. The tests are part 1's, and they need no workspace, so they run on GitHub's own machines in minutes and cost nothing on Databricks.

Let GitHub sign in without a token

The deploy jobs need to sign in to Databricks. The old way was a personal access token saved as a GitHub secret: a long-lived password that can leak and has to be rotated. Workload identity federation replaces it. GitHub signs a short-lived token for each workflow run that says which repository and environment it is from, and Databricks exchanges it for its own short-lived token, if a federation policy on a service principal trusts that repository and environment.

Set it up once:

1. In the Databricks account, create a service principal for each target, such as shop-dev and shop-prod, and give each the rights its target needs: production's needs to write silver.orders_daily and read the landing volume. Put the production one's application ID in part 4's run_as.
2. Create a federation policy on each service principal for your GitHub organization, the repository and the entity type Environment, which Databricks recommends, with the environment dev or prod. The policy's subject must match the sub claim GitHub sends exactly, such as repo:<org>/<repo>:environment:prod. Repositories created, renamed or moved since 15 July 2026 use a form with numeric IDs; GitHub's OpenID Connect page shows how to find yours.
3. In the repository's settings, create 2 environments, dev and prod. In each, add a variable DATABRICKS_HOST with the workspace URL and a secret DATABRICKS_CLIENT_ID with that target's service principal application ID. On prod, add a rule that a reviewer approves each deployment, if your team wants a person to press the button.

No token is stored anywhere. A fork of the repository, or another repository, cannot sign in, because its tokens do not match the policy.

The workflow

Save this as .github/workflows/shop.yml:

name: shop

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

# Workload identity federation needs id-token: write.
permissions:
  id-token: write
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: "17"
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install . pyspark==4.2.0 pytest
      - run: python -m pytest tests

  deploy_dev:
    if: github.event_name == 'pull_request'
    needs: test
    runs-on: ubuntu-latest
    environment: dev
    env:
      DATABRICKS_AUTH_TYPE: github-oidc
      DATABRICKS_HOST: ${{ vars.DATABRICKS_HOST }}
      DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CLIENT_ID }}
    steps:
      - uses: actions/checkout@v4
      - uses: databricks/setup-cli@main
      - run: pip install uv
      - run: databricks bundle validate -t dev
      - run: databricks bundle deploy -t dev

  deploy_prod:
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    needs: test
    runs-on: ubuntu-latest
    environment: prod
    concurrency: prod
    env:
      DATABRICKS_AUTH_TYPE: github-oidc
      DATABRICKS_HOST: ${{ vars.DATABRICKS_HOST }}
      DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CLIENT_ID }}
    steps:
      - uses: actions/checkout@v4
      - uses: databricks/setup-cli@main
      - run: pip install uv
      - run: databricks bundle validate -t prod
      - run: databricks bundle deploy -t prod

What each part does:

• on starts the workflow for a pull request into main and for a push to main, which is what a merge is.
• test installs Java 17 for Spark, Python, the shop package from pyproject.toml, PySpark and pytest, then runs part 1's tests. Installing shop with pip install . is what lets the tests import it from src/.
• needs: test makes each deploy wait for the tests, and skips it if they fail.
• if picks the deploy: development for a pull request, production for a push to main.
• environment names the GitHub environment, which gives the job that environment's host and client ID, and is what the federation policy trusts.
• pip install uv is there because the bundle's wheel is built with uv build on the runner during the deploy.
• concurrency: prod lets only 1 production deploy run at a time, and production mode's own deployment lock guards the workspace as well.

What happens on a change

Open a pull request that changes orders_for_day. The test job runs first. If a test fails, the pull request shows the failed test and its rows, and neither deploy runs. If the tests pass, deploy_dev validates the bundle and deploys it to the dev workspace as the shop-dev service principal, prefixed with its name and with the schedule paused, so a reviewer can run that copy with databricks bundle run -t dev shop_daily --params run_date=2026-09-16 and read the result.

Merge the pull request, and the push to main runs the tests again on the merged code, then deploy_prod deploys the same bundle to production: the job named shop_daily, running as the production service principal, with its schedule on at 02:00 UTC. Nobody copied a script or clicked through the workspace, and the repository's history says who changed what and who approved it.

A deploy changes only what the bundle describes. Data is not rolled back by a deploy; if a bad version loaded wrong rows, revert the commit, let the workflow deploy the fixed version and rerun the affected days with module 10's backfill, which replaces each day it loads.

What you built

• Unit tests of the load's logic with pytest and assertDataFrameEqual, on rows made up for each case.
• A data contract checked before every load, and Delta constraints that refuse bad rows from any path.
• Quality metrics on every run, compared with the week before, to catch a valid file that is still wrong.
• The job, its code and its contract as 1 bundle with development and production targets.
• A workflow that tests every change, deploys it to development for review and to production when it is merged, with no stored token.

Module 12 turns to what the pipeline costs: serverless or classic compute, finding the slow step, and the bill for a month before it runs.

ToolDatabricks cost calculatorDBUs and machines for jobs, all-purpose and pipeline clusters, serverless and SQL warehouses, on AWS, Azure and Google Cloud.