Databricks Asset Bundles: deploy a job from databricks.yml to dev and prod

Part 3 finished the job's steps: the contract check, the load, the day's check and the quality check. In module 10 the job was made by hand in the workspace, with its scripts copied in. That works once. It does not tell anyone what changed, cannot be reviewed, and cannot be made again the same way in another workspace. A bundle fixes that: the job is a file beside its code, and 1 command deploys both. This part uses parts 1 to 3's files. To rebuild from nothing, start from part 1.

What a bundle is

A Declarative Automation Bundle, formerly called a Databricks Asset Bundle, is a folder with a databricks.yml at its top. The file names the project, the Databricks resources it makes, such as jobs and pipelines, and the targets it deploys to, such as a development and a production workspace. The Databricks CLI reads it: databricks bundle validate checks it, deploy copies the code and creates or updates the resources, and run starts a job. Because everything is in files, a change to the job is a change in Git, reviewed like any other.

The shop's bundle:

shop/
├── databricks.yml          the bundle: the job and its targets
├── pyproject.toml          builds src/shop as a Python wheel
├── contracts/orders.yaml   part 2's contract
├── src/
│   ├── shop/
│   │   ├── __init__.py
│   │   └── orders.py       part 1's functions
│   ├── check_contract.py
│   ├── load_day.py
│   ├── check_day.py
│   └── quality.py
└── tests/
    ├── conftest.py
    └── test_orders.py

The 4 scripts are the task entry points, with module 10 part 2's changes for Databricks: the volume path for the files, silver.orders_daily for the table, and SparkSession.builder.getOrCreate() for the session.

Package the shared code

The scripts import shop.orders. On serverless compute the working folder of a task is not guaranteed, so a script cannot count on finding shop beside it. The reliable way is to build shop as a Python wheel and install it in the job's environment. Save this as pyproject.toml:

[project]
name = "shop"
version = "0.1.0"
requires-python = ">=3.10"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.hatch.build.targets.wheel]
packages = ["src/shop"]

uv build --wheel turns it into a file in dist/. The bundle runs that build on every deploy.

The bundle file

Save this as databricks.yml. The job is module 10's, with part 2's contract check before the load and part 3's quality check after it:

bundle:
  name: shop

# Build src/shop as a wheel on every deploy; the job's environment installs it.
artifacts:
  shop:
    type: whl
    build: uv build --wheel
    path: .

resources:
  jobs:
    shop_daily:
      name: shop_daily
      schedule:
        quartz_cron_expression: "0 0 2 * * ?"
        timezone_id: UTC
      max_concurrent_runs: 1
      parameters:
        - name: run_date
          default: "{{job.trigger.time.iso_date}}"
      tasks:
        - task_key: check_contract
          spark_python_task:
            python_file: src/check_contract.py
            parameters: ["{{job.parameters.run_date}}", "${workspace.file_path}/contracts/orders.yaml"]
          environment_key: default
        - task_key: load_day
          depends_on: [{ task_key: check_contract }]
          spark_python_task:
            python_file: src/load_day.py
            parameters: ["{{job.parameters.run_date}}"]
          environment_key: default
          max_retries: 2
          min_retry_interval_millis: 300000
        - task_key: check_day
          depends_on: [{ task_key: load_day }]
          spark_python_task:
            python_file: src/check_day.py
            parameters: ["{{job.parameters.run_date}}"]
          environment_key: default
        - task_key: quality
          depends_on: [{ task_key: check_day }]
          spark_python_task:
            python_file: src/quality.py
            parameters: ["{{job.parameters.run_date}}"]
          environment_key: default
      environments:
        - environment_key: default
          spec:
            environment_version: "2"
            dependencies:
              - dist/*.whl
              - pyyaml

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: <your workspace URL>
  prod:
    mode: production
    workspace:
      host: <your workspace URL>
      root_path: /Workspace/Shared/.bundle/${bundle.name}/${bundle.target}
    run_as:
      service_principal_name: <the service principal's application ID>
    # Only production's schedule runs. A pause_status in the job itself would unpause dev too.
    resources:
      jobs:
        shop_daily:
          schedule:
            quartz_cron_expression: "0 0 2 * * ?"
            timezone_id: UTC
            pause_status: UNPAUSED

What changed from the job of module 10:

• python_file paths are relative to databricks.yml. The bundle copies the folder to the workspace on deploy, so nobody copies scripts by hand.
• The contract's path is ${workspace.file_path}/contracts/orders.yaml, a substitution the CLI fills in with the folder it deployed the files to. {{job.parameters.run_date}} is different: a dynamic value that Databricks fills in when the job runs.
• The environment installs the shop wheel and PyYAML for the contract check.
• The 2 targets deploy the same job to 2 places, in 2 modes. A target can also change a resource: prod gives the schedule again with pause_status: UNPAUSED, so only production's copy is switched on in the file.

Development and production

mode: development is for your own copy. When you deploy to it, the CLI puts [dev <your name>] in front of the job's name, tags it dev, and pauses its schedule, so your copy never runs on its own at 02:00 and never clashes with production's. The exception is a schedule that says pause_status: UNPAUSED itself, which development mode leaves running; that is why the job leaves it out and only the prod target sets it. It also allows concurrent runs, unless the job says otherwise, as this one does with max_concurrent_runs: 1.

mode: production is for the real job. It checks that you deploy from the Git branch the target names, if it names one, and Databricks recommends running production jobs as a service principal, an identity for automation rather than a person, set with run_as. Then the job does not stop working when the person who deployed it leaves. root_path puts the deployed files in a shared folder rather than in a user's.

Validate, deploy and run

Install the Databricks CLI, version 0.218.0 or later. On a Mac with Homebrew:

brew tap databricks/tap
brew trust databricks/tap
brew install databricks

Sign in to your workspace once, with databricks auth login --host <your workspace URL>, which opens the browser. Then, from the bundle's folder:

databricks bundle validate
databricks bundle deploy -t dev
databricks bundle run -t dev shop_daily

validate reads every file of the bundle and prints a summary of what it would deploy, or the errors. deploy builds the wheel, uploads the files and creates the job, or updates it on the next deploy. run starts the deployed job, waits for it and prints a link to the run. For a run of 1 day, pass the date: databricks bundle run -t dev shop_daily --params run_date=2026-09-16. databricks bundle destroy -t dev removes your copy when you are done.

Nobody runs deploy -t prod from a laptop. Part 5 has CI run the tests on every pull request and deploy to production when a change is merged.