# Build it: what a Databricks job costs a month, from system.billing.usage

Join system.billing.usage to system.billing.list_prices on the SKU and the time the price applied, keep the rows whose usage_metadata.job_id is your job, and add up usage_quantity times pricing.effective_list.default for each job_run_id. That is each run's cost at list price; the average run times the runs a month is the month. Before a job goes live, run it on a few representative days in development and read the same tables, as Databricks recommends, then set a budget with an alert.

- Page: https://tecyfy.com/data-engineering/course/performance-and-cost/build-it
- Course: Data engineering course (https://tecyfy.com/data-engineering/course), Module 12: Performance and cost, Build it: what the job costs a month
- Parts of this module: Serverless or classic compute (https://tecyfy.com/data-engineering/course/performance-and-cost); Find the slow step (https://tecyfy.com/data-engineering/course/performance-and-cost/find-the-slow-step); Build it: what the job costs a month (https://tecyfy.com/data-engineering/course/performance-and-cost/build-it)

At the end of this part you can:

- Read the DBUs and the list price cost of each run of a job from the billing system tables
- Estimate a job's month before it goes live from a few representative runs
- Keep a pipeline's cost in sight with tags, budgets and a monthly check

[Part 2](/data-engineering/course/performance-and-cost/find-the-slow-step) found where a run spends its time. This part turns that time into money, for the shop's job, the way an owner asks for it: what does it cost each run, and each month? It needs the job from [module 11](/data-engineering/course/testing-and-ci-cd/build-it), deployed, and a few of its runs; the system tables are read with SQL in the workspace.

## Build it: the cost of each run

Databricks writes every unit of billed usage to `system.billing.usage`: the SKU, the quantity of DBUs, when, and in `usage_metadata` which job and which run it was for. `system.billing.list_prices` holds each SKU's list price over time. Joined, they give each run's cost at list price:

```sql
-- The DBUs and list price cost of each run of shop_daily, over the last 30 days.
WITH runs AS (
  SELECT u.usage_metadata.job_run_id AS job_run_id,
         min(u.usage_start_time) AS started,
         sum(u.usage_quantity) AS dbus,
         sum(u.usage_quantity * p.pricing.effective_list.default) AS list_cost
  FROM system.billing.usage AS u
  JOIN system.billing.list_prices AS p
    ON p.sku_name = u.sku_name
   AND p.cloud = u.cloud
   AND u.usage_end_time >= p.price_start_time
   AND (p.price_end_time IS NULL OR u.usage_end_time < p.price_end_time)
  WHERE u.usage_metadata.job_id IN (SELECT job_id FROM system.lakeflow.jobs WHERE name = 'shop_daily')
    AND u.usage_date >= current_date() - INTERVAL 30 DAYS
    AND u.usage_unit = 'DBU'
  GROUP BY 1
)
SELECT job_run_id, started, round(dbus, 3) AS dbus, round(list_cost, 3) AS list_cost
FROM runs
ORDER BY started
```

3 parts of it matter:

- **The price window.** A SKU's price can change, so the join takes the price that applied when the usage ended: after `price_start_time` and before `price_end_time`, or with no end time for the current price.
- **Corrections.** When Databricks corrects a record, it adds a **retraction** with the same quantity negative and a **restatement** with the right one. Adding up `usage_quantity`, as the query does, cancels the wrong record and keeps the right one, so never filter on `record_type`.
- **List price.** `pricing.effective_list.default` is the published price, with any promotion everyone gets. A contract's discounts are not in these tables, so the figure is what the work would cost at list price, which is the right number to compare 2 versions of a job, and an upper bound on the bill.

`usage_metadata.job_run_id` is filled for serverless jobs and jobs compute. A job run on an all-purpose cluster has no `job_run_id` here, so its runs cannot be told apart, one more reason, after part 1's price, to keep scheduled jobs off all-purpose clusters.

## The month

From the runs, the month is 1 more query. Replace the last `SELECT` of the query above with:

```sql
SELECT count(*) AS runs,
       round(avg(dbus), 3) AS dbus_per_run,
       round(avg(list_cost), 3) AS list_cost_per_run,
       round(max(list_cost), 3) AS dearest_run,
       round(avg(list_cost) * 30, 2) AS list_cost_for_30_runs
FROM runs
```

`dearest_run` is worth as much as the average. A run that costs several times the usual is a run that read more, retried, or waited on part 2's slowest step, and a month with a few of those is a different month. To see where the DBUs went, group the same rows by `u.sku_name` and `u.product_features.performance_target`, which also shows whether the job really runs in part 1's standard mode.

On classic compute, this is only the Databricks part of the bill: your cloud bills the machines, their disks and network separately, and they are not in these tables. On serverless, the DBU price already includes the machines.

## Before the job goes live

The best estimate of a job's month is the job itself, run on representative data, which is what Databricks recommends for serverless. With the bundle from module 11, run the development copy on 4 days of September:

```bash
for day in 2026-09-08 2026-09-15 2026-09-22 2026-09-29; do
  databricks bundle run -t dev shop_daily --params run_date=$day
done
```

Wait for the records: they typically arrive within 12 hours, and serverless usage can take up to 24. Then run the month query with `name LIKE '[dev %] shop_daily'` in place of `name = 'shop_daily'`, to read the development copy. The average of the 4 runs, times the runs a month, is the estimate, and the dearest of them says how far 1 run can go above it. With real data, include the busiest day you know of, such as the end of a sale. For a first figure before any run, the [Databricks cost calculator](/data-engineering/databricks-cost) prices compute by the hour on each cloud.

Run the estimate again when the data grows. The shop's job reads 1 day of orders, so its cost follows the size of a day; a job that reads the whole table grows with the table instead, and is a candidate for part 2's fixes before it is a cost problem.

## Keep it in sight

A cost nobody looks at grows. 3 habits keep it visible:

- **Tag it.** Serverless usage carries the custom tags of the user's **serverless usage policy**, and classic compute carries its own tags, both in `custom_tags`. A tag such as `team: shop` lets 1 query or budget follow everything the team runs. Tags cannot be added to past usage, so set them before the job goes live.
- **Set a budget.** An account budget, filtered by those tags, emails you when the month's spending reaches its amount. Set it a little above the estimate, so it says something only when something has changed.
- **Look monthly.** Save the month query as a dashboard or an alert, and compare each month with the last. A rise with no rise in the data means a step got slower, and part 2 says which.

Some costs are not your job's runs. Predictive optimization, from [module 4](/data-engineering/course/delta-lake-and-iceberg/optimize-and-vacuum), and data quality monitoring, from [module 11](/data-engineering/course/testing-and-ci-cd/data-quality), run on serverless and bill under the serverless jobs SKU, outside your job's runs. A monthly look at all usage by `billing_origin_product` shows them.

## What you built

- A choice of compute for the shop's job: serverless, in standard mode, with retries that are safe because every task replaces its day.
- A ranking of the job's tasks by time, and a way from the slowest task to its slowest operator and its fix.
- The cost of every run at list price, the month it adds up to, an estimate made before the job went live, and the tags and budget that keep it visible.

Module 13 builds what the shop's data is increasingly for: the tables and the vector index an AI answer is built from.

- Tool: [Spark sizing calculator](https://tecyfy.com/data-engineering/spark-calculator). The Databricks workers to pick, or the executors, memory and shuffle partitions for your nodes, with every number explained.

## Check yourself

1. You ran the shop's job in development at 10:00 and query system.billing.usage at 10:30. It returns no rows for the run. What is most likely?

   A. The run was free
   B. The query's join to list_prices is wrong
   C. Development runs are never billed
   D. The usage records have not arrived yet; they are typically there within 12 hours

### Answers

1. D. Billing records typically arrive within 12 hours, and Databricks says serverless usage can take up to 24 hours to appear. Development runs are billed like any other. Run the query again the next day before reading anything into an empty result.

## Questions

### How do I see what a Databricks job costs?

Query system.billing.usage for the rows whose usage_metadata.job_id is the job, join system.billing.list_prices on sku_name where the usage falls between price_start_time and price_end_time, and add up usage_quantity times pricing.effective_list.default, grouped by usage_metadata.job_run_id for each run. This gives the cost at list price; your contract's discounts are not in it.

### How do I estimate Databricks cost before running a job in production?

Run the job on a few representative days of data in a development workspace, wait for the usage records, typically within 12 hours, and read what each run used from system.billing.usage. Multiply the average run by the runs a month. For classic compute, add the cloud's bill for the machines, which is not in Databricks' tables. The Databricks cost calculator gives a first figure before any run.

## Sources

- [Billable usage system table reference](https://docs.databricks.com/aws/en/admin/system-tables/billing)
- [Pricing system table reference](https://docs.databricks.com/aws/en/admin/system-tables/pricing)
- [Monitor costs using system tables](https://docs.databricks.com/aws/en/admin/usage/system-tables)
- [Connect to serverless compute](https://docs.databricks.com/aws/en/compute/serverless/)
- [Attribute usage with serverless usage policies](https://docs.databricks.com/aws/en/admin/usage/budget-policies)
- [Best practices for cost optimization](https://docs.databricks.com/aws/en/lakehouse-architecture/cost-optimization/best-practices)