Why is my Databricks bill so high? Find the cause

If your Databricks bill is higher than expected, check 2 things first: scheduled work running on all-purpose clusters, and plain Python running on multi-node clusters. Both are common, both are fixable in a day, and neither needs a new engine. Before you change anything, rank your cost by cluster and job with one query on the billing system tables, so you fix the clusters that actually cost the most.

How Databricks compute is billed

On classic compute, you pay twice:

• Databricks, in DBUs. Each machine type uses a set number of DBUs an hour, and the price of a DBU depends on the kind of compute (all-purpose, jobs, pipelines, SQL), your tier and your cloud.
• Your cloud, for the machines. The virtual machines run in your own cloud account and come on that bill.

Serverless compute is one charge from Databricks, with the machines included.

A worked example on AWS, Premium tier, US East, with list prices: a cluster of 1 driver and 4 workers, all m6id.2xlarge (8 cores, 32 GB), without Photon. Each machine uses 1.52 DBUs an hour and costs $0.4746 an hour from AWS.

The cluster is the same; the bill is almost twice as much. The Databricks cost calculator prices any cluster on any cloud the same way.

Step 1: find where the money goes

Databricks records every DBU in system.billing.usage, with the cluster, job and person behind it. This query ranks the last 30 days by cluster, at list price, with each cluster's name, owner and idle timeout:

WITH cost AS (
  SELECT
    u.workspace_id,
    u.usage_metadata.cluster_id AS cluster_id,
    u.usage_metadata.job_id     AS job_id,
    u.sku_name,
    u.usage_quantity * p.pricing.effective_list.default AS list_cost
  FROM system.billing.usage u
  JOIN system.billing.list_prices p
    ON p.sku_name = u.sku_name
   AND u.usage_end_time >= p.price_start_time
   AND (p.price_end_time IS NULL OR u.usage_end_time < p.price_end_time)
  WHERE u.usage_date >= current_date() - INTERVAL 30 DAYS
),
clusters AS (
  SELECT workspace_id, cluster_id, cluster_name, owned_by, auto_termination_minutes
  FROM system.compute.clusters
  QUALIFY row_number() OVER (PARTITION BY workspace_id, cluster_id ORDER BY change_time DESC) = 1
)
SELECT
  c.cluster_name,
  c.owned_by,
  c.auto_termination_minutes,
  cost.sku_name,
  count(DISTINCT cost.job_id) AS jobs,
  round(sum(cost.list_cost), 2) AS list_cost_usd
FROM cost
LEFT JOIN clusters c
  ON c.workspace_id = cost.workspace_id AND c.cluster_id = cost.cluster_id
GROUP BY ALL
HAVING sum(cost.list_cost) <> 0
ORDER BY list_cost_usd DESC
LIMIT 20;

How to read it:

• A SKU with ALL_PURPOSE near the top is an interactive cluster. If its owner is a service account, or the same cluster shows up every night, it is doing scheduled work at the all-purpose price.
• auto_termination_minutes of 0 means the cluster never shuts itself down.
• Rows with no cluster name are serverless or SQL warehouses; group by usage_metadata.warehouse_id or notebook_id for those.
• The figures are list prices for DBUs only. A negotiated discount lowers them, and the machines for classic compute are on your cloud bill. To split that bill the same way, put a team or project tag on every cluster. A cluster's custom tags reach its cloud machines (a cluster made from a pool takes the pool's tags instead) and show in the custom_tags column of the usage table.

Step 2: does this work need Spark?

Plain Python and pandas do not run on Spark. On a cluster, they run on the driver, and the workers wait, paid for. A notebook that reads a CSV with pandas, loops over rows or calls an API one record at a time uses one machine however many you started.

Signs a cluster is the wrong size for its code:

• The Spark UI shows few or no tasks while a cell runs for minutes.
• Worker CPU stays near zero in the cluster's metrics while the driver is busy.
• The code imports pandas and calls .toPandas() or reads files with open().

The fixes:

• Small data, plain Python: a single-node cluster. It is meant for small data and work that does not spread across machines: the driver does all the work and there are no workers to pay for.
• Data too big for one machine, pandas-style code: the pandas API on Spark. import pyspark.pandas as ps gives the same calls, and Spark spreads the work over the workers.
• Data that fits on a laptop, run once a day: consider whether it needs a cluster at all. A small job compute cluster, or serverless jobs, start, run and stop with the work.

Step 3: stop paying for idle clusters

An all-purpose cluster keeps billing DBUs and machines while it waits for its next command. It stops only when its idle timeout (auto termination) passes, which can be set from 10 to 10,000 minutes, or never.

Take a 2-hour nightly job on the cluster above, for 30 days:

The second is 3.7 times the first for the same work. And one all-purpose cluster with auto termination off, left on from Friday evening to Monday morning (63 hours), costs $412.84 for nothing.

Try itWhat a nightly job pays for

Runs on

All-purposeJob compute

Workers

4workers

plus 1 driver

1416

Work each night

2hours

15 min2 h8 h

Idle timeout

120minutes

auto termination

1060600

midnight6 a.m.noon6 p.m.midnight

WorkingIdle, still billedOff

30 days

$786

Databricks (DBUs)

$502

AWS (machines)

$285

An hour of the cluster

$6.55

What to change:

• Move scheduled work to job compute. In a job's settings, give each task a new job cluster, or serverless, in place of an existing all-purpose cluster.
• Keep all-purpose for people working. Notebooks and exploration, with a short idle timeout (30 to 60 minutes covers a lunch break).
• Enforce it with a cluster policy, so new clusters cannot skip it:

{
  "autotermination_minutes": { "type": "range", "maxValue": 60, "defaultValue": 30 },
  "custom_tags.team": { "type": "fixed", "value": "analytics" },
  "autoscale.max_workers": { "type": "range", "maxValue": 8, "defaultValue": 2 },
  "dbus_per_hour": { "type": "range", "maxValue": 40 }
}

dbus_per_hour caps the size of any cluster made under the policy, and the tag lets you split the bill by team.

Step 4: then compare engines, not before

Once the cost is attributed and the 2 common leaks are closed, a workload that is still expensive is worth a look at alternatives: a SQL warehouse for BI queries, or a single machine with DuckDB or Polars for data that fits in its memory. Compare on your own workload and your own prices.

Treat published "performance per dollar" benchmarks with care. Many are written by a vendor about its own engine, on a dataset and settings it chose. Ask who ran it, on what size of data, with which machine types and prices, and whether the other engine was tuned as carefully.

Keep it from coming back

Budgets in the account console email you when spending passes a threshold, filtered by workspace or tag. They do not stop spending; the cluster policy is what limits it. Once a month, run the query above, look at the top 10 clusters, and ask of each: is it on the right kind of compute, the right size, and shutting down when idle?

The Spark sizing calculator picks a cluster for a given amount of data, and the m6id.2xlarge page shows that worker's price on every cloud and region.