Serverless or classic compute on Databricks: how a job is billed and which to choose

Module 11 shipped the shop's job as a bundle that runs every night. This module asks what it costs and how fast it is, and what to change when either is wrong. This part needs nothing to rebuild: it is about the choice of compute in part 4 of module 11's databricks.yml, which had no compute settings at all, and so ran on serverless.

What a job is billed for

Databricks bills compute in DBUs, Databricks Units. Each kind of compute uses DBUs at its own rate while it runs, and each DBU has a price set by the product (serverless jobs, jobs compute, all-purpose compute, SQL warehouses), your plan and your region. A run's bill is the DBUs it used times that price.

What else you pay depends on the compute:

• Serverless: Databricks runs the machines in its own account, and the DBU price includes them. There is one bill.
• Classic: the machines run in your cloud account. Databricks bills the DBUs, and your cloud bills the machines, their disks and the network, separately.

On classic compute, you also pay for the minutes a cluster spends starting and the minutes it sits idle before it stops. That is why the kind of classic compute matters: jobs compute starts for a run and stops after it, and runs the same work for significantly less than all-purpose compute, the clusters people attach notebooks to. A scheduled job never belongs on an all-purpose cluster. The Databricks cost calculator puts these together for a worker type, a size and a number of hours on each cloud.

What serverless jobs do for you

A job with no compute settings, like the shop's, runs on serverless compute for jobs, which Databricks recommends for all job tasks. There is nothing to size: Databricks picks the machines, scales them during the run, and turns on Photon, its native engine, for every run. The runtime is versionless: it is upgraded for you, and every serverless job runs on the latest one.

2 settings are still yours.

Performance mode. Standard mode uses less compute, and fewer DBUs on the same price, in return for a start-up that can take 4 to 6 minutes. Performance optimized mode starts and runs faster, for jobs where minutes matter. In a bundle it is 1 line on the job:

resources:
  jobs:
    shop_daily:
      name: shop_daily
      performance_target: STANDARD

The shop's job runs at 02:00 for reports read at 09:00. It has hours of slack, so standard mode is the right one.

Retries. Serverless jobs retry a failed task on their own by default. That is safe for the shop's load, because module 10 made every task replace its day, so a second run changes nothing. A task that is not safe to run twice, such as one that sends an email or appends rows, should have that automatic retry turned off in its settings.

What serverless cannot do

Serverless compute runs code through Spark Connect. Some things do not run on it:

• R and the RDD API. Only DataFrame and SQL code, through Spark Connect, is supported.
• Hard-coded Spark settings. Only a short list of Spark settings can be set, and only in the session.
• The Spark UI. On serverless, you read a query's query profile instead. Part 2 uses it.

It also behaves differently in 3 ways that catch people out. ANSI mode is on, so a bad cast fails rather than returning null. The session time zone is UTC. And a task's working folder is not guaranteed, which is why module 11 built the shop's code as a wheel and passed the contract's path as an absolute one.

When classic compute is the right call

Databricks recommends serverless for most new work, and classic compute when a workload is not supported on serverless. Classic jobs compute is the right call when:

• the code needs R, the RDD API, a library given by Maven coordinates or a custom container, none of which serverless supports;
• you need control serverless does not give: an instance type, a Spark setting outside the allowed list, or the Spark UI and Spark logs;
• the job is large and steady, and you have measured that well-sized machines, with spot instances for workers, cost less than serverless for it.

When you do choose classic, Databricks' advice is short: jobs compute rather than all-purpose, the latest long-term support (LTS) runtime, autoscaling, a compute policy so nobody picks a size by guessing, spot instances where a lost worker only delays the run, and no hard-coded Spark settings, which override Databricks' own tuning. The Spark cluster calculator sizes the workers from the data a job reads.

Measure, then decide

Which compute is cheaper for a given job is a measurement, not a rule. Databricks' own advice for estimating serverless cost is to run a representative workload and read what it used in the system.billing.usage table. Part 3 does exactly that for the shop's job. First, part 2 finds where a run spends its time, because the cheapest DBU is the one a faster step never uses.

ToolDatabricks cost calculatorDBUs and machines for jobs, all-purpose and pipeline clusters, serverless and SQL warehouses, on AWS, Azure and Google Cloud.