# Serverless or classic compute on Databricks: how a job is billed and which to choose For most new Databricks jobs, choose serverless compute: there is nothing to size, it starts quickly, Photon and autoscaling are on, and its DBU price already includes the machines. Use the standard performance mode for a job that can wait a few minutes to start, which uses fewer DBUs. Choose classic jobs compute when the workload is not supported on serverless, such as R or RDD code, or when you need control over machine types, spot instances or Spark settings; then you pay DBUs to Databricks and the machines to your cloud. - Page: https://tecyfy.com/data-engineering/course/performance-and-cost - Course: Data engineering course (https://tecyfy.com/data-engineering/course), Module 12: Performance and cost, Serverless or classic compute - Parts of this module: Serverless or classic compute (https://tecyfy.com/data-engineering/course/performance-and-cost); Find the slow step (https://tecyfy.com/data-engineering/course/performance-and-cost/find-the-slow-step); Build it: what the job costs a month (https://tecyfy.com/data-engineering/course/performance-and-cost/build-it) At the end of this part you can: - Say what a Databricks job is billed for on serverless and on classic compute - Choose serverless or classic compute for a job, and the performance mode of a serverless job - Name what serverless compute does not support, and what to use instead [Module 11](/data-engineering/course/testing-and-ci-cd) shipped the shop's job as a bundle that runs every night. This module asks what it costs and how fast it is, and what to change when either is wrong. This part needs nothing to rebuild: it is about the choice of compute in part 4 of module 11's `databricks.yml`, which had no compute settings at all, and so ran on serverless. ## What a job is billed for Databricks bills compute in **DBUs**, Databricks Units. Each kind of compute uses DBUs at its own rate while it runs, and each DBU has a price set by the product (serverless jobs, jobs compute, all-purpose compute, SQL warehouses), your plan and your region. A run's bill is the DBUs it used times that price. What else you pay depends on the compute: - **Serverless**: Databricks runs the machines in its own account, and the DBU price includes them. There is one bill. - **Classic**: the machines run in your cloud account. Databricks bills the DBUs, and your cloud bills the machines, their disks and the network, separately. On classic compute, you also pay for the minutes a cluster spends starting and the minutes it sits idle before it stops. That is why the kind of classic compute matters: **jobs compute** starts for a run and stops after it, and runs the same work for significantly less than **all-purpose compute**, the clusters people attach notebooks to. A scheduled job never belongs on an all-purpose cluster. The [Databricks cost calculator](/data-engineering/databricks-cost) puts these together for a worker type, a size and a number of hours on each cloud. ## What serverless jobs do for you A job with no compute settings, like the shop's, runs on serverless compute for jobs, which Databricks recommends for all job tasks. There is nothing to size: Databricks picks the machines, scales them during the run, and turns on Photon, its native engine, for every run. The runtime is versionless: it is upgraded for you, and every serverless job runs on the latest one. 2 settings are still yours. **Performance mode.** Standard mode uses less compute, and fewer DBUs on the same price, in return for a start-up that can take 4 to 6 minutes. Performance optimized mode starts and runs faster, for jobs where minutes matter. In a bundle it is 1 line on the job: ```yaml resources: jobs: shop_daily: name: shop_daily performance_target: STANDARD ``` The shop's job runs at 02:00 for reports read at 09:00. It has hours of slack, so standard mode is the right one. **Retries.** Serverless jobs retry a failed task on their own by default. That is safe for the shop's load, because [module 10](/data-engineering/course/orchestration) made every task replace its day, so a second run changes nothing. A task that is not safe to run twice, such as one that sends an email or appends rows, should have that automatic retry turned off in its settings. ## What serverless cannot do Serverless compute runs code through Spark Connect. Some things do not run on it: - **R** and the **RDD API**. Only DataFrame and SQL code, through Spark Connect, is supported. - **Hard-coded Spark settings.** Only a short list of Spark settings can be set, and only in the session. - **The Spark UI.** On serverless, you read a query's **query profile** instead. Part 2 uses it. It also behaves differently in 3 ways that catch people out. ANSI mode is on, so a bad cast fails rather than returning null. The session time zone is UTC. And a task's working folder is not guaranteed, which is why module 11 built the shop's code as a wheel and passed the contract's path as an absolute one. ## When classic compute is the right call Databricks recommends serverless for most new work, and classic compute when a workload is not supported on serverless. Classic jobs compute is the right call when: - the code needs R, the RDD API, a library given by Maven coordinates or a custom container, none of which serverless supports; - you need control serverless does not give: an instance type, a Spark setting outside the allowed list, or the Spark UI and Spark logs; - the job is large and steady, and you have measured that well-sized machines, with spot instances for workers, cost less than serverless for it. When you do choose classic, Databricks' advice is short: jobs compute rather than all-purpose, the latest long-term support (LTS) runtime, autoscaling, a compute policy so nobody picks a size by guessing, spot instances where a lost worker only delays the run, and no hard-coded Spark settings, which override Databricks' own tuning. The [Spark cluster calculator](/data-engineering/spark-calculator) sizes the workers from the data a job reads. ## Measure, then decide Which compute is cheaper for a given job is a measurement, not a rule. Databricks' own advice for estimating serverless cost is to run a representative workload and read what it used in the `system.billing.usage` table. [Part 3](/data-engineering/course/performance-and-cost/build-it) does exactly that for the shop's job. First, [part 2](/data-engineering/course/performance-and-cost/find-the-slow-step) finds where a run spends its time, because the cheapest DBU is the one a faster step never uses. - Tool: [Databricks cost calculator](https://tecyfy.com/data-engineering/databricks-cost). DBUs and machines for jobs, all-purpose and pipeline clusters, serverless and SQL warehouses, on AWS, Azure and Google Cloud. ## Check yourself 1. The shop's job runs once a day at 02:00 and the reports are read at 09:00. Which compute fits it best? A. An all-purpose cluster left running, so the job starts at once B. Serverless compute in standard performance mode C. Serverless compute in performance optimized mode D. Classic jobs compute with 8 large workers 2. A team moves a classic job to serverless and it fails. Which of these is the likely reason? A. The job reads a Delta table B. The job is written in Python C. The job uses the RDD API, which serverless does not support D. The job has 4 tasks ### Answers 1. B. The job has hours of slack, so a start that takes a few minutes longer costs nothing, and standard mode uses fewer DBUs for the same work. An all-purpose cluster bills for every idle hour and costs more per DBU than jobs compute; 8 large workers are sized for a load many times the shop's. 2. C. Serverless compute supports only the Spark Connect APIs, so RDD code, and R, do not run on it. Rewriting the RDD part with DataFrames is usually the fix; if that is not possible, the job stays on classic compute. ## Questions ### What is a DBU in Databricks? A Databricks Unit, the unit Databricks bills compute in. Each kind of compute, such as serverless jobs, classic jobs compute or a SQL warehouse, uses DBUs at its own rate per hour, and each DBU has a price that depends on the product, the plan and the region. On classic compute your cloud also bills you for the machines, disks and network; on serverless the DBU price already includes the machines. ### Is serverless cheaper than classic compute on Databricks? It depends on the workload, so measure it. Serverless has no idle machines and no start-up to pay for, and its standard mode uses fewer DBUs, which favours short and occasional jobs. A long, steady job on well-sized classic compute with spot instances can cost less. Databricks recommends running a representative workload and reading its cost in the system.billing.usage table. ## Sources - [Run your Lakeflow Jobs with serverless compute for workflows](https://docs.databricks.com/aws/en/jobs/run-serverless-jobs) - [Serverless compute limitations](https://docs.databricks.com/aws/en/compute/serverless/limitations) - [Connect to serverless compute](https://docs.databricks.com/aws/en/compute/serverless/) - [Classic compute configuration best practices](https://docs.databricks.com/aws/en/compute/cluster-config-best-practices) - [Best practices for cost optimization](https://docs.databricks.com/aws/en/lakehouse-architecture/cost-optimization/best-practices) - [What is Photon?](https://docs.databricks.com/aws/en/compute/photon)