Skip to the page
Tecyfy

Databricks interview question · easy

How does Databricks handle the scaling of compute resources for data analysis tasks?

The short answerDatabricks scales compute in three ways: classic clusters add and remove worker nodes with autoscaling, SQL warehouses add and remove whole clusters as queries queue up, and serverless compute is scaled by Databricks with no settings at all.

The full answer

Databricks scales compute in three ways: classic clusters add and remove worker nodes with autoscaling, SQL warehouses add and remove whole clusters as queries queue up, and serverless compute is scaled by Databricks with no settings at all.

1. Classic clusters: autoscaling workers

You set a minimum and maximum number of workers. Databricks adds workers when tasks are waiting and removes them when they are idle.

  • Optimized autoscaling (Premium plan and above) goes from minimum to maximum in at most two steps, and can scale down even while the cluster is busy by checking which workers hold shuffle files still needed. On jobs compute it scales down after 40 seconds of low use; on all-purpose compute after 150 seconds.
  • Standard autoscaling (the standard plan) starts by adding 8 nodes and grows exponentially, and scales down only when 90% of nodes have not been busy for 10 minutes.
  • Autoscaling local storage attaches more disk to a worker before it runs out of space.
  • Auto termination stops an idle all-purpose cluster entirely after the minutes you choose; jobs compute ends with its run.

Autoscaling suits workloads that vary. Streaming jobs on classic compute are usually better with a fixed size, or with Lakeflow pipelines, which have their own autoscaling for streaming.

2. SQL warehouses: more clusters for more queries

A warehouse has a size (2X-Small up to 5X-Large), which sets how big each cluster is, and a range of clusters.

  • Serverless warehouses use Intelligent Workload Management: each query's needs are predicted, queries are queued if capacity is short, and clusters are added quickly as wait times rise.
  • Pro and classic warehouses add clusters based on the estimated load of running and queued queries, scale up if a query waits 5 minutes, and scale down after 15 minutes of low load.

The size helps single large queries; more clusters help many concurrent users.

3. Serverless compute for notebooks, jobs and pipelines

Databricks runs the machines, starts them in seconds and scales them to the work. There is no cluster to size, and you pay only while work runs.

4. Choosing machines

On classic compute you still choose worker types: memory-optimised for large joins and caching, compute-optimised for CPU-heavy transformations, GPU instances for deep learning, and Photon for faster SQL and DataFrame work.

The result is that capacity follows demand: more machines during heavy work, few or none when idle, and less time spent sizing clusters by hand.

Last updated 16 October 2026

Asked next