Databricks interview questions for Data Engineer roles

Delta Lake gets ACID guarantees from its transaction log: a table version exists only once its commit file is written to the _delta_log folder, so every change is all or nothing.

1. Atomicity: a write adds new Parquet files and then commits one log entry that adds them (and removes any replaced files). If the job fails before the commit, readers never see its files.
2. Consistency: the commit checks the table's schema and constraints (such as NOT NULL and CHECK), so a write that breaks them is rejected.
3. Isolation: writers use optimistic concurrency control. Each writer reads a snapshot, and at commit time Delta checks whether another commit conflicts; if so, the later writer fails or retries. Readers always see a consistent snapshot, never a half-written one.
4. Durability: committed data and log files live in cloud object storage, which is durable.

Because every commit is kept in the log, the same mechanism also gives table history, time travel and RESTORE.

Apache Spark is an open-source engine for processing data in parallel across many machines; Databricks is a commercial platform, from the company Spark's creators founded, that runs Spark for you and adds storage, governance, orchestration and AI tools around it.

A short way to put it: Spark is the engine, Databricks is the whole car.

Apache Spark

• Splits data processing across a cluster of machines.
• APIs in Python (PySpark), SQL, Scala, Java and R.
• Libraries for SQL, streaming, machine learning and graph processing.
• You install, configure and run it yourself, on any infrastructure.

Databricks adds

• Managed compute: clusters, serverless compute and SQL warehouses, with an optimised Spark runtime (and the Photon engine).
• Delta Lake tables with ACID transactions and time travel.
• Unity Catalog for permissions, lineage and auditing across data and AI assets.
• Lakeflow for ingestion, declarative pipelines and job orchestration.
• Notebooks, SQL editor, dashboards and Genie for analysis.
• Machine learning and AI tools such as MLflow and model serving.

Spark is free and portable; Databricks charges for usage but removes most of the work of running Spark well.

Ron can undo it with Delta Lake time travel: find the table version just before his UPDATE in the table history, then run RESTORE TABLE to that version.

1. Find the version before the mistake

DESCRIBE HISTORY my_table;

The history lists every commit, newest first, with its version number, timestamp, operation (UPDATE, MERGE, WRITE and so on), who ran it and its metrics. Ron looks for his UPDATE and takes the version just before it, say version 41.

2. Check the old data first (optional but wise)

SELECT count(*) FROM my_table VERSION AS OF 41;
SELECT * FROM my_table TIMESTAMP AS OF '2026-10-16 09:00:00' LIMIT 10;

In Python: spark.read.option("versionAsOf", 41).table("my_table"). On Databricks the short form my_table@v41 also works. Reading an old version never changes the table.

3. Restore

RESTORE TABLE my_table TO VERSION AS OF 41;
-- or
RESTORE TABLE my_table TO TIMESTAMP AS OF '2026-10-16 09:00:00';

In Python: DeltaTable.forName(spark, "my_table").restoreToVersion(41).

What RESTORE does. It does not delete history. It writes a new version whose contents equal version 41, so the bad update is still in the history and the restore itself can be undone. It needs MODIFY permission on the table, and it is quick because it mostly changes which files the log points to, rather than rewriting data.

Things to watch

• Retention. Time travel needs the old data files. VACUUM removes files older than the table's retention (7 days by default), and log entries are kept 30 days by default. If the update is older than that, or a vacuum with short retention ran, the version may be gone.
• Writes after the mistake. Restoring to version 41 also removes anything other jobs wrote after it. If others wrote since, it may be safer to fix the rows with a MERGE from the old version instead.
• Streaming readers. A restore counts as a data change, so streams reading the table may process the restored rows again and create duplicates downstream.

Preventing it next time. Run risky updates on a test copy first (a shallow clone, CREATE TABLE t_test SHALLOW CLONE t, is instant), check the number of affected rows the UPDATE reports before moving on, and keep write permission on production tables with jobs rather than people.

You create and manage clusters on the Compute page of a Databricks workspace (or through the API, CLI or Terraform), choosing the runtime, machine types, size, access mode and auto termination; admins then control what users may create with compute policies.

Creating a cluster

Open Compute, choose Create compute, and set:

• Policy: most users create compute through a policy, which fixes or limits the other settings.
• Databricks Runtime version: prefer a current LTS (long-term support) version. ML runtimes add machine learning libraries.
• Access mode: Standard (shared by several users, with data isolation between them; Databricks' recommended default) or Dedicated (for one user or group, needed for some features). These replaced the older "shared", "single user" and "no isolation" modes.
• Single node or multi node: single node for small data and non-distributed work; multi node for Spark jobs on large data.
• Worker and driver types: memory-, compute- or storage-optimised machines, and Photon acceleration if wanted.
• Autoscaling: a minimum and maximum number of workers.
• Auto termination: minutes of inactivity before the cluster shuts down, so idle compute stops costing money.
• Tags for cost reporting, and under advanced options Spark settings, environment variables and init scripts.

Managing clusters

• Start, restart, edit and terminate from the Compute page. A restart applies configuration changes.
• Permissions: grant users Can attach to, Can restart or Can manage.
• Monitoring: the cluster's metrics, event log, driver logs and the Spark UI show load, failures and slow stages.
• Libraries: install them on the cluster, or better in notebooks with %pip so code carries its own dependencies.
• Cost: use jobs compute (created per run and terminated afterwards) for scheduled work, keep all-purpose clusters for interactive use, and watch spending in the billing system tables.

Choosing the type of compute

• All-purpose compute: interactive notebooks, shared by a team.
• Jobs compute: scheduled and production jobs.
• Serverless compute: Databricks manages the machines, starts in seconds and scales automatically; often the simplest choice for notebooks, jobs and pipelines.
• SQL warehouses: SQL queries, dashboards and BI tools.

A good answer mentions policies, access modes and auto termination, because those are what keep shared workspaces secure and costs under control.

Delta Lake turns files in cloud storage into reliable tables: it adds a transaction log on top of Parquet files that gives ACID transactions, schema enforcement, table history and time travel, and the statistics needed for fast queries.

1. ACID transactions

• Problem: plain data lakes have no transactions, so a failed job can leave half-written files, and readers can see data mid-write.
• Delta Lake: every change is committed as one entry in the _delta_log transaction log. A version either exists completely or not at all, and readers always see a consistent snapshot. Concurrent writers are handled with optimistic concurrency control.

2. One table for batch and streaming

• Problem: separate systems for streaming and batch data meant duplicated pipelines.
• Delta Lake: the same table can be written by a stream and read in batch, or the other way round.

3. Schema enforcement and evolution

• Problem: a file with the wrong columns silently breaks downstream jobs.
• Delta Lake: writes that do not match the table's schema are rejected, and the schema can be evolved on purpose (mergeSchema, ALTER TABLE). NOT NULL and CHECK constraints add rules.

4. History and time travel

• Problem: undoing a bad load in a data lake means rebuilding by hand.
• Delta Lake: DESCRIBE HISTORY shows every change with who made it and when; VERSION AS OF and TIMESTAMP AS OF query old versions, and RESTORE rolls a table back.

5. Row-level changes

• MERGE, UPDATE and DELETE work on lake files, which makes upserts, change data capture and privacy deletions practical. Deletion vectors make small changes cheap by marking rows instead of rewriting files.

6. Performance

• The log stores file-level statistics, so queries skip files that cannot match (data skipping).
• OPTIMIZE compacts small files, and liquid clustering keeps data laid out by the columns queries filter on.
• On Unity Catalog managed tables, predictive optimization runs this maintenance automatically.

In summary, Delta Lake gives a data lake the reliability of a warehouse while keeping open files in your own storage. That combination is what Databricks calls the lakehouse.

No, and the reason is not the file format alone. Auto Loader infers and evolves schemas for JSON, CSV, XML, Avro and Parquet, but by default it only adds new columns: it does not change an existing column's type, and it stops the stream once each time it adds a column. Text and binary files have a fixed schema, so there is nothing to evolve.

How it works for each format

• JSON, CSV and XML have no types in the file, so Auto Loader infers every column as a string, including nested JSON fields, unless you set cloudFiles.inferColumnTypes to true. Strings make type changes impossible, which is why it is the safe default.
• Avro and Parquet carry their own schema. If two files give a column different types, Auto Loader picks the wider type; a value that cannot fit goes to the rescued data column.
• Text and binaryFile always have the same fixed columns.

The schema evolution modes (cloudFiles.schemaEvolutionMode):

Example

(spark.readStream.format("cloudFiles")
   .option("cloudFiles.format", "json")
   .option("cloudFiles.schemaLocation", "/Volumes/main/raw/_schemas/events")
   .option("cloudFiles.inferColumnTypes", "true")
   .option("cloudFiles.schemaHints", "amount DECIMAL(10,2)")
   .load("/Volumes/main/raw/events/"))

schemaLocation is where Auto Loader keeps the schema and its history. Schema hints fix the types of the columns you care about while the rest is inferred.

In practice: run the stream as a Lakeflow Job or pipeline with retries, so the planned stop on a new column restarts by itself; keep the _rescued_data column in bronze and alert when it fills; and decide in silver which new columns to use.

No. Libraries installed on a cluster belong to that cluster, not to the notebook, so a cloned notebook in another workspace only has the library if the compute it runs on there has it too.

A notebook stores code and its outputs. Libraries come from one of three places:

1. Compute-scoped libraries, installed on a cluster. Every notebook attached to that cluster can use them.
2. Notebook-scoped libraries, installed with %pip install at the top of the notebook. These travel with the notebook, because the install command is part of its code.
3. Environments on serverless compute, which list the packages a notebook or job needs.

To make the cloned notebook work in the new workspace, either install the library on the cluster it will use there, or better, add a %pip install cell (or a serverless environment) so the dependency is declared in the notebook itself. For repeatable setups, pin the version, for example %pip install requests==2.32.3.

No. Autoscaling shrinks the cluster only down to the minimum number of workers you set; it never removes the workers below that. Removing everything is a separate setting, auto termination, which shuts down the whole cluster after it has been idle for the number of minutes you choose.

• Autoscaling adds workers when work queues up and removes them when they sit idle, always staying between the minimum and maximum. (The only time a cluster has fewer than the minimum is when the cloud provider takes instances away; Databricks then tries to get back to the minimum.)
• Auto termination stops the driver and all workers once nothing has run for the idle period. Restarting the cluster brings it back with the same configuration.
• Job compute is created for a job run and terminates when the run ends.
• Serverless compute is managed by Databricks and scales without these settings.

So with a workload that drops to zero, an autoscaling cluster first scales down to its minimum workers, and only terminates entirely if auto termination is enabled and the idle time passes.

%run runs another notebook inline in the same context, so its variables and functions become available to the caller; dbutils.notebook.run() runs the other notebook as a separate ephemeral job, passing string arguments in and getting one string back.

%run

• Runs the whole notebook immediately, in the caller's context. Whatever it defines can be used afterwards.
• Must be alone in its cell.
• Typical use: loading shared helper functions or configuration.

%run ./utilities/helpers

dbutils.notebook.run(path, timeout_seconds, arguments)

• Starts a new ephemeral job for the call, with its own context; nothing is shared with the caller.
• arguments sets the target notebook's widgets; timeout_seconds limits the run (0 means no timeout).
• The child returns a value with dbutils.notebook.exit("..."). Parameters and return values are strings, so pass larger results as a table name or path.
• Allows logic around the call, such as loops, conditions and retries.

result = dbutils.notebook.run("./child", 600, {"date": "2026-10-16"})

Databricks now recommends neither for most cases: put shared code in workspace files (Python modules you import), and orchestrate notebooks with Lakeflow Jobs tasks.

By default nothing stops: an expectation without an action only warns, so failing rows are still written to the target table and counted in the pipeline's data quality metrics. To change that, give the expectation an action: drop the failing rows, or fail the update.

An expectation is a named condition on a dataset. Each has one of three actions:

Python

from pyspark import pipelines as dp

@dp.table
@dp.expect("valid_age", "age BETWEEN 0 AND 120")
@dp.expect_or_drop("has_id", "customer_id IS NOT NULL")
@dp.expect_or_fail("positive_amount", "amount > 0")
def customers():
    return spark.readStream.table("raw.customers")

@dp.expect_all, @dp.expect_all_or_drop and @dp.expect_all_or_fail take a dictionary of several conditions at once. (Older code uses import dlt and @dlt.expect..., which still works.)

SQL

CREATE OR REFRESH STREAMING TABLE customers (
  CONSTRAINT has_id EXPECT (customer_id IS NOT NULL) ON VIOLATION DROP ROW
) AS SELECT * FROM STREAM(raw.customers);

What "fail" does. The table update is rolled back atomically, so no partial data is written, and the update stops with a message naming the expectation. In a triggered pipeline, other flows running in parallel are not failed; in a continuous pipeline the failed flow and the flows that depend on it stop. You fix the data or the rule and run the pipeline again. Fail records no metrics, because the update did not complete.

Seeing the results. Warn and drop counts appear in the dataset's Data quality tab in the pipeline UI and in the pipeline's event log, which you can query to track quality over time.

Choosing: warn for rules you are still learning about, drop for rows that are simply unusable, and fail only when a breach means the whole load is wrong. A common pattern is to quarantine instead of dropping: write rows that fail the rules to a separate table so they can be inspected. To do that, define the rules once in a dictionary, apply them with @dp.expect_all_or_drop on the clean table, and build a second table that keeps only the rows where the same conditions are false. Nothing is then lost silently: the clean table stays clean, and the quarantine table shows exactly which records broke which rule, so the source team can fix them and the rows can be reprocessed.

Unity Catalog records lineage automatically, so you can open the table in Catalog Explorer and look at its Lineage tab, or query the system.access.table_lineage system table for its upstream sources.

1. Catalog Explorer

Open Catalog, browse to the catalog, schema and table, and choose the Lineage tab. It lists the upstream tables (and downstream tables, notebooks, jobs, pipelines and dashboards). See lineage graph shows them as a graph you can expand, including column-level lineage.

2. SQL on the system table

SELECT DISTINCT source_table_full_name, entity_type, created_by, max(event_time) AS last_seen
FROM system.access.table_lineage
WHERE target_table_full_name = 'main.sales.orders_gold'
  AND source_table_full_name IS NOT NULL
GROUP BY ALL
ORDER BY last_seen DESC;

entity_type says what wrote the table (a notebook, job, pipeline and so on). system.access.column_lineage gives the same at column level.

Things to know

• Lineage is captured from queries run on Databricks compute, for tables registered in Unity Catalog. Work done outside Databricks is not seen.
• System tables keep a rolling year of lineage.
• You only see lineage for objects you have permission to see.

Lisa grants the team's group SELECT on the table, plus USE CATALOG on its catalog and USE SCHEMA on its schema. SELECT alone is not enough: without the two usage privileges the group cannot reach the table at all.

GRANT USE CATALOG ON CATALOG main TO `data-science`;
GRANT USE SCHEMA ON SCHEMA main.sales TO `data-science`;
GRANT SELECT ON TABLE main.sales.orders TO `data-science`;

USE CATALOG and USE SCHEMA only let the group pass through to objects inside; they do not grant access to any table on their own. So the team can read orders and nothing else in the schema.

What she must not grant: MODIFY (insert, update and delete), ALL PRIVILEGES, CREATE TABLE on the schema, or ownership of the table. Only the owner (or someone with MANAGE) can change the table's schema, so a group with just SELECT cannot alter it.

In the UI: open the table in Catalog Explorer, choose the Permissions tab, then Grant, pick the group and tick SELECT. The USE CATALOG and USE SCHEMA grants are made the same way on the catalog and the schema.

If only some columns should be visible: create a view with just those columns and grant SELECT on the view instead of the table, or add a column mask to sensitive columns so other groups see masked values.

Good practice: always grant to an account group, not to individual users, so people joining or leaving the team need no permission changes, and check what the group can do with SHOW GRANTS ON TABLE main.sales.orders.

The two streams used different schema evolution modes, most likely without anyone choosing them. Auto Loader's default, when you let it infer the schema, is to add the new column and then stop the stream with an UnknownFieldException, so the next start picks up the new schema. When you give it a schema, the default changes to ignoring new columns, so the field silently never arrives. Choose the mode on purpose for each stream and plan for the restart.

The modes (cloudFiles.schemaEvolutionMode)

How to set it up

• Run the stream as a job with retries. With addNewColumns, the failure is expected: the first run records the new schema in the schema location, the retry continues. A stream started by hand from a notebook just stays down.
• Always keep a rescued data column. It catches new columns, type mismatches and fields whose case differs from the schema, as JSON with the source file path, instead of dropping them. With an explicit schema, set rescuedDataColumn.
• Use schema hints, not a full schema, for the columns you care about. JSON is inferred as strings; hints such as amount decimal(18,2), event_ts timestamp fix the types you depend on while still letting new columns evolve. A full schema forces none or rescue.
• Alert on rescued data. A daily count of rows with a non-null _rescued_data tells you the source changed before a user does.

Bronze and silver are different decisions

Bronze should keep everything, so addNewColumns or rescue with job retries fits. Silver tables that downstream reports depend on should change on purpose: read the new column from bronze when you have decided what it means, and add it to silver with a reviewed change.

What interviewers want: knowing the modes and that the default depends on whether a schema was given, that the addNewColumns failure is designed to be restarted by a job, the rescued data column, schema hints, and a bronze-to-silver split for schema decisions.

Query the billing system table, group the spend by SKU, workspace, compute and tags, and compare the two months until the increase sits in a few rows. Most Databricks cost surprises come from a short list: scheduled jobs running on all-purpose clusters, interactive clusters or warehouses that never shut down, a job that got slower or runs more often, and oversized compute. Then put guardrails in place so the next one shows up in days, not on the invoice.

1. Break down the spend

system.billing.usage has one row per usage record, with the SKU, date, quantity, tags and the IDs of the job, cluster or warehouse:

SELECT sku_name,
       usage_metadata.job_id,
       usage_metadata.cluster_id,
       usage_metadata.warehouse_id,
       date_trunc('month', usage_date) AS month,
       SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_date >= '2026-08-01'
GROUP BY ALL
ORDER BY dbus DESC;

Join to system.billing.list_prices to turn DBUs into money, and compare month against month.

2. What to look for

A useful sign: job_id is empty for jobs that ran on all-purpose compute. A lot of all-purpose usage with no job ID but a job-like daily pattern is usually scheduled work on the wrong compute.

3. Guardrails

• Tags on clusters, warehouses and jobs (team, project, environment). Tags only apply from when they are added, so start now.
• Compute policies that cap worker counts, restrict instance types and require auto-termination.
• Budgets and alerts per team, with a weekly report from the billing table.
• Review the top 10 jobs by cost each month. Those few usually make up most of the bill, and Photon, serverless or a better layout often makes them both faster and cheaper.

What interviewers want: using the billing system table rather than guessing, the common causes (all-purpose vs job compute, idle compute, slowing jobs), and tags, policies and budgets as the lasting fix.

VACUUM deleted every data file that the current version of the table no longer references. That included the files older versions point to, so time travel to them is gone, and the files that running jobs were still reading or had written but not yet committed. Recover what you can from the current version and other copies, rerun the failed job, and make sure a retention below 7 days cannot be set again.

Why it broke

A Delta table's history is a log of versions, each pointing to a set of data files. An UPDATE or MERGE does not change files in place; it writes new ones and marks the old ones as removed. Those old files are what time travel and RESTORE read. VACUUM deletes removed files older than the retention period, 7 days by default. With 0 hours:

• Time travel stops working for every earlier version. The log still lists them for 30 days by default, but the data files are gone, so queries and restores fail.
• Running jobs break. A query that started on an older snapshot may need files that were just deleted. A long write may have written files it had not yet committed; VACUUM saw them as unreferenced and removed them.

The safety check retentionDurationCheck exists to block this; someone turned it off.

What to do now

1. Stop scheduled VACUUM on the table and check the table's history for the VACUUM operation, its user and time.
2. Rerun the failed job. It starts from the current version, which is intact.
3. Recover lost versions only from other copies: cloud storage versioning or soft delete if it was enabled on the bucket or container, a deep clone, or a backup. If none exists, earlier versions cannot be restored.
4. Tell the people affected that rollbacks and audits on that table cannot reach before the VACUUM.

Prevent it

• Keep retention at 7 days or more, set on the table (deletedFileRetentionDuration on Databricks Runtime 18.0 and above), and longer than your longest job and your rollback needs.
• Never disable retentionDurationCheck in shared code. Restrict who can run maintenance on production tables.
• Use VACUUM ... DRY RUN to see what would be deleted before a manual run.
• On Unity Catalog managed tables, let predictive optimization run VACUUM instead of ad hoc commands.
• If saving storage was the goal, measure it first: often the bigger saving is from fewer table versions or from tables nobody reads.

What interviewers want: that time travel depends on old data files, what VACUUM deletes, why 0 hours breaks running jobs, the retention check, honest recovery options, and prevention through table settings and permissions.

The job now runs with the service principal's privileges, not yours, and the service principal lacks some of what you had. Find exactly which object and privilege the error names, grant that to a group the service principal belongs to, and repeat until the job runs. Do not fix it by running production as a person or by granting broad admin rights; the whole point of the service principal is a small, known set of permissions.

Why it worked for you

A Lakeflow job uses the privileges of its Run as identity for every task. By default that is the creator, so while you developed it, the job quietly used everything you can reach: tables you were granted years ago, catalogs your team owns, the main catalog that everyone can use. Switching Run as to a service principal reveals all of those hidden dependencies at once.

Find what is missing

Read the error: it names the privilege and the object, for example USE SCHEMA on sales.raw or SELECT on sales.raw.orders. A frequent surprise is that SELECT on a table is not enough. The identity also needs USE CATALOG on the catalog and USE SCHEMA on the schema.

Typical set for a pipeline that reads raw tables and writes curated ones:

Grant it the right way

• Grant to a group, such as etl-sales-prod, and add the service principal to it. Grants then survive replacing the principal and are easy to review.
• Grant at the narrowest level that is still practical. Schema-level SELECT is inherited by every table in it, which fits a schema the job owns entirely; table-level grants fit shared schemas.
• Write grants as code (SQL scripts or Terraform) next to the job, so a new environment gets the same permissions and changes are reviewed.

Avoid it next time

Develop and test with the same identity, or run the job as the service principal in a test workspace before release. Then the first permission error appears during testing, not at 2 a.m. on the first production run.

What interviewers want: that Run as decides the privileges, why "works for me" hides dependencies, the catalog and schema USE privileges on top of SELECT, group-based least-privilege grants kept as code, and testing with the production identity.

It depends on the kind of table reading the source and the kind of change: appended data is picked up by the next update, while updates or deletes to a source that a streaming table reads make that flow fail, unless you tell it to skip them.

Lakeflow declarative pipelines (the product that used to be called Delta Live Tables) have two main kinds of dataset, and they react differently.

Streaming tables read their source incrementally. Each update processes only data added since the last update, and remembers how far it got in a checkpoint.

• New rows appended to the source while an update runs: whatever is committed after the update read its batch is processed by the next update, or within seconds in continuous mode. Nothing is lost or read twice.
• Rows updated, deleted, merged or overwritten in a Delta source: a streaming read accepts only appends, so the flow fails with an error. Two fixes:
  • set skipChangeCommits on the read to ignore commits that change existing data (they then never reach the streaming table), or
  • read the source's change data feed, or use AUTO CDC, if those changes need to flow through.

Materialized views compute a result from their sources as of the moment each update reads them, like a query. A change made during an update shows up in the next update. Because they recompute what changed, updates and deletes in the source are reflected correctly.

What this means in practice:

• Each update reads a consistent snapshot of a Delta source, so a run never sees half of someone else's write.
• Choose the update mode for freshness: triggered updates process what is there and stop; continuous pipelines keep running and pick up new data as it lands.
• If a source is changed in place regularly, do not read it as a stream. Read it into a materialized view, or have the source publish changes (change data feed or CDC) and apply them with AUTO CDC.

A good interview answer names both cases: appends are handled incrementally and safely, but in-place changes to a streaming source break the stream unless the pipeline is designed for them.

Let the new column in on purpose: enable schema evolution for that write (mergeSchema, or automatic schema evolution for MERGE), or use Auto Loader's schema evolution, and check the new data's quality before it reaches downstream tables.

Why the write fails. Delta Lake enforces the table's schema on write. A DataFrame with a column the table does not have is rejected, which protects the table from accidental changes.

1. Evolve the schema on append

(new_data.write
   .option("mergeSchema", "true")
   .mode("append")
   .saveAsTable("sales.orders"))

New columns are added to the end of the schema; existing rows read them as NULL. Incompatible changes, such as a string arriving where the table has an integer, still fail. (Databricks can also widen some types, such as INT to BIGINT, when type widening is enabled on the table.)

For MERGE INTO, use MERGE WITH SCHEMA EVOLUTION INTO ... (or enable automatic schema evolution) so new source columns are added.

2. Let Auto Loader handle it at ingestion

(spark.readStream.format("cloudFiles")
   .option("cloudFiles.format", "json")
   .option("cloudFiles.schemaLocation", "/Volumes/main/raw/_schemas/orders")
   .load("/Volumes/main/raw/orders/"))

Auto Loader stores the inferred schema in schemaLocation. By default (addNewColumns), when a new column appears the stream stops once, records the new schema, and continues after a restart (jobs can restart automatically). Values that do not fit the schema go into a _rescued_data column instead of being lost. Other modes are rescue (never change the schema, keep new fields in _rescued_data) and failOnNewColumns.

3. Keep quality in check

• Land new columns in bronze with evolution on, and decide in silver whether to use them.
• Add expectations or CHECK constraints for the columns that matter.
• Alert on schema changes, for example by watching DESCRIBE HISTORY for metadata changes or the Auto Loader schema log.

When not to evolve: for curated silver and gold tables that others depend on, keep an explicit schema and change it deliberately with ALTER TABLE ... ADD COLUMN, so a source change cannot silently reshape a table people report from.

Turn on optimized writes and auto compaction for the table. Optimized writes make each micro-batch write fewer, larger files, and auto compaction merges remaining small files right after a write succeeds, on the same cluster, so no separate compaction job is needed. On Unity Catalog managed tables, predictive optimization can also run OPTIMIZE for you.

Why streaming creates small files. Every micro-batch writes at least one file per task, so a stream triggered every few seconds produces thousands of small files. Queries then spend more time opening files than reading data.

1. Set the table properties

ALTER TABLE events SET TBLPROPERTIES (
  'delta.autoOptimize.optimizeWrite' = 'true',
  'delta.autoOptimize.autoCompact' = 'true'
);

• Optimized writes shuffle data before writing so each partition gets fewer, larger files. The cost is a small shuffle in each write.
• Auto compaction runs after a write succeeds, synchronously on the cluster that did the write, and combines small files that have not been compacted before. It adds a little time to the micro-batch rather than a separate job.

On Databricks both are already on by default for MERGE, UPDATE and DELETE, so streams that upsert with foreachBatch and MERGE already benefit. Plain appends from a stream need the properties set.

You can also turn them on for a whole session with spark.databricks.delta.optimizeWrite.enabled and spark.databricks.delta.autoCompact.enabled.

2. Use predictive optimization

For Unity Catalog managed tables, predictive optimization runs OPTIMIZE (and VACUUM) automatically in the background on serverless compute, deciding when a table needs it. It is independent of auto compaction; the two can be used together.

3. Write less often

A longer trigger interval, such as .trigger(processingTime="1 minute"), writes fewer, bigger files. Choose the longest interval your freshness needs allow.

4. Lay the data out well

Use liquid clustering (CLUSTER BY) rather than partitions on high-cardinality columns. Over-partitioned tables are a common cause of small files, because each micro-batch writes a file into every partition it touches.

With these in place, a streaming table stays fast to query without a separate nightly compaction job, though for very large tables an occasional OPTIMIZE (or predictive optimization) is still worthwhile.

No. Horovod only parallelizes deep learning training code that you change to use its API, and it is no longer the recommended route on Databricks: Horovod and HorovodRunner are deprecated and are not installed in Databricks Runtime 16.0 ML and later. Databricks now points to TorchDistributor for PyTorch, the DeepSpeed distributor for very large models, tf.distribute.Strategy for TensorFlow, and Ray.

What Horovod did, and what any of these tools still require

They all use data parallelism: each worker holds a copy of the model, trains on its own share of the data, and the workers average their gradients (an all-reduce) after each step so the copies stay identical. That only works for models trained by gradient descent in a training loop, so:

• Your code must change: initialise the distributed backend, wrap the model (for example PyTorch's DistributedDataParallel), give each worker its own share of the data, and save checkpoints from one worker only.
• Traditional models such as scikit-learn's random forests are not parallelised this way.
• Arbitrary Python code is not distributed at all.

The current way with PyTorch

from pyspark.ml.torch.distributor import TorchDistributor

def train(epochs):
    import torch.distributed as dist
    from torch.nn.parallel import DistributedDataParallel as DDP
    dist.init_process_group("nccl")
    model = DDP(build_model().cuda())
    # ... training loop with a DistributedSampler ...
    dist.destroy_process_group()

TorchDistributor(num_processes=8, local_mode=False, use_gpu=True).run(train, 5)

TorchDistributor starts the training function on the cluster's workers as a Spark job and sets up the environment that PyTorch's own distributed tools expect.

Other options

• DeepSpeed distributor for models too large for one GPU's memory.
• Ray on Databricks for scaling training and tuning across frameworks.
• Spark ML (pyspark.ml.connect) for classic models on Spark DataFrames.

Before distributing at all, Databricks advises training on a single machine whenever the model and data fit, ideally one with several GPUs. Distribution adds communication overhead and complexity, and it pays off only when one machine is too slow or too small.

How to answer in an interview: say what data parallelism is, that the training code must be adapted, that it applies to gradient-trained models only, and that Horovod has given way to TorchDistributor and DeepSpeed on Databricks. Then add the judgement: measure single-machine training first, scale to several GPUs on one machine next, and use several machines only when the numbers show it is needed. That order is what keeps cost and debugging effort down.

Delta uses optimistic concurrency: each job reads a snapshot, writes new files, and checks at commit time whether another job changed the data it read. If so, the later commit fails. ConcurrentAppendException means the other job added files in the part of the table your MERGE read. The fix is to make the two jobs touch different data, or let Delta detect conflicts per row instead of per file, and retry the operation that fails.

Why two MERGEs conflict even on different rows

Without row-level concurrency, conflicts are detected per file. Job A updates customer 17 and job B updates customer 942, but if both rows live in the same file, both jobs rewrite that file and one of them fails. On an unpartitioned table, a MERGE that scans the whole table "reads" every file, so almost any concurrent write conflicts with it.

Diagnosis

1. Read the exception and the table history (DESCRIBE HISTORY) for the two commits: which operations, which partitions, at what times.
2. Check the table: partitioned or not, deletion vectors on or off, liquid clustering, identity column.
3. Check the MERGE condition: does it filter the target to the slice it needs, or does it scan everything?

Fixes, in order of preference

Retry, but retry the right thing

Under the default isolation level, plain appends never conflict; the operation that read the changed data is the one that fails. So put retry with backoff around the MERGE, UPDATE or DELETE. A retry re-reads the latest snapshot, which is safe for an idempotent MERGE.

Other causes to rule out

• A schema change, ALTER TABLE or table property change during the run fails every concurrent writer. Schedule them in a quiet window.
• Tables with identity columns do not support concurrent transactions at all.
• OPTIMIZE ... ZORDER BY can conflict with MERGE even with row-level concurrency.

What interviewers want: optimistic concurrency at commit time, why file-level conflict detection hits different rows, row-level concurrency and its requirements, explicit filters on the target in the MERGE condition, and retrying the reading operation.

The table is laid out for one filter and not the other. Databricks skips files using the minimum and maximum of each column in each file. If the table is partitioned or ordered by date, a date filter skips most files; but every file holds a wide spread of customer IDs, so a customer filter has to read nearly all of them. The fix is a layout that groups both columns, which on Databricks is liquid clustering on the columns people filter by.

1. Confirm it in the query profile

Run the slow query and open its profile. On the scan, compare files and bytes read with the table's total, and look at the share pruned. A customer filter that prunes almost nothing is a layout problem. A scan that prunes well but is followed by a huge join or sort is a different problem.

2. Check the usual causes

3. Fix the layout

-- partitioned table, Databricks Runtime 18.1 and above
ALTER TABLE events REPLACE PARTITIONED BY WITH CLUSTER BY (event_date, customer_id);
-- unpartitioned table
ALTER TABLE events CLUSTER BY (event_date, customer_id);

OPTIMIZE events;

Liquid clustering co-locates rows by both keys, so files have narrow ranges for each. Keys can be changed later without rewriting the table, and OPTIMIZE clusters new data incrementally. On Unity Catalog managed tables, CLUSTER BY AUTO lets predictive optimization pick keys from the actual queries. Keep the key list short: on smaller tables, extra keys weaken skipping for single-column filters.

If statistics were the issue, set delta.dataSkippingStatsColumns to the filter columns and run ANALYZE TABLE events COMPUTE DELTA STATISTICS so existing files get them too.

4. Measure

Rerun the same query and compare files read, bytes read and time. Keep a few representative queries as a benchmark, since a layout that suits today's filters may not suit next year's.

What interviewers want: file-level min/max skipping as the mechanism, reading pruning from the query profile, the 32-column statistics default, why date-ordered data cannot skip on customer ID, and liquid clustering instead of more partitions.

The changes were applied in the order they arrived, not the order they happened. CDC events can arrive late, be redelivered, or come in the same batch as later changes to the same row. A MERGE that simply takes "the row in this batch" lets an old update overwrite a newer one, and lets an update that arrives after a delete bring the row back. Apply each key's changes by a sequence number from the source and ignore anything older than what the table already holds.

Three bugs that cause this

The fix with plain MERGE

Keep the source's ordering column (a log sequence number, or commit timestamp plus an ID) on the target, and use it both within a batch and against the table:

from pyspark.sql import functions as F, Window

w = Window.partitionBy("customer_id").orderBy(F.col("lsn").desc())
latest = (changes.withColumn("rn", F.row_number().over(w))
                 .filter("rn = 1").drop("rn"))

(target.alias("t")
   .merge(latest.alias("s"), "t.customer_id = s.customer_id")
   .whenMatchedDelete(condition="s.op = 'DELETE' AND s.lsn > t.lsn")
   .whenMatchedUpdateAll(condition="s.op <> 'DELETE' AND s.lsn > t.lsn")
   .whenNotMatchedInsertAll(condition="s.op <> 'DELETE'")
   .execute())

This handles duplicates and late updates. For a delete followed by a late update, the row is already gone, so whenNotMatchedInsertAll would bring it back. Keep a soft-delete flag (is_deleted = true with its lsn) instead of physically deleting, and filter it out in views; then the late update loses on sequence.

The managed option on Databricks

In a Lakeflow pipeline, AUTO CDC does this bookkeeping for you: give it the keys and SEQUENCE BY lsn (or a STRUCT of columns to break ties), mark which rows are deletes, and choose SCD type 1 or type 2. Late events with a lower sequence than the current state are ignored.

Repair the damaged table

Rebuild the affected keys from the CDC history in order, or take a fresh snapshot from the source and reconcile. Then add a daily check that compares row counts and a sample of rows with the source.

What interviewers want: that arrival order is not event order, a sequence column from the source, deduplicating per key within a batch, conditional MERGE on the sequence, the late-update-after-delete case, and knowing AUTO CDC.