Databricks Certified Data Engineer Associate

Not this time. Databricks recommends serverless compute for most automated workloads: fewer settings, faster start-up, automatic scaling and lower cost. Classic jobs compute is for non-SQL jobs that need custom cluster settings serverless does not offer, which this one does not. All-purpose compute, shared or not, is the one Databricks says to avoid for jobs, since it is not optimised for automated work, and a shared cluster mixes the job with other people's sessions. Sizing classic compute for the peak pays for that size every night.

Not this time. COPY INTO is retriable and idempotent: files in the source that it has already loaded are skipped on later runs, so running it again picks up only what the failed run missed. Setting force to true turns that off and loads files whether or not they were loaded before, which duplicates rows. Truncating and reloading the whole folder works but throws away history and rereads everything. A plain INSERT over read_files keeps no record of loaded files, so every run would reload the folder.

Not this time. The default mode is addNewColumns. Before the stream fails, Auto Loader adds the new column to the schema in the schema location, so a restart carries on with the updated schema. That is why Databricks recommends running Auto Loader streams from Lakeflow Jobs that restart on failure. Deleting the checkpoint reprocesses every file. Rescue mode never evolves the schema: the new column goes into the rescued data column instead. None ignores new columns altogether.

Not this time. union resolves columns by position, as SQL does, so a different column order puts values under the wrong names. unionByName resolves them by name; with allowMissingColumns=True it also fills a column that one side lacks with nulls. unionAll is the same as union. distinct() removes duplicate rows, not misplaced values, and a full outer join on every column is not a union at all.

Not this time. A materialized view stores precomputed results, which is what keeps a dashboard fast, and a refresh on update (TRIGGER ON UPDATE) keeps it current when the source changes on no fixed schedule. A view recomputes the aggregate on every query. A streaming table handles each input row once, which suits append-only ingestion, not totals that must change when earlier orders are corrected. A nightly rebuild is stale for most of the day.

Not this time. The default threshold is 10 MB, so a 30 MB table is not broadcast automatically. Raising the threshold above its estimated size, or putting a broadcast hint on the small side, sends the stores table to every executor and removes the shuffle of the 40 GB side. Then measure again, as the guide asks. Setting the threshold to -1 turns broadcasting off. More shuffle partitions change the size of each task, not the join strategy, and driver memory alone changes neither.

Not this time. A table update trigger watches Unity Catalog tables for changes such as updates, merges and deletes, and can start the job when any or all of the watched tables have been updated, with a minimum time between triggers if needed. It costs nothing extra while it waits. The job's Run as principal needs SELECT on both tables, plus USE CATALOG and USE SCHEMA, or the trigger cannot evaluate. A file arrival trigger is for files landing in storage, a continuous job runs and costs all day, and polling every 15 minutes starts runs that find nothing.

Not this time. All done runs the task after all its dependencies have run, whatever their result. All succeeded is the default and marks the task Upstream failed if any load fails. None failed also skips it after a failure. At least one failed runs it only after a failure, so it would never clean up after a good night.

Not this time. Custom variables are declared in the variables mapping and referenced with ${var.<name>}. Each target can set its own value under targets.<target>.variables, so the same code is promoted from dev to prod and only the target changes. Separate files per branch drift apart, an if statement on the workspace URL hides configuration in code, and a hand edit in the UI is overwritten by the next deployment.

Not this time. Liquid clustering replaces partitioning and ZORDER and cannot be combined with either. From Databricks Runtime 18.1, REPLACE PARTITIONED BY WITH CLUSTER BY converts a partitioned table in place, for managed and external tables, and CLUSTER BY AUTO lets predictive optimization choose the keys on a managed table. Plain CLUSTER BY is for unpartitioned tables. A second partition column on a high-cardinality key makes the small-file problem worse.

Not this time. SELECT lets a principal query a table, but in Unity Catalog every object inside a catalog or schema also needs USE CATALOG on the catalog and USE SCHEMA on the schema. Neither gives access to data by itself, so granting them adds nothing beyond what SELECT allows. MODIFY is for changing data, and ALL PRIVILEGES on the metastore or table ownership grant far more than reading.

Not this time. When a managed table is dropped, Databricks deletes its data files after the recovery period, 7 days by default and set at catalog or schema level, and UNDROP recovers it within that period. Dropping an external table removes only its metadata; the files stay in your storage, and UNDROP can bring an external table back too if you have CREATE EXTERNAL TABLE on its location. RESTORE TABLE returns a table that still exists to an earlier version, it does not undo a drop.