Databricks Certified Associate Developer for Apache Spark

Not this time. filter and groupBy(...).count() on a DataFrame are transformations: Spark only adds them to the plan, which is lazy evaluation. Nothing runs until an action asks for a result, and writing the DataFrame is one, so the write starts the job and Spark plans the filter and the aggregation together. A shuffle splits the job into stages; it does not start a job by itself.

Not this time. An action starts a job. Spark splits the job into stages at shuffle boundaries: narrow transformations such as filter, select and withColumn run in the same stage, one task per partition, and a wide transformation that has to move rows between partitions ends the stage. Executors run tasks; they do not shape the plan.

Not this time. cache() is lazy, like a transformation: it marks the DataFrame, and the first action that computes it, here count(), stores the partitions. For a DataFrame, cache() uses the default storage level, MEMORY_AND_DISK_DESER, so partitions that do not fit in memory go to disk. persist() is the same call with a storage level you choose.

Not this time. Spark SQL can query a file or folder directly by putting the format before the path in backticks: json.path, parquet.path, csv.path, orc.path or text.path (and delta.path where Delta Lake is installed). The other 2 forms are not Spark SQL syntax.

Not this time. Temporary views are session-scoped: they disappear when the session that made them ends and other sessions cannot see them. A global temporary view lasts until the Spark application ends and is shared by its sessions; it lives in the system database global_temp, so it must be named with that prefix. Caching changes how the data is stored, not who can see the view.

Not this time. The default save mode is errorifexists (also written error): if the target already exists, the write fails. append adds rows, overwrite replaces the data, and ignore is the mode that quietly does nothing when the target exists. Set the mode you mean every time.

Not this time. from_unixtime converts a number of seconds since the Unix epoch (1970-01-01 00:00:00 UTC) to a string in the format you give, so milliseconds must be divided by 1,000 first. Passing milliseconds straight in reads them as seconds and gives a date tens of thousands of years ahead. to_date and date_format expect a date, timestamp or date string, not an epoch number.

Not this time. dropDuplicates keeps one row per key but does not promise which one, and sorting first does not change that, because the rows are shuffled to deduplicate them. row_number() over a window partitioned by the key and ordered newest first numbers each order's rows, so keeping number 1 gives the newest full row every time. groupBy with max returns only the key and the newest time, not the rest of the row.

Not this time. Spark guarantees that each task's update to an accumulator is applied once only when the update is made inside an action, such as foreach. In a transformation, a task that is re-run, for a failure, a lost executor or a recomputed stage, adds its update again. Count in an action, or count with a filter and count() instead.

Not this time. coalesce(50) merges existing partitions through a narrow dependency, so going from 2,000 to 50 partitions needs no shuffle; each new partition takes several of the old ones. repartition(50) does the same with a full shuffle, which gives even sizes but costs more. spark.sql.shuffle.partitions sets how many partitions a shuffle makes; it does not merge the partitions you already have, and repartition(2000, ...) keeps 2,000 files.

Not this time. AQE re-plans the query from statistics collected at runtime. Its main features are coalescing post-shuffle partitions, converting a sort-merge join to a broadcast hash join when one side turns out smaller than the adaptive broadcast threshold, and splitting skewed partitions in sort-merge joins. Adding executors is dynamic allocation, a separate feature, and AQE never rewrites UDFs or caches data.

Not this time. Without a watermark, Spark cannot know when a duplicate can no longer arrive, so it keeps every past event_id as state. A watermark bounds that: with dropDuplicatesWithinWatermark, Spark drops duplicates within the watermark's delay and clears older state. Complete mode, a trigger interval and caching do not limit deduplication state.

Not this time. The file sink supports only append mode. With an event-time aggregation and a watermark, append mode writes each window once, when the watermark has passed its end and the count can no longer change. Complete and update modes rewrite results, which files written once cannot do.

Not this time. Spark Connect separates the client from the Spark driver: the client sends DataFrame operations to the server as unresolved logical plans over gRPC. Because the client is not in the driver's process, it has no SparkContext and does not support the RDD API, so code that uses them has to be rewritten with DataFrames. DataFrame code, SQL and Structured Streaming work over Spark Connect.

Not this time. A pandas UDF is run by Spark with Apache Arrow moving the data and pandas working on whole batches of a column, which is much faster than calling Python once per row. toPandas() loads every row into the driver's memory, which fails or crawls on large data. Calling the same UDF from SQL changes nothing, and more memory does not remove the per-row cost.