PySpark on Databricks serverless, and when DuckDB or Polars is enough

Code you test on your laptop should be the code you ship. On Databricks serverless, which Free Edition uses, that holds for almost everything in this module, and this part covers the few things that break, the one habit that is slow everywhere, and 2 tools that make Spark unnecessary for small data.

Spark Connect: the same code, sent somewhere else

On your laptop, PySpark runs Spark itself in the same program. Databricks serverless works differently: your notebook is a client that sends the plan to Spark running elsewhere, through Spark Connect. The client turns each DataFrame call into a plan, sends it over the network, and gets the results back. Your code looks the same, and almost all of it runs the same, but a few things are not there:

• No spark.sparkContext and no RDDs. These belong to the old, lower-level API that runs inside the same program as Spark. Anything written with sc.parallelize or df.rdd needs rewriting with DataFrames.
• No cache(), persist() or checkpoint(). Save to a table where you would have cached, as part 1 does with the bronze copy.
• Errors appear at the action. Spark Connect checks column names when the plan runs, so a misspelled column in line 10 is reported when line 30 calls show(). Read the error's message for the column name, not the line number.
• No global temporary views, and custom Python code such as a UDF has at most 1 GB of memory and no internet access.

Run the same cells in both places and compare:

Try itRun the same cells on a laptop and on serverless

Runs on

Your laptopDatabricks serverless

Run cell 1Run all

1. [1]good = spark.table("bronze_orders")
   Not run
2. [2]typo = good.select("order_idd")
   Not run
3. [3]good.cache()
   Not run
4. [4]good.rdd
   Not run
5. [5]spark.sparkContext
   Not run
6. [6]typo.show()
   Not run
7. [7]good.groupBy("status").count().show()
   Not run

Every code block in this module runs both ways, because it uses only DataFrames, SQL and files. Keep to those and the difference stops mattering.

Python functions, and why to avoid them

You can turn any Python function into a column function, a user-defined function or UDF. Start from the good orders of part 2:

@F.udf("string")
def size_label(amount):
    return "large" if amount is not None and amount >= 30 else "small"

good.select("order_id", size_label("amount").alias("size")).explain()

Its top 3 lines:

== Physical Plan ==
*(2) Project [order_id#66L, pythonUDF0#245 AS size#244]
+- ArrowEvalPython [size_label(amount#69)#243], [pythonUDF0#245], 101
   +- *(1) Project [order_id#66L, amount#69]

The plan has a new step, ArrowEvalPython. Spark runs on the Java virtual machine, so to run your function it sends every row out to a separate Python process and back, packed with Apache Arrow. The built-in version, F.when(F.col("amount") >= 30, "large").otherwise("small"), has no such step: it runs inside Spark, and Spark can optimise around it. F.when with otherwise is SQL's CASE WHEN; without otherwise, rows that match no branch get null.

Look for a built-in function first; there are hundreds, for dates, strings, arrays, JSON and maths. Reach for a UDF only when nothing built in does the job, such as calling a Python library.

QuizPySpark UDFs & Custom Functions: EssentialsMaster the basics of PySpark User-Defined Functions (UDFs) and custom functions for efficient data processing

When DuckDB or Polars is enough

Spark is built for data that does not fit on 1 machine. Much data does fit. A daily file of a few gigabytes, a monthly report, a notebook exploring a sample: for these, starting Java and a Spark session is overhead you do not need. 2 tools do the same work on 1 machine, with no server and no Java. Install them with pip install duckdb polars.

DuckDB is a SQL database that runs inside your Python program. The whole job of parts 1 to 3, from the raw file to revenue per day, is 1 query:

import duckdb

daily = duckdb.sql("""
    WITH latest AS (
        SELECT DISTINCT * FROM read_csv('raw/orders.csv', types = {'amount': 'VARCHAR'})
        WHERE try_cast(amount AS DECIMAL(10,2)) IS NOT NULL
        QUALIFY row_number() OVER (PARTITION BY order_id ORDER BY updated_at DESC) = 1
    )
    SELECT CAST(ordered_at AS DATE) AS day, c.country,
           count(*) AS orders, sum(CAST(amount AS DECIMAL(10,2))) AS revenue
    FROM latest LEFT JOIN read_csv('raw/customers.csv') AS c USING (customer_id)
    WHERE status <> 'cancelled'
    GROUP BY ALL
    ORDER BY day, c.country
""")
print(daily)

QUALIFY filters on a window function, so the newest-row step of part 2 is 1 line.

Polars is a DataFrame library. scan_csv makes a lazy query, planned and optimised like Spark's, that runs at collect():

import polars as pl

orders = (
    pl.scan_csv("raw/orders.csv", try_parse_dates=True)
    .with_columns(pl.col("amount").cast(pl.Decimal(10, 2), strict=False))
    .filter(pl.col("amount").is_not_null())
    .unique()
    .filter(pl.col("updated_at") == pl.col("updated_at").max().over("order_id"))
)
customers = pl.scan_csv("raw/customers.csv")

daily = (
    orders.join(customers, on="customer_id", how="left")
    .filter(pl.col("status") != "cancelled")
    .group_by(pl.col("ordered_at").dt.date().alias("day"), "country")
    .agg(pl.len().alias("orders"), pl.col("amount").sum().alias("revenue"))
    .sort("day", "country")
    .collect()
)
print(daily)

Both print the same 3 rows as Spark. Neither keeps the rejected row the way the Spark version does; they skip it, which is fine for a report and not for a pipeline someone has to fix.

So which? Use DuckDB or Polars when the data fits on 1 machine with room to spare and the job runs on its own. Use Spark when the data, or its growth, will not fit; when the job runs on Databricks, next to governed tables, jobs and pipelines; or when it has to stream. The skills carry over: a lazy plan, a schema, a window to keep the newest row and a check on every join are the same ideas in all 3.

ToolSpark sizing calculatorThe Databricks workers to pick, or the executors, memory and shuffle partitions for your nodes, with every number explained.