# PySpark on Databricks serverless, and when DuckDB or Polars is enough

Databricks serverless runs PySpark through Spark Connect, so your notebook sends each query plan to Spark running elsewhere. DataFrame and SQL code runs the same as on your laptop, but spark.sparkContext, RDDs, cache(), persist() and checkpoint() are not there, and a misspelled column is reported when an action runs. For data that fits on 1 machine, DuckDB or Polars does the same job with no Java and no cluster.

- Page: https://tecyfy.com/data-engineering/course/pyspark-dataframes/serverless-duckdb-polars
- Course: Data engineering course (https://tecyfy.com/data-engineering/course), Module 2: PySpark DataFrames, Serverless, DuckDB and Polars
- Parts of this module: Read files with a schema (https://tecyfy.com/data-engineering/course/pyspark-dataframes); Keep the newest row (https://tecyfy.com/data-engineering/course/pyspark-dataframes/keep-the-newest-row); Join, aggregate and write (https://tecyfy.com/data-engineering/course/pyspark-dataframes/join-aggregate-and-write); Serverless, DuckDB and Polars (https://tecyfy.com/data-engineering/course/pyspark-dataframes/serverless-duckdb-polars); Build it: clean the raw orders (https://tecyfy.com/data-engineering/course/pyspark-dataframes/build-it)

At the end of this part you can:

- Write PySpark that runs the same on your laptop and on Databricks serverless
- Say what a Python UDF costs, and find the built-in function to use instead
- Do the module's cleaning job in DuckDB and in Polars, and say when either is enough

Code you test on your laptop should be the code you ship. On Databricks serverless, which Free Edition uses, that holds for almost everything in this module, and this part covers the few things that break, the one habit that is slow everywhere, and 2 tools that make Spark unnecessary for small data.

## Spark Connect: the same code, sent somewhere else

On your laptop, PySpark runs Spark itself in the same program. Databricks serverless works differently: your notebook is a client that sends the plan to Spark running elsewhere, through Spark Connect. The client turns each DataFrame call into a plan, sends it over the network, and gets the results back. Your code looks the same, and almost all of it runs the same, but a few things are not there:

- **No `spark.sparkContext` and no RDDs.** These belong to the old, lower-level API that runs inside the same program as Spark. Anything written with `sc.parallelize` or `df.rdd` needs rewriting with DataFrames.
- **No `cache()`, `persist()` or `checkpoint()`.** Save to a table where you would have cached, as [part 1](/data-engineering/course/pyspark-dataframes) does with the bronze copy.
- **Errors appear at the action.** Spark Connect checks column names when the plan runs, so a misspelled column in line 10 is reported when line 30 calls `show()`. Read the error's message for the column name, not the line number.
- **No global temporary views**, and custom Python code such as a UDF has at most 1 GB of memory and no internet access.

Run the same cells in both places and compare:

(An interactive demo goes here on the page: https://tecyfy.com/data-engineering/course/pyspark-dataframes/serverless-duckdb-polars)

Every code block in this module runs both ways, because it uses only DataFrames, SQL and files. Keep to those and the difference stops mattering.

## Python functions, and why to avoid them

You can turn any Python function into a column function, a user-defined function or UDF. Start from the `good` orders of [part 2](/data-engineering/course/pyspark-dataframes/keep-the-newest-row):

```python
@F.udf("string")
def size_label(amount):
    return "large" if amount is not None and amount >= 30 else "small"

good.select("order_id", size_label("amount").alias("size")).explain()
```

Its top 3 lines:

```text
== Physical Plan ==
*(2) Project [order_id#66L, pythonUDF0#245 AS size#244]
+- ArrowEvalPython [size_label(amount#69)#243], [pythonUDF0#245], 101
   +- *(1) Project [order_id#66L, amount#69]
```

The plan has a new step, `ArrowEvalPython`. Spark runs on the Java virtual machine, so to run your function it sends every row out to a separate Python process and back, packed with Apache Arrow. The built-in version, `F.when(F.col("amount") >= 30, "large").otherwise("small")`, has no such step: it runs inside Spark, and Spark can optimise around it. `F.when` with `otherwise` is SQL's `CASE WHEN`; without `otherwise`, rows that match no branch get null.

Look for a built-in function first; there are hundreds, for dates, strings, arrays, JSON and maths. Reach for a UDF only when nothing built in does the job, such as calling a Python library.

- Quiz: [PySpark UDFs & Custom Functions: Essentials](https://tecyfy.com/quiz/pyspark/pyspark-udf-custom-functions-essentials). Master the basics of PySpark User-Defined Functions (UDFs) and custom functions for efficient data processing

## When DuckDB or Polars is enough

Spark is built for data that does not fit on 1 machine. Much data does fit. A daily file of a few gigabytes, a monthly report, a notebook exploring a sample: for these, starting Java and a Spark session is overhead you do not need. 2 tools do the same work on 1 machine, with no server and no Java. Install them with `pip install duckdb polars`.

DuckDB is a SQL database that runs inside your Python program. The whole job of parts 1 to 3, from the raw file to revenue per day, is 1 query:

```python
import duckdb

daily = duckdb.sql("""
    WITH latest AS (
        SELECT DISTINCT * FROM read_csv('raw/orders.csv', types = {'amount': 'VARCHAR'})
        WHERE try_cast(amount AS DECIMAL(10,2)) IS NOT NULL
        QUALIFY row_number() OVER (PARTITION BY order_id ORDER BY updated_at DESC) = 1
    )
    SELECT CAST(ordered_at AS DATE) AS day, c.country,
           count(*) AS orders, sum(CAST(amount AS DECIMAL(10,2))) AS revenue
    FROM latest LEFT JOIN read_csv('raw/customers.csv') AS c USING (customer_id)
    WHERE status <> 'cancelled'
    GROUP BY ALL
    ORDER BY day, c.country
""")
print(daily)
```

`QUALIFY` filters on a window function, so the newest-row step of part 2 is 1 line.

Polars is a DataFrame library. `scan_csv` makes a lazy query, planned and optimised like Spark's, that runs at `collect()`:

```python
import polars as pl

orders = (
    pl.scan_csv("raw/orders.csv", try_parse_dates=True)
    .with_columns(pl.col("amount").cast(pl.Decimal(10, 2), strict=False))
    .filter(pl.col("amount").is_not_null())
    .unique()
    .filter(pl.col("updated_at") == pl.col("updated_at").max().over("order_id"))
)
customers = pl.scan_csv("raw/customers.csv")

daily = (
    orders.join(customers, on="customer_id", how="left")
    .filter(pl.col("status") != "cancelled")
    .group_by(pl.col("ordered_at").dt.date().alias("day"), "country")
    .agg(pl.len().alias("orders"), pl.col("amount").sum().alias("revenue"))
    .sort("day", "country")
    .collect()
)
print(daily)
```

Both print the same 3 rows as Spark. Neither keeps the rejected row the way the Spark version does; they skip it, which is fine for a report and not for a pipeline someone has to fix.

So which? Use DuckDB or Polars when the data fits on 1 machine with room to spare and the job runs on its own. Use Spark when the data, or its growth, will not fit; when the job runs on Databricks, next to governed tables, jobs and pipelines; or when it has to stream. The skills carry over: a lazy plan, a schema, a window to keep the newest row and a check on every join are the same ideas in all 3.

- Tool: [Spark sizing calculator](https://tecyfy.com/data-engineering/spark-calculator). The Databricks workers to pick, or the executors, memory and shuffle partitions for your nodes, with every number explained.

## Check yourself

1. A notebook adds a column with df.rdd.map(...). It runs on a laptop, and on Databricks serverless it fails with "[NOT_IMPLEMENTED] rdd is not implemented." What is the fix?

   A. Make the RDD with spark.sparkContext.parallelize instead of df.rdd
   B. Call df.cache() first, so the RDD is built from the cached rows
   C. Rewrite the step with DataFrame functions, such as withColumn with F.when
   D. Install a newer PySpark in the notebook with %pip, which adds RDD support

### Answers

1. C. Serverless runs only Spark Connect, which has no RDD API and no sparkContext, and does not support cache(). Installing another PySpark does not change that, because the limit is in how the notebook talks to Spark, not in the version. Rewrite the step with DataFrame or SQL functions, which run the same everywhere and which Spark can optimise.

## Questions

### Why does my PySpark code work on my laptop but fail on Databricks serverless?

Serverless compute runs only Spark Connect, so code that uses spark.sparkContext, RDDs, df.cache(), persist or checkpoint fails there. Spark Connect also checks column names when the query runs, not when you write the line, so an error can appear later than you expect. Write DataFrame and SQL code only, and save to a table where you would have cached.

### Why are Python UDFs slow in PySpark?

Spark runs on the Java virtual machine, so to run a Python function on a column it sends the rows out to a separate Python process and back. Spark cannot look inside the function to optimise around it. A built-in function from pyspark.sql.functions runs inside Spark and avoids both costs, so check for one before writing a UDF.

### Should I use pandas, Polars, DuckDB or PySpark?

If the data fits comfortably on one machine and the job runs on its own, DuckDB or Polars is usually simpler and quicker to start, with no Java and no cluster. Choose PySpark when the data or the team outgrows one machine, when the job runs on Databricks next to governed tables, or when it has to stream. pandas suits small data you explore by hand.

## Sources

- [Spark Connect overview](https://spark.apache.org/docs/latest/spark-connect-overview.html)
- [Serverless compute limitations](https://docs.databricks.com/aws/en/compute/serverless/limitations)
- [Unleashing UDFs and UDTFs](https://spark.apache.org/docs/latest/api/python/user_guide/udfandudtf.html)
- [Why DuckDB](https://duckdb.org/why_duckdb)
- [CSV import](https://duckdb.org/docs/current/data/csv/overview.html)
- [Lazy API](https://docs.pola.rs/user-guide/concepts/lazy-api/)