DuckDB vs Spark: when one laptop is enough

DuckDB vs Spark comes down to one question: does the job fit on one machine? DuckDB is a SQL engine that runs inside your Python program, with no server and no Java, and it is the simpler choice while the data, and the memory the query needs, fit one computer's disk and RAM. Spark is built to spread one job over many machines, so it earns its start-up time and its cluster when the data outgrows a single box, when the job streams, or when many writers share governed tables.

DuckDB 2.0.0, released on 21 October 2026, moved that line. Its server mode, Quack, is now polished and has a CONNECT statement, it creates Delta tables, and it reads Parquet on S3 with asynchronous I/O. It is still one machine. This page sets out where each engine stops, so the choice is a fact about your job rather than a habit.

What each one is

DuckDB is a library. pip install duckdb and it runs inside the Python process that imports it: no database server to install, no dependencies, MIT licence. It reads Parquet, CSV and JSON files where they are, on disk or in object storage, and answers SQL over them using every core of the machine.

Spark is a cluster engine. A driver program plans the job and sends tasks to executors, Java processes on the machines of a cluster (cluster mode overview). It runs on one machine too, in local[*] mode, but then the parts built for many machines (the JVM, the scheduler, the shuffle that moves rows between executors) are overhead with nothing to spread over. If the shuffle is new to you, Spark ShuffleSpark ShuffleThe step that moves rows between executors so rows with the same key end up together, for a join, group by or repartition; it splits a job into stages. shows what it moves and why it costs time.

Side by side

When one machine is enough

A daily extract, a month of events, a sample for a model, a dbt project's development runs: for these, DuckDB runs them on a laptop, and you skip the cluster start, the JVM and the bill.

Three checks tell you whether yours fits:

1. The input fits the disk, and you read it once. DuckDB reads Parquet in place and only the columns a query names, so a Parquet folder far larger than RAM is workable when the query reads a few of its columns and filters on how the files are sorted or partitioned.
2. The working memory fits, or spilling is acceptable. DuckDB's docs ask for at least 125 MB per thread and suggest 1 to 2 GB per thread for aggregation-heavy work and 3 to 4 GB for join-heavy work (environment guide). On a 12-core laptop with 24 GB, that is under 2 GB per thread: enough for aggregation-heavy work, short of the 3 to 4 GB join-heavy work wants. By default DuckDB takes up to 80% of RAM (memory_limit).
3. One process writes. A DuckDB database file has one reader-writer process at a time, or many read-only ones (concurrency).

If all 3 hold, DuckDB is the simpler tool. If the first fails, no amount of tuning makes one machine enough, and Spark is the answer.

What happens when the data is larger than memory

DuckDB does not fail just because the data is bigger than RAM. The operators that must see all their input before they produce a row, joins, GROUP BY, ORDER BY and window functions, spill to a temporary folder on disk (tuning workloads). The query runs slower, at disk speed, which is why the docs ask for an SSD or NVMe disk for this kind of work.

The docs also list where it can still run out of memory:

• a query with several blocking operators at once (a big join feeding a big GROUP BY feeding a window), where their memory needs add up;
• string_agg() and other aggregates that cannot spill;
• aggregates that use sorting, such as one with an ORDER BY inside it, which need all rows first and can run out of memory on large datasets.

Two settings help before you reach for Spark. SET preserve_insertion_order = false; lets DuckDB return rows in any order when the query has no ORDER BY, which can lower memory use on big imports and exports. And SET memory_limit = '10GB'; with SET temp_directory = '/fast/disk/tmp'; keeps DuckDB from crowding out the rest of the machine and puts the spill on your fastest disk.

Spark spills too (Spark SpillSpark SpillWhen a task's sort, join or aggregation no longer fits in execution memory, Spark writes part of it to local disk and reads it back later, so the task finishes, only slower.), but its answer to a bigger job is more executors, each with its own memory (Spark Executor MemorySpark Executor MemoryThe memory each Spark worker process gets: a Java heap, split into a shared pool for shuffles, joins and cached data and a pool for your own objects, plus overhead outside the heap.). That is the line: DuckDB makes one machine go further, and Spark adds machines. The Spark cluster calculator sizes that cluster when you get there.

The same job in both

The job: keep the newest version of each order, join the customer's country, and sum revenue per day and country. The SQL is the same in both engines.

WITH latest AS (
  SELECT * FROM (
    SELECT *, row_number() OVER (PARTITION BY order_id ORDER BY updated_at DESC) AS rn
    FROM orders
  ) WHERE rn = 1
)
SELECT CAST(ordered_at AS DATE) AS day, c.country,
       count(*) AS orders, sum(amount) AS revenue
FROM latest JOIN customers AS c USING (customer_id)
WHERE status <> 'cancelled'
GROUP BY CAST(ordered_at AS DATE), c.country
ORDER BY day, c.country

In DuckDB, point 2 views at the Parquet folders and run it:

import duckdb

con = duckdb.connect()
con.sql("CREATE VIEW orders AS FROM read_parquet('orders/*.parquet')")
con.sql("CREATE VIEW customers AS FROM read_parquet('customers.parquet')")
daily = con.sql(open("job.sql").read()).fetchall()

In Spark, register the same files as views:

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
spark.read.parquet("orders").createOrReplaceTempView("orders")
spark.read.parquet("customers.parquet").createOrReplaceTempView("customers")
daily = spark.sql(open("job.sql").read()).collect()

Two things to notice. Moving a SQL job between the engines is mostly moving file paths, so starting in DuckDB does not lock you in. And in either engine, if 2 versions of an order share the same updated_at, row_number() picks either one and the totals can change from run to run; add a second column to the ORDER BY that breaks the tie. The PySpark course part on DuckDB and Polars writes a fuller version of this cleaning job both ways, with QUALIFY, which DuckDB supports and which turns the subquery into one line.

If your code is PySpark DataFrames rather than SQL, DuckDB ships duckdb.experimental.spark, which runs a subset of the PySpark API on DuckDB. Its docs mark it experimental, bug fixes only, to be deprecated, and point to SQLFrame instead, which does not support DuckDB 2.0 yet (Spark API). For now, SQL is the dependable bridge.

What DuckDB 2.0 changed

Released on 21 October 2026 and supported until at least October 2027 (announcement), 2.0 removes some of the old reasons to pick Spark:

• DuckDB as a server. A polished Quack protocol (first released in 1.5.3) and the new CONNECT statement let one DuckDB process serve a database to others, so several processes can write to it through that server. duckdb -serve and duckdb -connect set this up from the command line.
• Concurrent writers on shared tables. DuckLake 1.1, released with 2.0, with PostgreSQL as its catalog database, keeps table metadata there so several DuckDB instances read and write the same tables.
• Faster reads from object storage. Asynchronous I/O keeps several requests in flight for Parquet files, uncompressed CSV files and DuckDB files, which matters most on s3://.
• Delta writes. The delta extension now creates Delta tables, with partitioning and table properties, and now reads them as of a timestamp as well as a version.

It still does not stream, and it is still one machine per query. Before upgrading a shared setup, note one break: DuckDB versions before 2.0 cannot read database files created by 2.0, though existing files that 2.0 writes to stay readable, and the team suggests waiting for the first patch release, due in November, for critical work.

Reading lakehouse tables from DuckDB

A common pattern is Spark or Databricks writing the tables and DuckDB reading them for analysis, tests or a small service. That works, within limits (Delta extension):

• delta_scan('s3://bucket/path') reads a Delta table from S3, Azure or Google Cloud Storage, with a secret for the credentials.
• DuckDB can append to a Delta table but cannot UPDATE, DELETE or MERGE into it. A pipeline that upserts stays on Spark.
• A Unity Catalog managed table that uses catalog-managed commits fails when read straight from storage, because only the catalog tracks its commits. Attach the catalog with the unity_catalog extension, using the workspace URL, a token and the storage region, plus a secret for the storage, and query through it.
• Iceberg tables are read with iceberg_scan, or read and written through an Iceberg REST catalog.

If you are still choosing the table format, Delta Lake vs Apache Iceberg compares them, and Delta LakeDelta LakeAn open table format that turns Parquet files in cloud storage into a table with transactions, a version history, schema checks, and updates and deletes, so a data lake behaves like a database. explains what the log adds to plain Parquet.

When Spark is the answer

Pick Spark, or keep it, when any of these is true:

• The data does not fit one machine, or the job reads most of a table too big for one disk.
• The job streams. Structured Streaming, with state kept between batches, has no DuckDB equivalent; transformWithState shows what that state looks like.
• Many jobs update the same tables with MERGE, UPDATE and DELETE, as a medallion pipeline does.
• The tables are governed on Databricks, with Unity Catalog permissions, lineage and scheduled jobs around them.
• The team already writes PySpark, and one codebase for laptop and cluster is worth more than DuckDB's speed on small inputs. Spark Connect, which sends plans from a thin client to a Spark server, lets the same DataFrame code run against a remote Spark server, though RDD and SparkContext code does not carry over.

Spark's cost is the cluster, not the licence. If the bill is the reason you are looking at DuckDB, why your Databricks bill is too high finds the waste first, and the Databricks cost calculator prices the cluster you would keep.

DuckDB vs Snowflake

The other search next to this one is DuckDB vs Snowflake, and the answer is similar. Snowflake is a shared warehouse: many users and dashboards query governed tables at the same time. The Snowflake cost calculator prices one. DuckDB is not built for many small queries from many users at once; its docs say so plainly. For the 2 big warehouses side by side, see Snowflake vs Databricks.