pandas vs Polars vs PySpark: which to use for your data size

Use pandas when your data fits comfortably in memory, Polars when you want one machine to go faster or handle more than its memory, and PySpark when the data is too big for one machine or already lives on a Spark platform. The deciding number is the size of the table in memory compared with the RAM you have, not the number of rows.

All 3 are DataFrame libraries for Python, and since this autumn 2 of them have changed in ways worth knowing: pandas 3.0 changed how text and copies work, and Polars 2.0 made streaming the default.

How they differ

pandas' own guide says it plainly: pandas works in memory, and even a dataset that is a sizable fraction of memory becomes unwieldy, because some operations make intermediate copies. Polars and Spark both build a plan first, so they can read only the columns and rows a query needs (projection and predicate pushdown) before doing any work.

The same job in each

Revenue by country from paid orders, read from Parquet:

import pandas as pd

orders = pd.read_parquet("orders.parquet", columns=["country", "amount", "status"])
result = (
    orders[orders["status"] == "paid"]
    .groupby("country", as_index=False)["amount"].sum()
    .sort_values("amount", ascending=False)
)

import polars as pl

result = (
    pl.scan_parquet("orders.parquet")
    .filter(pl.col("status") == "paid")
    .group_by("country")
    .agg(pl.col("amount").sum())
    .sort("amount", descending=True)
    .collect()
)

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.getOrCreate()
result = (
    spark.read.parquet("s3://my-bucket/orders/")
    .filter(F.col("status") == "paid")
    .groupBy("country")
    .agg(F.sum("amount").alias("amount"))
    .orderBy(F.desc("amount"))
)
result.show()

In pandas you choose the columns yourself; Polars' scan_parquet and Spark's reader work them out from the query. The code is close enough that moving between the 3 is mostly renaming.

A worked sizing

An orders table has 6 number and date columns and 3 text columns averaging 12 characters. In memory each number takes 8 bytes, and each text value its bytes plus up to 16 bytes of bookkeeping, so a row is about 48 + 3 × 28 = 132 bytes.

• 50 million rows: about 6.1 GB. On a 32 GB laptop that is under a fifth of memory: pandas works, and Polars will be quicker on many cores.
• 400 million rows: about 49 GB, more than the 32 GB machine holds. pandas cannot load it at once. Polars' lazy API with the streaming engine reads it in batches on the same laptop.
• 50 billion rows: about 6 TB. One machine streaming it would take hours just to read it; this is Spark's job. The Spark cluster calculator sizes the cluster.

Parquet on disk is usually much smaller, because it compresses, so judge by the in-memory size.

Try itWill your table fit, and which tool then?

Rows (millions)

50 million

Number and date columns

6

Text columns

3

Average characters per text value

12

Machine memory (GB)

32 GB

Table6.1 GB

Bar = machine memory; dashed line = a third of it.

Use: pandas or Polars Fits

What changed this year

pandas 3.0 (January 2026; 3.1 came out on 22 October): text columns are now a dedicated str type, backed by PyArrow when installed, so code that checks for object dtype needs updating. Copy-on-write is the only behaviour: every result of indexing acts as a copy, so chained assignment no longer works and the SettingWithCopyWarning is gone. pd.col("a") + pd.col("b") replaces many lambdas. It needs Python 3.11 or newer.

Polars 2.0 (6 October 2026): LazyFrame.collect() now uses the streaming engine by default, which also handles data larger than memory. The catch: streaming does not keep row order for group_by, joins or unpivot. If a pipeline relied on rows coming out in the order they went in, add an explicit sort (as the example above does) or maintain_order=True where the operation offers it.

Which fits

• pandas if the data is small, you are exploring in a notebook, or a library you depend on wants a pandas DataFrame.
• Polars for single-machine pipelines, larger-than-memory files on one VM, and anywhere a Spark cluster's start-up and cost are not worth it.
• PySpark for data spread over many files in a lake, tables in Delta Lake or Iceberg, streaming, and teams already on Databricks. If Spark jobs run out of memory, our guide to Spark out of memory errors helps.

What to do

1. Estimate the in-memory size with the widget above, not the row count.
2. Below a third of RAM, stay with what your team knows.
3. Above that, try Polars' lazy API before reaching for a cluster.
4. Upgrading to Polars 2.0? Search for code that depends on row order. Upgrading to pandas 3? Search for object dtype checks.
5. To learn the Spark side properly, start with the PySpark DataFrames module of our data engineering course.