dbt vs Spark Declarative Pipelines: how they differ and when to use which

Both tools are declarative: you write what each table should hold, not the steps to build it, and the tool reads the references between tables to work out the order. The difference is where the work happens and what kind of data each one expects. dbt sends SQL to a warehouse in batches. Spark Declarative Pipelines runs on Spark and treats data that keeps arriving as the normal case.

What each one is

dbt turns select statements into tables. Each model is a SQL file; ref() names the models it reads, which gives dbt the order, the lineage and the selection (dbt build --select daily_revenue+). dbt does none of the computing itself: it compiles each model and runs it in your platform, whether that is Snowflake, BigQuery, Databricks, Redshift or DuckDB on a laptop. Tests, snapshotsSlowly Changing DimensionsThe ways a warehouse handles a change to a descriptive record, such as a customer's address: type 1 overwrites it, type 2 adds a new dated version, and type 3 keeps the previous value in an extra column. and documentation live in the same project. The current generation, dbt v2, is a Rust engine that reads your SQL and catches incorrect SQL before it reaches the warehouse.

Spark Declarative Pipelines is part of open-source Spark, and Databricks' Lakeflow pipelines, formerly Delta Live Tables, are built on it. You define datasets: a streaming table that grows as new data arrives, a materialized view that holds a query's result, or a temporary view for steps in between. The pipeline reads the definitions, finds the order and runs them, keeping checkpoints so a streaming table never reads the same row twice. On Databricks the same code runs as a Lakeflow pipeline, which adds expectations for data quality, AUTO CDC for change feeds and a continuous mode.

Side by side

Which one fits

The same table in each

{{ config(materialized='incremental', unique_key=['order_date', 'country']) }}

with orders as (
    select
        order_id,
        country,
        cast(amount as decimal(10, 2)) as amount,
        cast(ordered_at as date) as order_date
    from {{ source('shop', 'orders') }}
    where order_id is not null
    {% if is_incremental() %}
      -- only the days from the newest one already built: that day is rebuilt, later ones added
      and cast(ordered_at as date) >= (select max(order_date) from {{ this }})
    {% endif %}
)

select order_date, country, sum(amount) as revenue, count(*) as orders
from orders
group by order_date, country

from pyspark import pipelines as dp
from pyspark.sql import DataFrame, functions as F


@dp.table
def orders() -> DataFrame:
    # A streaming table: each run reads only the files that arrived since the last run.
    return (
        spark.readStream.format("csv")
        .option("header", True)
        .schema("order_id STRING, country STRING, amount DECIMAL(10,2), ordered_at TIMESTAMP")
        .load("../data/orders")
    )


@dp.materialized_view
def daily_revenue() -> DataFrame:
    # A materialized view: the pipeline keeps it equal to this query over the orders table.
    return (
        spark.table("orders")
        .where("order_id IS NOT NULL")
        .groupBy(F.to_date("ordered_at").alias("order_date"), "country")
        .agg(F.sum("amount").alias("revenue"), F.count("*").alias("orders"))
    )

The difference that matters most is in the handling of new data. The dbt model has to say what is new: the is_incremental() block reads from the newest day already built, and the unique_key makes rebuilding that day safe. Get that filter wrong and the model loses or doubles rows, which is why dbt's docs advise a full refresh when an incremental model's logic changes. The Spark file says nothing about new data at all. The streaming table's checkpoint knows which files it has read, and the second run read only the second day's file.

The other difference is what each file assumes is around it. The dbt model runs wherever its profile points, and the same file works on a laptop with DuckDB and on Snowflake. The Spark file needs Spark, and on a laptop a Hive catalog so the tables survive between runs; on Databricks, streaming tables and materialized views are Unity Catalog managed tables.

Things that trip people up

• A dbt incremental model is only as right as its filter. Rows that arrive late, or a filter on the wrong column, are missed for good. See idempotent pipelinesIdempotent Pipelines and Exactly-OnceA pipeline step is idempotent when running it twice gives the same result as running it once. That is what makes retries safe, and it is how systems that deliver a message more than once still count it exactly once. for why the rebuild of a recent window is the safe pattern.
• New columns in an incremental dbt model are ignored by default. on_schema_change decides, and no setting fills old rows. Plan a full refresh with any change of columns.
• Spark pipeline code runs more than once. The pipeline evaluates the definitions during planning and runs, so a dataset function should only build its DataFrame: no collect(), count(), toPandas() or writes inside it.
• Expectations are not in open-source Spark. They, AUTO CDC and continuous mode come with Lakeflow on Databricks. Code that uses them will not run on plain Spark.
• A full refresh of a streaming table reads everything again. --full-refresh clears its data and checkpoint, so it rereads every file still in the source. Make sure the source still has them.

Using both

They are not rivals on Databricks so much as layers. Ingest with Lakeflow pipelines into bronze and silverMedallion ArchitectureOrganising a lakehouse in 3 layers of rising quality: bronze for raw data, silver for cleaned and joined data, and gold for tables shaped for reports and AI. tables as data arrives, then build the gold tables with dbt, where analysts review the SQL and the tests run with every build. Off Databricks, dbt in the warehouse is usually the whole answer, and an orchestrator runs it on a schedule.