Spark Declarative Pipelines: A Practical Guide, from Delta Live Tables to Apache Spark

Spark Declarative Pipelines (SDP) is a way to build data pipelines on Apache Spark by declaring the tables you want and the query that fills each one, and letting Spark work out the order, the checkpoints and the retries. It arrived in open-source Spark with version 4.1. On Databricks the same idea has been sold for years as Delta Live Tables (DLT), which is now called Lakeflow pipelines and is built on SDP.

This guide explains the idea, walks through a small pipeline you can run on a laptop, and then shows what Databricks adds on top of the open-source version.

What "declarative" means here

In an ordinary Spark job you write the steps: read these files, filter, join, write the result, and remember where you stopped so the next run does not read the same files again. You decide the order, and if one step fails you work out what to re-run.

In a declarative pipeline you write only the definitions:

• a streaming table: a table filled by a streaming read, so each run adds only the data that arrived since the last one;
• a materialized view: a query whose result is stored as a table and recomputed when the pipeline runs;
• a temporary view: a named step that other definitions can read, which exists only while the pipeline runs.

Spark reads every definition, sees which ones read which, and builds a graph. A run then executes the graph in the right order, runs independent parts in parallel, and keeps the streaming checkpoints for you. The Spark guide puts it as "you define what tables should exist and what their contents should be".

The query inside each definition is ordinary Spark SQL or a DataFrame, so nothing new has to be learned there. What changes is who owns the plumbing. It is the same split as in the Medallion ArchitectureMedallion ArchitectureOrganising a lakehouse in 3 layers of rising quality: bronze for raw data, silver for cleaned and joined data, and gold for tables shaped for reports and AI.: raw data lands in one table, cleaned data in the next, business numbers in the last, and each layer is just a definition that reads the one before.

Delta Live Tables, Lakeflow and SDP: one product, three names

The naming confuses people, so here it is in order:

1. Delta Live Tables (DLT) was the Databricks product for declarative pipelines, written with import dlt and @dlt.table.
2. Databricks renamed it to Lakeflow pipelines. Databricks says no migration is required: DLT code keeps working.
3. Databricks gave the core of the engine to Apache Spark, where it became Spark Declarative Pipelines, starting in Spark 4.1. Lakeflow pipelines now extend SDP and run on the Databricks Runtime.

The new Python names are the same in both places:

The SQL statements, CREATE STREAMING TABLE and CREATE MATERIALIZED VIEW, did not change. A few things still carry the old name on Databricks: the classic billing SKUs begin with DLT and the event log schemas keep dlt in their names.

A pipeline you can run on a laptop

The open-source version installs with pip. You need Java for Spark itself.

pip install "pyspark[pipelines]"
spark-pipelines init --name shop

init creates a folder with a spec file, spark-pipeline.yml, and a transformations/ folder with two examples. The spec names the pipeline, where its code is and where streaming checkpoints are kept:

name: shop
storage: file:///absolute/path/to/shop/pipeline-storage
libraries:
  - glob:
      include: transformations/**

Replace the examples with two files. The first, in Python, defines a streaming table that picks up order files as they land in a folder:

# transformations/orders.py
from pyspark import pipelines as dp
from pyspark.sql import DataFrame

LANDING = "/absolute/path/to/landing"


@dp.table
def raw_orders() -> DataFrame:
    # A streaming table: each run reads only the files it has not seen yet.
    return (
        spark.readStream.format("json")
        .schema("order_id INT, customer STRING, amount DOUBLE, status STRING, ordered_at TIMESTAMP")
        .load(LANDING)
    )

The second, in SQL, defines a materialized view of paid revenue per day that reads the streaming table:

-- transformations/revenue.sql
CREATE MATERIALIZED VIEW daily_revenue AS
SELECT date(ordered_at) AS order_date, count(*) AS orders, sum(amount) AS revenue
FROM raw_orders
WHERE status = 'paid'
GROUP BY date(ordered_at);

Python and SQL files sit side by side in one pipeline, and the SQL file refers to raw_orders by name. Nothing says "run the Python file first". Since Spark 4.2 the spark session is put into each Python file for you; in 4.1 each file had to assign it with spark = SparkSession.active().

Put a file with three orders in the landing folder (two paid on 1 October, one cancelled on 2 October) and run:

spark-pipelines run --conf spark.sql.catalogImplementation=hive

The log shows the graph at work: raw_orders runs first, then daily_revenue, then "Run is COMPLETED". daily_revenue holds one row, 2 orders and 55.5 for 1 October. Add a second file with one paid order on 2 October and run again. raw_orders now has 4 rows, not 7: the streaming table read only the new file, because its checkpoint remembered the old one. daily_revenue was recomputed and now has a row for each day.

We ran exactly this with PySpark 4.2.0. One trap is worth knowing. Without the --conf setting, Spark keeps its table catalog in memory, so the second run no longer knows the tables exist, finds their folders already on disk and fails with LOCATION_ALREADY_EXISTS. The Hive setting keeps the catalog in a small local database between runs. On a real cluster you would point the spec's catalog at your shared catalog instead.

Two more commands are worth knowing:

• spark-pipelines dry-run builds and checks the whole graph without reading or writing any data. It catches syntax errors, a table or column that does not exist, and cycles. It is a good check to run before every change goes in.
• spark-pipelines run --full-refresh raw_orders clears that table and its checkpoint and recomputes it from scratch; --full-refresh-all does it for every table. Plain run is incremental.

Rules that come with letting Spark run the plan

Because the pipeline reads your definitions to build the graph, it runs your Python more than once: while planning and again during a run. So a function that defines a table should do one thing, build and return a DataFrame. The Spark guide lists operations that must never appear inside one: collect(), count(), pivot(), toPandas(), save(), saveAsTable(), start() and toTable(). Writing is the pipeline's job, not the function's. In SQL, the PIVOT clause is not supported.

You can still generate tables in a loop, one per region for example, as long as the list you loop over only grows. A table that disappears from the list disappears from the graph.

To send data out of the pipeline, to Kafka or another system, SDP has sinks (dp.create_sink) and append flows that write into them.

What Databricks adds: Lakeflow pipelines

Lakeflow pipelines run the same code and add features the open-source version does not have. Databricks' own comparison lists these as Lakeflow only:

• Expectations, data quality rules on a table. A row that breaks one can be kept and counted, dropped, or made to stop the update:

  CREATE OR REFRESH STREAMING TABLE orders_clean (
    CONSTRAINT valid_amount EXPECT (amount >= 0) ON VIOLATION DROP ROW
  ) AS SELECT * FROM STREAM(raw_orders);
  

  In Python the same rule is @dp.expect_or_drop("valid_amount", "amount >= 0") above the table. The counts appear in the pipeline's event log, which is how Data QualityData QualityHow far data can be trusted: complete, correct, on time and in the expected shape. Pipelines check it with rules and stop or flag rows that fail. checks become something you can watch over time.

• AUTO CDC, which applies a feed of inserts, updates and deletes to a table in the right order, even when events arrive late. It can keep only the current row (SCD type 1) or the full history (SCD type 2). This is Change Data CaptureChange Data CaptureCopying only the rows that were added, changed or deleted in a source database, read from its change log, instead of copying the whole table again. without hand-written MERGE logic:

  CREATE OR REFRESH STREAMING TABLE customers_history;
  
  CREATE FLOW customers_cdc AS AUTO CDC INTO customers_history
  FROM STREAM(customers_changes)
  KEYS (customer_id)
  APPLY AS DELETE WHEN operation = 'DELETE'
  SEQUENCE BY sequence_num
  COLUMNS * EXCEPT (operation, sequence_num)
  STORED AS SCD TYPE 2;
  

  AUTO CDC needs serverless pipelines or the Pro or Advanced edition.

• The event log, continuous mode, update flows and foreach_batch sinks, plus the managed side: serverless compute, scheduling, Unity Catalog permissions and the pipeline graph in the Databricks interface.

So code written with only streaming tables, materialized views, temporary views, append flows and sinks runs in both places. Once you use expectations or AUTO CDC, you are writing a Lakeflow pipeline.

When a declarative pipeline is the right choice

It fits well when:

• the work is a chain of tables, each defined by a query on the ones before, which describes most Data PipelineData PipelineA set of steps that moves data from where it is made to where it is used, cleaning and reshaping it on the way, run on a schedule or as data arrives. work from ingestion to reporting tables;
• some sources stream or arrive as new files, and you want each run to be incremental without managing checkpoints by hand;
• you want the dependency order to come from the code rather than from a separately maintained job schedule.

It fits less well when the job is mostly side effects (calling an API per row, moving files around), when you need fine control over each write, or when one step must run only if another reports a particular result. Ordinary Spark jobs, orchestrated by Lakeflow Jobs or Airflow, still do that better, and a pipeline can be one task among them.

Tecyfy Takeaway

Spark Declarative Pipelines turn a pipeline into a set of table definitions and hand the order, the incremental reads and the retries to Spark. Delta Live Tables was the Databricks version of the same idea and is now Lakeflow pipelines, built on the open-source SDP that arrived in Spark 4.1. Start with spark-pipelines init and a dry run on your laptop; move to Lakeflow when you want expectations, AUTO CDC and managed compute.