What is a data pipeline?
A set of steps that moves data from where it is made to where it is used, cleaning and reshaping it on the way, run on a schedule or as data arrives.
beginner
A sales report, a fraud alert and an assistant that answers from company documents all begin the same way: with data that was made somewhere else. An order is written to a shop's database. A policy is saved as a PDF. Neither is in the place, or the shape, where a report or a model can use it. A data pipeline is the set of steps that moves it there.
The 3 jobs every pipeline does
Every pipeline, however large, does 3 things. It reads data from a source: a database, files landing in cloud storage, another company's API. It changes the data: removes duplicates, fixes types, joins the order to its customer, adds up a day's sales. And it writes the result where it is used: a table behind a dashboard, a feature for a model, an index an AI search reads.
Which of the last 2 comes first is a choice with a name. Changing data before you store it is ETL; storing it raw and changing it afterwards is ELTELTLoading raw data into the warehouse or lakehouse first and transforming it there, the reverse of the older ETL order, which transformed data before loading it., the usual order today.
On a schedule or as it arrives
A batch pipeline runs at set times and handles whatever arrived since the last run. A streaming pipeline handles data as it arrives, record by record or in small groups. Many teams use both, and the tools have caught up: Lakeflow pipelines on Databricks, built on Apache Spark Declarative Pipelines, can be either.
Good pipelines are incremental. They do not reread everything each time. Auto Loader, for example, picks up each new file in cloud storage once. For a database, the same idea is change data captureChange Data CaptureCopying only the rows that were added, changed or deleted in a source database, read from its change log, instead of copying the whole table again.: read only the rows that changed.
Orders arrived
0
On the dashboard
0
Arrived, waitingOn the dashboardA batch run
Press Play to run one day of orders through the pipeline.
The orders and times are made up. Nightly and hourly are batch runs; As data arrives is streaming, which on Databricks the same Lakeflow pipeline can also do.
The hard part is what goes wrong
Copying data is easy. The work is in the failures. A source adds or renames a column (schema driftSchema DriftWhen the shape of incoming data changes without warning, such as a renamed or new column, and breaks or silently corrupts the pipelines that read it.). A run stops halfway. A day's data arrives late. A row says an order cost minus 40.
So a real pipeline carries more than its 3 jobs. It checks the data as it passes (data qualityData QualityHow far data can be trusted: complete, correct, on time and in the expected shape. Pipelines check it with rules and stop or flag rows that fail. rules that keep, drop or stop on a bad row). It runs its steps in order, retries what failed and tells someone when it breaks, which is OrchestrationOrchestrationRunning the steps of data pipelines in the right order and at the right time, retrying failures and alerting someone when a step breaks.. And it is usually built in layers of rising trust, raw then cleaned then ready to use, the pattern called the medallion architectureMedallion ArchitectureOrganising a lakehouse in 3 layers of rising quality: bronze for raw data, silver for cleaned and joined data, and gold for tables shaped for reports and AI..
Why AI raised the stakes
A wrong number on a dashboard often gets noticed, because a person who knows the business reads it. An assistant reading the same broken table states the wrong number fluently, with nothing to show it came from a failed load.
AI also adds pipelines of its own. An assistant that answers from documents needs its search index rebuilt as documents change, and on Databricks an AI Search index stays in step with its source table only because a pipeline keeps feeding it changes. That is an embedding pipelineEmbedding PipelineThe pipeline that turns source documents into searchable vectors: it reads new and changed content, chunks it, embeds each chunk and keeps the vector index in step with the source., and it fails in the same ways as any other.
When an AI answer is wrong, look at the pipeline before the model. The model can only be as current and as clean as the data it was handed.
Sources