# What is a data pipeline?

A set of steps that moves data from where it is made to where it is used, cleaning and reshaping it on the way, run on a schedule or as data arrives.

- Page: https://tecyfy.com/concepts/data-pipeline
- Subject: Data engineering in the AI era (https://tecyfy.com/concepts)
- Level: beginner

A sales report, a fraud alert and an assistant that answers from company documents all begin the same way: with data that was made somewhere else. An order is written to a shop's database. A policy is saved as a PDF. Neither is in the place, or the shape, where a report or a model can use it. A data pipeline is the set of steps that moves it there.

## The 3 jobs every pipeline does

Every pipeline, however large, does 3 things. It **reads** data from a source: a database, files landing in cloud storage, another company's API. It **changes** the data: removes duplicates, fixes types, joins the order to its customer, adds up a day's sales. And it **writes** the result where it is used: a table behind a dashboard, a feature for a model, an index an AI search reads.

Which of the last 2 comes first is a choice with a name. Changing data before you store it is ETL; storing it raw and changing it afterwards is [ELT](https://tecyfy.com/concepts/elt), the usual order today.

## On a schedule or as it arrives

A **batch** pipeline runs at set times and handles whatever arrived since the last run. A **streaming** pipeline handles data as it arrives, record by record or in small groups. Many teams use both, and the tools have caught up: Lakeflow pipelines on Databricks, built on Apache Spark Declarative Pipelines, can be either.

Good pipelines are incremental. They do not reread everything each time. Auto Loader, for example, picks up each new file in cloud storage once. For a database, the same idea is [change data capture](https://tecyfy.com/concepts/change-data-capture): read only the rows that changed.

(An interactive demo goes here on the page: https://tecyfy.com/concepts/data-pipeline)

## The hard part is what goes wrong

Copying data is easy. The work is in the failures. A source adds or renames a column ([schema drift](https://tecyfy.com/concepts/schema-drift)). A run stops halfway. A day's data arrives late. A row says an order cost minus 40.

So a real pipeline carries more than its 3 jobs. It checks the data as it passes ([data quality](https://tecyfy.com/concepts/data-quality) rules that keep, drop or stop on a bad row). It runs its steps in order, retries what failed and tells someone when it breaks, which is [Orchestration](https://tecyfy.com/concepts/orchestration). And it is usually built in layers of rising trust, raw then cleaned then ready to use, the pattern called the [medallion architecture](https://tecyfy.com/concepts/medallion-architecture).

## Why AI raised the stakes

A wrong number on a dashboard often gets noticed, because a person who knows the business reads it. An assistant reading the same broken table states the wrong number fluently, with nothing to show it came from a failed load.

AI also adds pipelines of its own. An assistant that answers from documents needs its search index rebuilt as documents change, and on Databricks an AI Search index stays in step with its source table only because a pipeline keeps feeding it changes. That is an [embedding pipeline](https://tecyfy.com/concepts/embedding-pipeline), and it fails in the same ways as any other.

When an AI answer is wrong, look at the pipeline before the model. The model can only be as current and as clean as the data it was handed.

## Related

- [Change Data Capture](https://tecyfy.com/concepts/change-data-capture): Copying only the rows that were added, changed or deleted in a source database, read from its change log, instead of copying the whole table again.
- [ELT](https://tecyfy.com/concepts/elt): Loading raw data into the warehouse or lakehouse first and transforming it there, the reverse of the older ETL order, which transformed data before loading it.
- [Orchestration](https://tecyfy.com/concepts/orchestration): Running the steps of data pipelines in the right order and at the right time, retrying failures and alerting someone when a step breaks.
- [Data Quality](https://tecyfy.com/concepts/data-quality): How far data can be trusted: complete, correct, on time and in the expected shape. Pipelines check it with rules and stop or flag rows that fail.

## Sources

- [Data engineering with Databricks](https://docs.databricks.com/aws/en/data-engineering/)
- [What is Auto Loader?](https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/)
- [Manage data quality with pipeline expectations](https://docs.databricks.com/aws/en/ldp/expectations)
- [Lakeflow Jobs](https://docs.databricks.com/aws/en/jobs/)
- [What is the medallion lakehouse architecture?](https://docs.databricks.com/aws/en/lakehouse/medallion)
- [Databricks AI Search](https://docs.databricks.com/aws/en/vector-search/vector-search)