# What is data lineage?

The record of where a piece of data came from and everything it passed through, from source tables to the dashboards, models and AI answers built on it.

- Page: https://tecyfy.com/concepts/data-lineage
- Subject: Data engineering in the AI era (https://tecyfy.com/concepts)
- Level: intermediate

An assistant tells a manager that revenue in the north region fell 12% last month. The manager asks the obvious question: where did that number come from? Lineage is what lets someone answer.

Data lineage is the record of where data came from and everything it passed through: which source tables, which jobs, which transformations, down to which column fed which.

## 2 questions it answers

**Where did this come from?** Start at a number on a dashboard and walk backwards: the gold table it reads, the silver tables that built it, the bronze table under those, the source system at the start. When a number is wrong, this is how you find the step that broke it.

**What will this break?** Start at a column and walk forwards: every table, report and model that uses it. Before an app team drops a column, lineage shows who depends on it. That is impact analysis, and it is how you know which teams need a [data contract](https://tecyfy.com/concepts/data-contract) with whom.

(An interactive demo goes here on the page: https://tecyfy.com/concepts/data-lineage)

## Captured, not drawn

Lineage used to be a diagram someone drew in a wiki and nobody updated. Useful lineage is captured automatically from what actually runs.

On Databricks, Unity Catalog records it for every query run on the platform, across tables, notebooks, jobs, pipelines, dashboards and model versions. It goes down to single columns, so you can see that the `total` in a gold table came from `amount` in bronze, and it can be queried from system tables like any other data.

The catch is the word "on". Work run outside the platform is invisible to it unless that tool reports its own lineage. OpenLineage is the open standard for doing that: a shared description of datasets, jobs and runs that schedulers and processing engines can send to a lineage store. It is a graduate project of the LF AI & Data Foundation, and many tools speak it.

## Lineage across the layers

In the [medallion architecture](https://tecyfy.com/concepts/medallion-architecture), lineage is what keeps the layers honest. Every gold table can show the silver and bronze tables it was built from, and every silver table the checks it went through. A gold table whose lineage does not reach a known source should not be trusted.

## Why it matters for AI

An AI answer is the end of a long chain: a source system, a pipeline, a table, perhaps a [semantic layer](https://tecyfy.com/concepts/semantic-layer), a generated query, a model. When the answer is wrong, the model is the easiest thing to blame and often not the cause.

Lineage makes the chain visible. If the assistant's query read a gold table, lineage shows which silver tables built it and when they last updated. Unity Catalog includes model versions in its lineage, so the same trail can show which tables fed a model. Without lineage, "why did the assistant say that?" has no answer beyond guessing. With it, it is a short walk backwards.

## Related

- [Data Quality](https://tecyfy.com/concepts/data-quality): How far data can be trusted: complete, correct, on time and in the expected shape. Pipelines check it with rules and stop or flag rows that fail.
- [Orchestration](https://tecyfy.com/concepts/orchestration): Running the steps of data pipelines in the right order and at the right time, retrying failures and alerting someone when a step breaks.
- [Semantic Layer](https://tecyfy.com/concepts/semantic-layer): One shared definition of business terms and metrics, such as revenue or active customer, that reports, dashboards and AI assistants all read instead of writing their own.

## Sources

- [Data lineage in Unity Catalog](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-lineage)
- [About OpenLineage](https://openlineage.io/docs/)
- [What is the medallion lakehouse architecture?](https://docs.databricks.com/aws/en/lakehouse/medallion)