What is data lineage?

An assistant tells a manager that revenue in the north region fell 12% last month. The manager asks the obvious question: where did that number come from? Lineage is what lets someone answer.

Data lineage is the record of where data came from and everything it passed through: which source tables, which jobs, which transformations, down to which column fed which.

2 questions it answers

Where did this come from? Start at a number on a dashboard and walk backwards: the gold table it reads, the silver tables that built it, the bronze table under those, the source system at the start. When a number is wrong, this is how you find the step that broke it.

What will this break? Start at a column and walk forwards: every table, report and model that uses it. Before an app team drops a column, lineage shows who depends on it. That is impact analysis, and it is how you know which teams need a data contractData ContractAn agreement between the team that produces data and the teams that use it, written down and checked, that fixes its columns, meaning and freshness. with whom.

Try itWalk the lineage of an answer

Direction

Where it came fromWhat it feeds

Sourceshop ordersSourceCRM customersBronzeorders_rawBronzecustomers_rawSilverordersSilvercustomersGoldregion_revenueGoldcustomer_valueReportDashboardAIAssistantModelChurn model

Captured, not drawn

Lineage used to be a diagram someone drew in a wiki and nobody updated. Useful lineage is captured automatically from what actually runs.

On Databricks, Unity Catalog records it for every query run on the platform, across tables, notebooks, jobs, pipelines, dashboards and model versions. It goes down to single columns, so you can see that the total in a gold table came from amount in bronze, and it can be queried from system tables like any other data.

The catch is the word "on". Work run outside the platform is invisible to it unless that tool reports its own lineage. OpenLineage is the open standard for doing that: a shared description of datasets, jobs and runs that schedulers and processing engines can send to a lineage store. It is a graduate project of the LF AI & Data Foundation, and many tools speak it.

Lineage across the layers

In the medallion architectureMedallion ArchitectureOrganising a lakehouse in 3 layers of rising quality: bronze for raw data, silver for cleaned and joined data, and gold for tables shaped for reports and AI., lineage is what keeps the layers honest. Every gold table can show the silver and bronze tables it was built from, and every silver table the checks it went through. A gold table whose lineage does not reach a known source should not be trusted.

Why it matters for AI

An AI answer is the end of a long chain: a source system, a pipeline, a table, perhaps a semantic layerSemantic LayerOne shared definition of business terms and metrics, such as revenue or active customer, that reports, dashboards and AI assistants all read instead of writing their own., a generated query, a model. When the answer is wrong, the model is the easiest thing to blame and often not the cause.

Lineage makes the chain visible. If the assistant's query read a gold table, lineage shows which silver tables built it and when they last updated. Unity Catalog includes model versions in its lineage, so the same trail can show which tables fed a model. Without lineage, "why did the assistant say that?" has no answer beyond guessing. With it, it is a short walk backwards.