What is data quality, and how do pipelines check it?

Every data team has a story about the report that was wrong for weeks. A source started sending prices in cents instead of euros. A job failed silently and the table stopped updating. A test customer with a billion-euro order made it into the total. Nobody noticed, because the numbers still looked like numbers.

Data quality is how far data can be trusted for the job it is used for. In practice it comes down to a handful of plain questions.

The questions that matter

• Is it complete? Are all of yesterday's orders here, or only some?
• Is it correct? Is the amount positive, the date real, the country a real country?
• Is it on time? Did the table update when it should have?
• Is it the right shape? Are the columns the ones the next step expects? (When they change, that is schema driftSchema DriftWhen the shape of incoming data changes without warning, such as a renamed or new column, and breaks or silently corrupts the pipelines that read it..)
• Is it unique? Is each order here once?

Checking rows: rules in the pipeline

The first line of defence is a rule on each row as it passes through. On Databricks a Lakeflow pipeline writes these as expectations, such as CONSTRAINT valid_total EXPECT (total > 0), with a choice of what happens when a row breaks it:

• keep the row and count the failure, the default,
• drop the row,
• or stop the whole update.

Which to choose is a business decision. Dropping a bad order keeps the report clean but loses the order. Stopping the update protects everything downstream but means nothing updates until someone looks. Every result is written to the pipeline's event log, so you can see how many rows broke which rule.

In the medallion architectureMedallion ArchitectureOrganising a lakehouse in 3 layers of rising quality: bronze for raw data, silver for cleaned and joined data, and gold for tables shaped for reports and AI. these checks mostly live where data moves from bronze to silver.

Checking tables: watching for what rules miss

Row rules cannot see a row that never arrived. So the second line watches whole tables over time. Databricks' data quality monitoring learns each table's usual pattern and flags it as stale when an update is unusually late, and as incomplete when far fewer rows arrived than expected. It needs no rule written per table, which is how it catches the job that quietly stopped.

Why it matters for AI

People reading a dashboard bring doubt with them. A sales manager who sees revenue double overnight asks why. An AI assistant reading the same table does not hesitate. It reports the doubled number in a clear sentence, and a person reading that sentence has fewer clues than the chart gave them.

So quality checks matter more when a model is the reader. Put the checks before the data reaches anything an assistant reads, and agree them with the team that sends the data, in a data contractData ContractAn agreement between the team that produces data and the teams that use it, written down and checked, that fixes its columns, meaning and freshness., so a change at the source is caught before it becomes an answer.