The full answer
Build checks into the pipeline at each stage, as rules that run automatically and decide what happens to bad data, rather than as occasional manual queries. On AWS, AWS Glue Data Quality is the managed way to do this.
1. Decide what "good" means, per table
| Dimension | Example rule |
|---|---|
| Completeness | order_id and customer_id are never null |
| Uniqueness | order_id is unique |
| Validity | amount > 0; status in a known list; dates parse |
| Consistency | every customer_id exists in the customers table |
| Freshness | the newest record is less than 2 hours old |
| Volume | today's row count is within the normal range |
2. Express rules in AWS Glue Data Quality (DQDL)
Rules = [
IsComplete "order_id",
IsUnique "order_id",
ColumnValues "amount" > 0,
ColumnValues "status" in ["NEW", "PAID", "SHIPPED", "CANCELLED"],
RowCount > avg(last(7)) * 0.5
]
- In a Glue ETL job, evaluate the ruleset before writing, and use the row-level results to send failing rows to a quarantine location while good rows continue.
- On Data Catalog tables, schedule evaluations of data at rest; Glue can recommend a starting ruleset from the data.
- Dynamic rules like the last one compare with recent history, and anomaly detection learns normal patterns over time.
- Results go to CloudWatch and EventBridge, so a failure can alert a team or stop the next step.
3. Check at the right stage
- Ingestion: schema and format checks; reject or quarantine unreadable records, keeping the raw data.
- Transformation: business rules and joins to reference data.
- Publication: volume, freshness and reconciliation (totals match the source) before a table is exposed to users.
4. Decide what a failure does
- Stop the pipeline when wrong data would be worse than late data, such as financial reports.
- Quarantine bad rows and continue, when partial data is useful.
- Warn for rules still being tuned.
5. Streaming data Validate in the consumer (a Lambda function or Flink application), route invalid events to a dead-letter queue, and keep processing idempotent so retries do not duplicate data.
6. Make it visible Track the data quality score per table over time, and add data contracts with producing teams so schema and rule changes are agreed before they happen.
Last updated 16 October 2026
Sources