You're using Delta Live Tables (DLT) with expectations to enforce data quality. An expectation fails. What is the default behavior of the pipeline, and how can you configure it to behave differently?

When a DLT expectation fails, the default behavior depends on the expectation's constraint and the pipeline's quality control settings.

Expectation Constraints:

• constraint (Default):
  If an expectation with a constraint fails, the entire pipeline fails. No further processing occurs for that update. This is the strictest level of enforcement, ensuring that only data meeting the expectations enters the target tables.

• expect:
  If an expectation with expect fails, the pipeline continues to run, but the records that violate the expectation are recorded in a separate table (the expectation results table). This allows you to monitor data quality issues without halting the pipeline.

Pipeline Quality Control Settings:

The quality setting within the DLT pipeline configuration controls how expectation failures are handled. There are three primary modes:

• fail_on_drop (Default for constraint):
  When an expectation with a constraint fails, the pipeline fails.

• drop_invalid_records (For expect):
  When an expectation with expect fails, the invalid records are dropped, but the pipeline continues. These dropped records are recorded in the expectation results table.

• warn (For expect):
  When an expectation with expect fails, a warning is logged, and the pipeline continues processing all records, including the ones that failed the expectation. This is useful for monitoring data quality without impacting pipeline execution.

How to Configure Different Behavior:

You configure the behavior of expectations using the expect or constraint keyword and the pipeline's quality control settings.

Examples:

1. Fail the pipeline on any violations (using constraint):

   @dlt.expect_or_fail("valid_id", "id > 0")  # Defaults to constraint
   @dlt.table
   def my_table():
       # ... your data transformation logic
   

   This will fail the entire pipeline if any id is not greater than 0.

2. Drop invalid records and continue (using expect and default drop_invalid_records quality):

   @dlt.expect("valid_value", "length(value) > 5")
   @dlt.table
   def my_table():
       # ... your data transformation logic
   

   Records where length(value) is not greater than 5 will be dropped, and the pipeline will continue.

3. Warn on violations and continue (using expect and warn quality):

   @dlt.expect("valid_date", "date is not null")
   @dlt.table(quality="warn")
   def my_table():
       # ... your data transformation logic
   

   The pipeline will continue even if some date values are null, but warnings will be logged.

Key Takeaways:

• Use constraint for strict data quality enforcement where any violation should halt the pipeline.
• Use expect for monitoring data quality and allowing the pipeline to continue even with violations.
• Configure the pipeline's quality setting (or use expect_or_fail) to control the behavior when expect expectations fail.

Source: Delta Live Tables - Expectations Documentation