A weekly retraining pipeline on SageMaker pushed a model to production that was worse than the one it replaced, and nobody noticed for 2 weeks. How do you change the process so that cannot happen again?

Separate training from releasing. The pipeline trains, evaluates on a fixed test set, and registers each result as a model version in the SageMaker Model Registry with its metrics, as PendingManualApproval. A condition step rejects any version that does not beat the current approved one, and only a version set to Approved reaches the endpoint, through CI/CD. Rolling back is setting the bad version to Rejected, which redeploys the latest approved one.

1. One Model Group per problem

Create a Model Group such as churn-model. Each pipeline run registers a new version in it, holding:

• the model artifact and inference image,
• the evaluation metrics (AUC, precision at the operating threshold, metrics per important segment),
• lineage back to the training data and code, so you can tell what changed between versions.

2. Evaluate against the champion, not against a fixed number

The weekly pipeline:

1. Train on the new data.
2. Evaluate the new model and the currently approved model on the same held-out set, which neither was trained on.
3. Condition step: if the new model is not better by a margin you set, mark it Rejected and stop.
4. Otherwise register it as PendingManualApproval with both sets of metrics.

A fixed threshold such as "AUC above 0.80" lets a model through that is worse than production but still above the line. Comparing with the champion catches exactly the failure in the question. Check segments too: a model can improve overall while getting much worse for a group that matters.

3. Approval drives deployment

Whether a person approves or the pipeline does it automatically depends on risk. For a credit or fraud model, keep a person in the loop: a reviewer reads the comparison and approves with update_model_package. For a low-risk model, the condition step can approve it.

4. Notice in hours, not weeks

The registry stops bad releases that the test set can catch. It cannot catch drift after release. Track live prediction distributions and, when labels arrive, the live metric against what the evaluation predicted, with an alarm on the gap.

What interviewers want: a registry with versions, metrics and lineage, evaluation against the current champion on a held-out set, approval status as the only path to production, the rollback behaviour of Approved → Rejected, and monitoring after release.