The full answer
Compare what the model sees and predicts now with what it saw in training, feature by feature, and check the inputs for broken data before blaming the world. A model goes stale when the data changes (drift), when the data breaks (nulls, wrong types, values out of range), or when the relationship between inputs and fraud changes. Azure ML model monitoring checks the first two on a schedule and alerts on thresholds; the third needs labels, which for fraud arrive late.
1. Rule out broken data first
A change in an upstream pipeline is a common cause and the cheapest to fix. Look at the data quality signal for recent production inputs:
- Null value rate: a feature that was always present now missing in 30% of requests.
- Data type error rate: an amount arriving as a string.
- Out-of-bounds rate: a country code the model never saw, or amounts in cents instead of currency units.
If one of these jumped on a specific date, find the change that shipped that day. Fix the feed, not the model.
2. Measure drift
| Signal | Compares | Tells you |
|---|---|---|
| Data drift | Each input feature's distribution, production against training | The customers or transactions have changed |
| Prediction drift | The distribution of scores the model outputs | The model behaves differently, even before labels arrive |
| Feature attribution drift (preview) | Which features drive predictions | The model now relies on different signals |
Drift is measured with distances and tests per feature (Jensen-Shannon distance, Population Stability Index, Kolmogorov-Smirnov and others), each with a threshold. Look at the important features first: drift in a feature the model hardly uses matters little.
3. Measure performance with labels
Confirmed fraud labels arrive days or weeks after the transaction. Join them back to the logged predictions and track recall and precision by week. This is the number that settles the question. Drift is an early warning; labelled performance is the verdict.
4. Decide what to do
- Broken data: fix the pipeline, then check the model recovers.
- Real drift with lower performance: retrain on recent data, validate on the newest labelled period, and roll out with a safe rollout on the endpoint.
- New fraud patterns: retraining alone may not be enough. Fraudsters adapt, so new features may be needed, chosen with the fraud team.
How it should have been caught
- Collect production data from day one. For online endpoints, enable model data collection so inputs and outputs are stored. For batch endpoints or models served elsewhere, you must store them yourself.
- Set up monitoring with training data as the reference, data quality and drift signals on the top features, and thresholds that alert the team.
- Track labelled performance as soon as labels arrive, with an alert on a drop in recall.
- Retrain on a schedule as well as on alerts, compared against the current model before it replaces it.
What interviewers want: data quality before drift, the difference between data drift, prediction drift and performance, delayed labels as the real test, collecting production inputs and outputs, and a retraining plan rather than a single fix.
Last updated 16 October 2026
Sources