Model drift: what it is and how to monitor AI models in production
Model drift is the degradation of a model's performance over time as the world changes. Data drift is a shift in the input distribution; concept drift is a change in the relationship between inputs and the target. Monitoring means tracking input distributions, prediction distributions, and — when labels arrive — accuracy, with alerts and retraining triggers. It is the 'ongoing monitoring' that model-risk guidance such as SR 11-7 expects.
What model drift is
A model is trained on a snapshot of the world. When the world moves — customer behavior shifts, fraud tactics evolve, an economy changes — the model's assumptions age and its accuracy quietly erodes. That erosion is model drift, and it often goes unnoticed without monitoring because the model keeps producing confident outputs.
In regulated settings, undetected drift is both a performance problem and a compliance problem.
Data drift vs concept drift
Data drift is a change in the input distribution — for example, a new mix of customers or products the model didn't see much in training. The input-to-target relationship may still hold, but the inputs look different.
Concept drift is a change in the relationship itself — what predicts the target changes, as when fraud patterns evolve so that yesterday's signals no longer mean the same thing. Concept drift is harder to catch because it may only show up once true labels arrive.
How to monitor — and act
Track three things: input distributions (to catch data drift early), prediction distributions (to spot behavior shifts), and accuracy against ground truth once labels are available (to catch concept drift). Set thresholds and alerts, and define retraining or rollback triggers in advance.
This closes the loop expected by model-risk frameworks like SR 11-7: ongoing monitoring and outcomes analysis, not just a one-time validation at launch.
Data drift vs concept drift
| Data drift | Concept drift | |
|---|---|---|
| What changes | The input distribution | The input-to-target relationship |
| Example | A new customer or product mix | Fraud patterns evolve |
| Detect with | Distribution tests on inputs and predictions | Accuracy drop once true labels arrive |
| Typical fix | Recalibrate or retrain on recent data | Retrain with newly labeled data; revisit features |
Frequently asked questions
What's the difference between data drift and concept drift?
Data drift is a change in the inputs' distribution; concept drift is a change in the relationship between inputs and the target. Concept drift is usually harder to detect and often needs fresh labels to confirm.
How often should you monitor for drift?
Continuously for input and prediction distributions, and as labels arrive for accuracy. The right cadence depends on how fast your domain changes and the model's risk tier.
How does drift monitoring relate to SR 11-7?
It is exactly the 'ongoing monitoring' and outcomes analysis SR 11-7 expects — evidence that a model keeps working, not just that it worked at launch.
This article is general information, not legal or compliance advice. Verify specifics against the current text of each framework and your own counsel.