The Global Risks Alliance (GRA)
A model validated at launch can fail later as customers, products, markets and data pipelines change. Monitoring tracks performance, input distributions and outcomes to detect drift and trigger review. Machine learning models that retrain frequently add their own risks. This page sets out how drift arises, how monitoring detects it, the measures used, and who decides when a model must change or stop.
GRA · In development · service level R0
Model monitoring and drift is the ongoing tracking of model performance and inputs after deployment to detect when changes in data or relationships make a model unreliable. Outcomes often arrive late, and frequent retraining adds risk. Model owners, model risk functions, supervisors and data owners hold the decisions.
How satellite radar turns a flood into an exposure update. Synthetic scene; no real event or place.
Changes over time and review thresholds. The dossier below sets out how the risk develops, what is measured, where the data come from, how it is modelled and who decides.
The distribution of inputs shifts as new customer segments arrive, products change or upstream systems alter fields. The model meets data it was not trained on.
The relationship between inputs and outcomes changes, for example when an economic shock alters who defaults. Accuracy falls even if inputs look stable.
Defaults and claims emerge months or years after decisions, so true performance is known late. Proxies and early indicators fill the gap with uncertainty.
Frequent retraining can absorb drift, and it can also learn from feedback loops or corrupted data. Each retrained version needs controls proportionate to its use.
Each term as this page uses it. Where a standard-setting body defines the term, the wording follows that body.
| Term | Definition |
|---|---|
| Data drift | A change in the distribution of a model's inputs compared with the training data. |
| Concept drift | A change in the relationship between inputs and the outcome a model predicts. |
| Monitoring threshold | A pre-set level of change that triggers review of a model. |
Each indicator carries its unit, its cadence, the kind of source behind it and the evidence label a value would carry. No values are shown: a programme supplies them under its own authority.
| Indicator | Unit | Cadence | Source type | Label |
|---|---|---|---|---|
| Population stability index by input | dimensionless | monthly | Monitoring | ESTIMATED |
| Performance against validation baseline | change in discrimination or calibration | monthly or quarterly | Monitoring | ESTIMATED |
| Monitoring alerts breaching thresholds | count | monthly | Monitoring records | OBSERVED |
| Models retrained in period | count | quarterly | Model records | DECLARED |
| Data pipeline incidents affecting models | count | quarterly | Incident records | REPORTED |
| Time from alert to remediation | days | per alert | Governance records | OBSERVED |
The topic as a computational graph: named sources feed the methods, an evidence review gates the outputs, and the decisions stay with the authorities that hold them.
Fig. 02Nodes carry their type, two metric rows and a state; connectors show what feeds what; dashed frames name the owners; the timeline is the cadence. Design of the pipeline, not a record of a run.
Named public datasets and institutional records, each with its licence class. Sources by agreement are used only under their terms and never republished.
| Source | What it gives | Licence class |
|---|---|---|
| PRA SS1/23 principles on model monitoring | Expectations for ongoing performance monitoring | Free, Bank of England terms |
| SR 11-7 on ongoing monitoring and outcomes analysis | United States expectations | Public domain |
| EBA guidelines on loan origination and monitoring | Monitoring of automated credit models | Free, EU reuse terms |
| Academic research on dataset shift | Methods for detecting and adapting to drift | Free to read, terms per publisher |
| Bank Pillar 3 model back-testing disclosures | Back-testing results for capital models | Public records, terms per bank |
AI and Financial Model Risk prepares evidence. The decisions below are taken by the authorities named, under their own rules, and never by the platform.
| Who decides | The question | Evidence this page serves |
|---|---|---|
| Model owners | When to recalibrate, redevelop or retire a model | Monitoring results against thresholds |
| Model risk functions | What monitoring standards apply by tier | Policy and monitoring evidence |
| Supervisors | Whether monitoring is adequate for capital and conduct models | Back-testing and review findings |
| Data owners | How upstream changes are controlled | Pipeline incident records |
Monitoring questions ask how soon a problem will be seen.
Data drift occurs when the distribution of inputs changes, for example a new customer segment or a change in a data source. Concept drift occurs when the relationship between inputs and outcomes changes, for example when an economic shock alters how income predicts repayment. Concept drift is more dangerous and harder to detect, because outcomes may take months or years to observe.
Feedback loops add complexity: a model's decisions change who becomes a customer, which changes future data. Retraining can restore performance but can also absorb errors or bias, so retrained models need governance of their own.
| Method or data | What it shows | Limits |
|---|---|---|
| Performance tracking against outcomes | Accuracy and calibration over time | Outcomes arrive with delay |
| Distribution shift statistics | Changes in input data | Shift does not always mean performance loss |
| Benchmark and challenger models | Relative performance | Challengers share some data limitations |
| Threshold and alerting frameworks | When to escalate | Thresholds involve judgement |
The current service level is R0: information and enquiries. An enquiry receives a reference and names the next step; applications, payments, compute allocations and registrations open only when their team, terms and review capacity exist.