GRA · AI and Financial Model Risk
Spatial financeWhere capital meets hazard, exposure and transition
Model monitoring and drift
A model validated at launch can fail later as customers, products, markets and data pipelines change. Monitoring tracks performance, input distributions and outcomes to detect drift and trigger review. Machine learning models that retrain frequently add their own risks. This page sets out how drift arises, how monitoring detects it, the measures used, and who decides when a model must change or stop.
Model monitoring and drift on the record.
Model monitoring and drift: how the risk develops.
Changes over time and review thresholds. The dossier below sets out how the risk develops, what is measured, where the data come from, how it is modelled and who decides.
Data drift
The distribution of inputs shifts as new customer segments arrive, products change or upstream systems alter fields. The model meets data it was not trained on.
Concept drift
The relationship between inputs and outcomes changes, for example when an economic shock alters who defaults. Accuracy falls even if inputs look stable.
Outcomes arrive late
Defaults and claims emerge months or years after decisions, so true performance is known late. Proxies and early indicators fill the gap with uncertainty.
Retraining adds risk
Frequent retraining can absorb drift, and it can also learn from feedback loops or corrupted data. Each retrained version needs controls proportionate to its use.
What model monitoring and drift means for exposure, loss and capital.
The exposure, loss, scenario and capital views a risk officer, supervisor or ministry of finance would ask for, with the frameworks they report against. Structures follow supervisory and actuarial practice. Values are illustrative: never prices, ratings or advice.
Transmission channels
How a shock in model monitoring and drift reaches credit, market, underwriting, liquidity and operational risk, and feeds back.
What is measured, and how well.
Each indicator carries its unit, its cadence, the kind of source behind it and the evidence label a value would carry. No values are shown: a programme supplies them under its own authority.
| Indicator | Unit | Cadence | Source type | Ileyibhile |
|---|---|---|---|---|
| Population stability index by input | dimensionless | monthly | Ukubeka iliso | KUQIKELELWE |
| Performance against validation baseline | change in discrimination or calibration | monthly or quarterly | Ukubeka iliso | KUQIKELELWE |
| Monitoring alerts breaching thresholds | count | monthly | Monitoring records | KUQWALASELWE |
| Models retrained in period | count | quarterly | Model records | DECLARED |
| Data pipeline incidents affecting models | count | quarterly | Incident records | KUXELIWE |
| Time from alert to remediation | days | per alert | Governance records | KUQWALASELWE |
Where the evidence comes from.
Named public datasets and institutional records, each with its licence class. Sources by agreement are used only under their terms and never republished.
| Umthombo | What it gives | Licence class |
|---|---|---|
| PRA SS1/23 principles on model monitoring | Expectations for ongoing performance monitoring | Free, Bank of England terms |
| SR 11-7 on ongoing monitoring and outcomes analysis | United States expectations | Yeyoluntu |
| EBA guidelines on loan origination and monitoring | Monitoring of automated credit models | Free, EU reuse terms |
| Academic research on dataset shift | Methods for detecting and adapting to drift | Free to read, terms per publisher |
| Bank Pillar 3 model back-testing disclosures | Back-testing results for capital models | Public records, terms per bank |
Who decides, and what the evidence serves.
AI and Financial Model Risk prepares evidence. The decisions below are taken by the authorities named, under their own rules, and never by the platform.
| Who decides | The question | Evidence this page serves |
|---|---|---|
| Model owners | When to recalibrate, redevelop or retire a model | Monitoring results against thresholds |
| Model risk functions | What monitoring standards apply by tier | Policy and monitoring evidence |
| Supervisors | Whether monitoring is adequate for capital and conduct models | Back-testing and review findings |
| Data owners | How upstream changes are controlled | Pipeline incident records |
Outputs this topic produces
- Guide to data drift and concept drift
- Method note on monitoring thresholds and alerts
- Brief on controls for frequently retrained models
- Questions for model owners on late-arriving outcomes
What experts ask.
Monitoring questions ask how soon a problem will be seen.
- Which performance measures are monitored, and how often?
- How quickly are outcomes, such as defaults or claims, observed, and what fills the gap meanwhile?
- Has the input data distribution shifted, and does it matter?
- What thresholds trigger review, recalibration or withdrawal, and who set them?
- How are continuously retrained models governed?
Mechanisms and drivers.
Data drift occurs when the distribution of inputs changes, for example a new customer segment or a change in a data source. Concept drift occurs when the relationship between inputs and outcomes changes, for example when an economic shock alters how income predicts repayment. Concept drift is more dangerous and harder to detect, because outcomes may take months or years to observe.
Feedback loops add complexity: a model’s decisions change who becomes a customer, which changes future data. Retraining can restore performance but can also absorb errors or bias, so retrained models need governance of their own.
How it is studied, and where the methods stop.
| Method or data | What it shows | Imida |
|---|---|---|
| Performance tracking against outcomes | Accuracy and calibration over time | Outcomes arrive with delay |
| Distribution shift statistics | Changes in input data | Shift does not always mean performance loss |
| Benchmark and challenger models | Relative performance | Challengers share some data limitations |
| Threshold and alerting frameworks | When to escalate | Thresholds involve judgement |
How it is studied, and what the methods cannot say.
- 01Probability and statisticsCalibration testing · Drift statistics · Group outcome comparisonApplied here
- 02Uncertainty quantification and model validationPrediction intervals for loss models · Model ensembles · Model risk capital or buffersApplied here
- 03Artificial intelligence and machine learningGradient-boosted credit models · Language models in operations · Anomaly detectionApplied here
- 04AI evaluation and assurance engineeringValidation of machine learning models · Red-teaming generative AI · Fairness assessmentApplied here
Imibuzo neempendulo.
What is the difference between data drift and concept drift?
What is a population stability index?
Should models retrain automatically?
Why is concept drift hard to detect?
Is automatic retraining a solution?
Does the platform monitor firms’ models?
Faka ubungqina obuvela kweli phepha.
Reports is the Consortium’s quarterly publication. What you know about this subject can become evidence, a case or a correction in the next edition, credited to you and published under CC BY 4.0 where it is your own work.
- EmailGateway GW-GRASend it from this pageEvidence, a correction, a case, a method, a review, a translation or profile input. It arrives tagged with this page, so the editor who owns it receives it.
- ZenodoOpen depositDeposit it and submit to the communityPublish your own record on Zenodo with your ORCID iD, then submit it to the Reports community there. Curators review it; the DOI stays yours.
- ReportsGlobal Risks AllianceSee every way to contributeThe contribution types, how each is reviewed, what credit you receive, the licence and what is never accepted.
Musa ukuthumela idatha yobuqu, iinkcukacha ezibuthathaka zeziseko zophuhliso okanye nantoni na ongenalungelo lokwabelana ngayo. Umnikelo uphononongwa phambi kokuba usetyenziswe; ukuwuthumela akunikezi bubhali okanye amalungelo okuhlela, kwaye uxhaso alusoze luthenge isigqibo.
Bring an exposure or capital question to the Alliance.
Information and enquiries are open today. An enquiry receives a reference and names the next step; applications, payments, compute allocations and registrations open only when their team, terms and review capacity exist.
Amaxwebhu asemthethweni.
Iimephu, idatha nemizobo ekule sayithi iboniswa njengemeko nolwazi jikelele. Ayingomacebiso, uqikelelo okanye izilumkiso; imida yolwazi ichazwe kwiZilumkiso neZaziso zeDatha.