GRA · AI and Financial Model Risk
Spatial financeWhere capital meets hazard, exposure and transition
Human in the loop controls
Human oversight is meant to catch model errors, handle exceptions and keep accountability with people. In practice reviewers can defer to models, lack time or information, or be removed as automation grows. This page sets out how human-in-the-loop controls work and fail in financial services, how their effectiveness is measured, and who decides where human judgement must stay.
Human in the loop controls on the record.
Human in the loop controls: how the risk develops.
Authority and approvals. The dossier below sets out how the risk develops, what is measured, where the data come from, how it is modelled and who decides.
Automation bias
Reviewers presented with a model recommendation tend to accept it, especially under time pressure or when the model is usually right. Oversight becomes approval in name only.
Reviewers lack context
Without explanations, confidence scores and access to underlying data, reviewers cannot judge when to override, so they rarely do.
Accountability must stay with people
Regulations and governance assign responsibility to named individuals and boards. Delegating decisions to models does not delegate accountability.
Overrides teach the system
Recorded overrides, and their outcomes, show where models fail. Unrecorded overrides lose that signal and hide inconsistent human judgement.
What human in the loop controls means for exposure, loss and capital.
The exposure, loss, scenario and capital views a risk officer, supervisor or ministry of finance would ask for, with the frameworks they report against. Structures follow supervisory and actuarial practice. Values are illustrative: never prices, ratings or advice.
Transmission channels
How a shock in human in the loop controls reaches credit, market, underwriting, liquidity and operational risk, and feeds back.
What is measured, and how well.
Each indicator carries its unit, its cadence, the kind of source behind it and the evidence label a value would carry. No values are shown: a programme supplies them under its own authority.
| Indicator | Unit | Cadence | Source type | Ileyibhile |
|---|---|---|---|---|
| Override rate | share of model recommendations overridden | monthly | Decision records | KUQWALASELWE |
| Outcome of overrides | performance of overridden against non-overridden decisions | quarterly | Decision and outcome records | KUQIKELELWE |
| Review time per case | minutes | monthly | Workflow data | KUQWALASELWE |
| Decisions fully automated | share of decisions | quarterly | Process records | DECLARED |
| Named accountable owner per model use | share of uses | annual | Governance records | DECLARED |
| Customer appeals of automated decisions | count and share upheld | quarterly | Complaints data | KUXELIWE |
Where the evidence comes from.
Named public datasets and institutional records, each with its licence class. Sources by agreement are used only under their terms and never republished.
| Umthombo | What it gives | Licence class |
|---|---|---|
| EU AI Act human oversight requirements for high-risk systems | Legal expectations for oversight design | Free, EU reuse terms |
| GDPR provisions on automated individual decision-making | Rights concerning solely automated decisions | Free, EU reuse terms |
| UK Senior Managers and Certification Regime documents | Individual accountability in UK financial firms | Free, FCA and PRA terms |
| Research on automation bias in decision support | Evidence on human reliance on automated advice | Free to read, terms per publisher |
| IAIS Application Paper on the supervision of AI | Accountability expectations for insurers | Free, IAIS terms |
Who decides, and what the evidence serves.
AI and Financial Model Risk prepares evidence. The decisions below are taken by the authorities named, under their own rules, and never by the platform.
| Who decides | The question | Evidence this page serves |
|---|---|---|
| Boards and senior managers | Which decisions must keep human judgement and who is accountable | Use risk, legal duties and override evidence |
| Operations leaders | How reviewers are resourced and informed | Workflow, time and override data |
| Regulators | Whether oversight arrangements meet legal requirements | Process design and outcome evidence |
| Model risk functions | How override data feed model review | Override and outcome analysis |
Outputs this topic produces
- Guide to designing meaningful human review
- Method note on analysing overrides and their outcomes
- Brief on accountability when decisions are automated
- Questions for boards on where judgement must remain human
- At which points in the process does a person review, and with what authority?
- Do reviewers have the information and time to disagree with the model?
- How often do reviewers override the model, and are overrides analysed?
- Who is accountable for a decision: the model owner, the reviewer or the institution?
- Can the process be stopped quickly if the model behaves unexpectedly?
Mechanisms and drivers.
Automation bias is the tendency to accept automated recommendations without sufficient scrutiny, especially under time pressure or when the system is usually right. It can turn human review into a formality. Conversely, inconsistent overrides can introduce bias. Effective oversight requires clear decision rights, meaningful information, training, workload that permits judgement and analysis of override patterns.
Oversight must be designed to the risk: fully automated processing may be acceptable for low-consequence decisions, while decisions with significant effects on people, or large financial exposures, call for stronger human involvement. Some legal frameworks give individuals rights in relation to solely automated decisions.
How it is studied, and where the methods stop.
| Method or data | What it shows | Imida |
|---|---|---|
| Process and decision-rights mapping | Where humans sit and what they can do | Formal design may differ from practice |
| Override analysis | How and why humans disagree with models | Requires good logging |
| Human factors research | Automation bias and workload effects | Laboratory findings may not transfer |
| Incident review | Failures of oversight | Incidents are under-reported |
How it is studied, and what the methods cannot say.
- 01AI evaluation and assurance engineeringValidation of machine learning models · Red-teaming generative AI · Fairness assessmentApplied here
- 02Artificial intelligence and machine learningGradient-boosted credit models · Language models in operations · Anomaly detectionApplied here
- 03Probability and statisticsCalibration testing · Drift statistics · Group outcome comparisonApplied here
Apho eli phepha likhokelela khona kuyo yonke i-Consortium.
Imibuzo neempendulo.
What is automation bias?
Does a human reviewer make a decision lawful?
Why analyse overrides?
Is human review always necessary?
Does the platform set oversight standards?
Faka ubungqina obuvela kweli phepha.
Reports is the Consortium’s quarterly publication. What you know about this subject can become evidence, a case or a correction in the next edition, credited to you and published under CC BY 4.0 where it is your own work.
- EmailGateway GW-GRASend it from this pageEvidence, a correction, a case, a method, a review, a translation or profile input. It arrives tagged with this page, so the editor who owns it receives it.
- ZenodoOpen depositDeposit it and submit to the communityPublish your own record on Zenodo with your ORCID iD, then submit it to the Reports community there. Curators review it; the DOI stays yours.
- ReportsGlobal Risks AllianceSee every way to contributeThe contribution types, how each is reviewed, what credit you receive, the licence and what is never accepted.
Musa ukuthumela idatha yobuqu, iinkcukacha ezibuthathaka zeziseko zophuhliso okanye nantoni na ongenalungelo lokwabelana ngayo. Umnikelo uphononongwa phambi kokuba usetyenziswe; ukuwuthumela akunikezi bubhali okanye amalungelo okuhlela, kwaye uxhaso alusoze luthenge isigqibo.
Bring an exposure or capital question to the Alliance.
Information and enquiries are open today. An enquiry receives a reference and names the next step; applications, payments, compute allocations and registrations open only when their team, terms and review capacity exist.
Amaxwebhu asemthethweni.
Iimephu, idatha nemizobo ekule sayithi iboniswa njengemeko nolwazi jikelele. Ayingomacebiso, uqikelelo okanye izilumkiso; imida yolwazi ichazwe kwiZilumkiso neZaziso zeDatha.