Skip to content

GRA · AI and Financial Model Risk

Spatial financeWhere capital meets hazard, exposure and transition

Human in the loop controls

Human oversight is meant to catch model errors, handle exceptions and keep accountability with people. In practice reviewers can defer to models, lack time or information, or be removed as automation grows. This page sets out how human-in-the-loop controls work and fail in financial services, how their effectiveness is measured, and who decides where human judgement must stay.

01Bhalisa

Human in the loop controls on the record.

02The question

Human in the loop controls: how the risk develops.

Authority and approvals. The dossier below sets out how the risk develops, what is measured, where the data come from, how it is modelled and who decides.

Indlela yokusebenza

Automation bias

Reviewers presented with a model recommendation tend to accept it, especially under time pressure or when the model is usually right. Oversight becomes approval in name only.

Indlela yokusebenza

Reviewers lack context

Without explanations, confidence scores and access to underlying data, reviewers cannot judge when to override, so they rarely do.

Indlela yokusebenza

Accountability must stay with people

Regulations and governance assign responsibility to named individuals and boards. Delegating decisions to models does not delegate accountability.

Indlela yokusebenza

Overrides teach the system

Recorded overrides, and their outcomes, show where models fail. Unrecorded overrides lose that signal and hide inconsistent human judgement.

03Exposure, loss and capital

What human in the loop controls means for exposure, loss and capital.

The exposure, loss, scenario and capital views a risk officer, supervisor or ministry of finance would ask for, with the frameworks they report against. Structures follow supervisory and actuarial practice. Values are illustrative: never prices, ratings or advice.

Transmission channels

How a shock in human in the loop controls reaches credit, market, underwriting, liquidity and operational risk, and feeds back.

Channel structure after the Basel Committee’s work on climate-related risk drivers and transmission channels; links shown are typical paths, not estimated exposures.
NGFS climate scenarios · Network for Greening the Financial SystemIFRS S2 Climate-related Disclosures · ISSBTCFD recommendations · Financial Stability Board TCFDPrinciples for the effective management of climate-related financial risks · Basel Committee on Banking Supervision
04Izalathisi

What is measured, and how well.

Each indicator carries its unit, its cadence, the kind of source behind it and the evidence label a value would carry. No values are shown: a programme supplies them under its own authority.

IndicatorUnitCadenceSource typeIleyibhile
Override rateshare of model recommendations overriddenmonthlyDecision recordsKUQWALASELWE
Outcome of overridesperformance of overridden against non-overridden decisionsquarterlyDecision and outcome recordsKUQIKELELWE
Review time per caseminutesmonthlyWorkflow dataKUQWALASELWE
Decisions fully automatedshare of decisionsquarterlyProcess recordsDECLARED
Named accountable owner per model useshare of usesannualGovernance recordsDECLARED
Customer appeals of automated decisionscount and share upheldquarterlyComplaints dataKUXELIWE
05Imithombo

Where the evidence comes from.

Named public datasets and institutional records, each with its licence class. Sources by agreement are used only under their terms and never republished.

UmthomboWhat it givesLicence class
EU AI Act human oversight requirements for high-risk systemsLegal expectations for oversight designFree, EU reuse terms
GDPR provisions on automated individual decision-makingRights concerning solely automated decisionsFree, EU reuse terms
UK Senior Managers and Certification Regime documentsIndividual accountability in UK financial firmsFree, FCA and PRA terms
Research on automation bias in decision supportEvidence on human reliance on automated adviceFree to read, terms per publisher
IAIS Application Paper on the supervision of AIAccountability expectations for insurersFree, IAIS terms
06Decisions

Who decides, and what the evidence serves.

AI and Financial Model Risk prepares evidence. The decisions below are taken by the authorities named, under their own rules, and never by the platform.

Who decidesThe questionEvidence this page serves
Boards and senior managersWhich decisions must keep human judgement and who is accountableUse risk, legal duties and override evidence
Operations leadersHow reviewers are resourced and informedWorkflow, time and override data
RegulatorsWhether oversight arrangements meet legal requirementsProcess design and outcome evidence
Model risk functionsHow override data feed model reviewOverride and outcome analysis

Outputs this topic produces

  • Guide to designing meaningful human review
  • Method note on analysing overrides and their outcomes
  • Brief on accountability when decisions are automated
  • Questions for boards on where judgement must remain human
07Core questions

What experts ask.

The questions ask whether human oversight is real or nominal.

  • At which points in the process does a person review, and with what authority?
  • Do reviewers have the information and time to disagree with the model?
  • How often do reviewers override the model, and are overrides analysed?
  • Who is accountable for a decision: the model owner, the reviewer or the institution?
  • Can the process be stopped quickly if the model behaves unexpectedly?
08Mechanisms

Mechanisms and drivers.

Automation bias is the tendency to accept automated recommendations without sufficient scrutiny, especially under time pressure or when the system is usually right. It can turn human review into a formality. Conversely, inconsistent overrides can introduce bias. Effective oversight requires clear decision rights, meaningful information, training, workload that permits judgement and analysis of override patterns.

Oversight must be designed to the risk: fully automated processing may be acceptable for low-consequence decisions, while decisions with significant effects on people, or large financial exposures, call for stronger human involvement. Some legal frameworks give individuals rights in relation to solely automated decisions.

09Methods and data

How it is studied, and where the methods stop.

Method or dataWhat it showsImida
Process and decision-rights mappingWhere humans sit and what they can doFormal design may differ from practice
Override analysisHow and why humans disagree with modelsRequires good logging
Human factors researchAutomation bias and workload effectsLaboratory findings may not transfer
Incident reviewFailures of oversightIncidents are under-reported
10Iindlela zokusebenza

How it is studied, and what the methods cannot say.

  1. AI evaluation and assurance engineeringValidation of machine learning models · Red-teaming generative AI · Fairness assessmentApplied here
  2. Artificial intelligence and machine learningGradient-boosted credit models · Language models in operations · Anomaly detectionApplied here
  3. Probability and statisticsCalibration testing · Drift statistics · Group outcome comparisonApplied here
12Imibuzo

Imibuzo neempendulo.

What is automation bias?
The tendency of people to over-rely on automated recommendations, accepting them without sufficient scrutiny even when they are wrong.
Does a human reviewer make a decision lawful?
Not by itself. Review must be meaningful, with time, information and authority to change the outcome, and legal duties still apply to the firm.
Why analyse overrides?
Because their frequency and outcomes show where models are weak and whether human judgement improves decisions or adds inconsistency.
Is human review always necessary?
Not for every decision. The appropriate level of oversight depends on consequences, model risk and the law.
Does the platform set oversight standards?
No. Standards are set by legislators, regulators and institutions. GRA publishes sector-level research.
IingxeloNikela

Faka ubungqina obuvela kweli phepha.

Reports is the Consortium’s quarterly publication. What you know about this subject can become evidence, a case or a correction in the next edition, credited to you and published under CC BY 4.0 where it is your own work.

Global Risks AllianceAI and Financial Model RiskMethods and TechnicalExponential Technology
I-AI noMngcipheko weModeli yezeMali

Bring an exposure or capital question to the Alliance.

Information and enquiries are open today. An enquiry receives a reference and names the next step; applications, payments, compute allocations and registrations open only when their team, terms and review capacity exist.