Skip to content

GRA · AI and Financial Model Risk

Spatial financeWhere capital meets hazard, exposure and transition

Human in the loop controls

Human oversight is meant to catch model errors, handle exceptions and keep accountability with people. In practice reviewers can defer to models, lack time or information, or be removed as automation grows. This page sets out how human-in-the-loop controls work and fail in financial services, how their effectiveness is measured, and who decides where human judgement must stay.

01رجسٹر

Human in the loop controls on the record.

02The question

Human in the loop controls: how the risk develops.

Authority and approvals. The dossier below sets out how the risk develops, what is measured, where the data come from, how it is modelled and who decides.

طریقۂ کار

Automation bias

Reviewers presented with a model recommendation tend to accept it, especially under time pressure or when the model is usually right. Oversight becomes approval in name only.

طریقۂ کار

Reviewers lack context

Without explanations, confidence scores and access to underlying data, reviewers cannot judge when to override, so they rarely do.

طریقۂ کار

Accountability must stay with people

Regulations and governance assign responsibility to named individuals and boards. Delegating decisions to models does not delegate accountability.

طریقۂ کار

Overrides teach the system

Recorded overrides, and their outcomes, show where models fail. Unrecorded overrides lose that signal and hide inconsistent human judgement.

03Exposure, loss and capital

What human in the loop controls means for exposure, loss and capital.

The exposure, loss, scenario and capital views a risk officer, supervisor or ministry of finance would ask for, with the frameworks they report against. Structures follow supervisory and actuarial practice. Values are illustrative: never prices, ratings or advice.

Transmission channels

How a shock in human in the loop controls reaches credit, market, underwriting, liquidity and operational risk, and feeds back.

Channel structure after the Basel Committee’s work on climate-related risk drivers and transmission channels; links shown are typical paths, not estimated exposures.
NGFS climate scenarios · Network for Greening the Financial SystemIFRS S2 Climate-related Disclosures · ISSBTCFD recommendations · Financial Stability Board TCFDPrinciples for the effective management of climate-related financial risks · Basel Committee on Banking Supervision
04اشاریے

What is measured, and how well.

Each indicator carries its unit, its cadence, the kind of source behind it and the evidence label a value would carry. No values are shown: a programme supplies them under its own authority.

IndicatorUnitCadenceSource typeلیبل
Override rateshare of model recommendations overriddenmonthlyDecision recordsمشاہدہ شدہ
Outcome of overridesperformance of overridden against non-overridden decisionsquarterlyDecision and outcome recordsتخمینی
Review time per caseminutesmonthlyWorkflow dataمشاہدہ شدہ
Decisions fully automatedshare of decisionsquarterlyProcess recordsDECLARED
Named accountable owner per model useshare of usesannualGovernance recordsDECLARED
Customer appeals of automated decisionscount and share upheldquarterlyComplaints dataرپورٹ شدہ
05ماخذات

Where the evidence comes from.

Named public datasets and institutional records, each with its licence class. Sources by agreement are used only under their terms and never republished.

ماخذWhat it givesLicence class
EU AI Act human oversight requirements for high-risk systemsLegal expectations for oversight designFree, EU reuse terms
GDPR provisions on automated individual decision-makingRights concerning solely automated decisionsFree, EU reuse terms
UK Senior Managers and Certification Regime documentsIndividual accountability in UK financial firmsFree, FCA and PRA terms
Research on automation bias in decision supportEvidence on human reliance on automated adviceFree to read, terms per publisher
IAIS Application Paper on the supervision of AIAccountability expectations for insurersFree, IAIS terms
06Decisions

Who decides, and what the evidence serves.

AI and Financial Model Risk prepares evidence. The decisions below are taken by the authorities named, under their own rules, and never by the platform.

کون فیصلہ کرتا ہےThe questionEvidence this page serves
Boards and senior managersWhich decisions must keep human judgement and who is accountableUse risk, legal duties and override evidence
Operations leadersHow reviewers are resourced and informedWorkflow, time and override data
RegulatorsWhether oversight arrangements meet legal requirementsProcess design and outcome evidence
Model risk functionsHow override data feed model reviewOverride and outcome analysis

Outputs this topic produces

  • Guide to designing meaningful human review
  • Method note on analysing overrides and their outcomes
  • Brief on accountability when decisions are automated
  • Questions for boards on where judgement must remain human
07Core questions

What experts ask.

The questions ask whether human oversight is real or nominal.

  • At which points in the process does a person review, and with what authority?
  • Do reviewers have the information and time to disagree with the model?
  • How often do reviewers override the model, and are overrides analysed?
  • Who is accountable for a decision: the model owner, the reviewer or the institution?
  • Can the process be stopped quickly if the model behaves unexpectedly?
08Mechanisms

Mechanisms and drivers.

Automation bias is the tendency to accept automated recommendations without sufficient scrutiny, especially under time pressure or when the system is usually right. It can turn human review into a formality. Conversely, inconsistent overrides can introduce bias. Effective oversight requires clear decision rights, meaningful information, training, workload that permits judgement and analysis of override patterns.

Oversight must be designed to the risk: fully automated processing may be acceptable for low-consequence decisions, while decisions with significant effects on people, or large financial exposures, call for stronger human involvement. Some legal frameworks give individuals rights in relation to solely automated decisions.

09Methods and data

How it is studied, and where the methods stop.

Method or dataWhat it showsحدود
Process and decision-rights mappingWhere humans sit and what they can doFormal design may differ from practice
Override analysisHow and why humans disagree with modelsRequires good logging
Human factors researchAutomation bias and workload effectsLaboratory findings may not transfer
Incident reviewFailures of oversightIncidents are under-reported
10طریقے

How it is studied, and what the methods cannot say.

  1. AI evaluation and assurance engineeringValidation of machine learning models · Red-teaming generative AI · Fairness assessmentApplied here
  2. Artificial intelligence and machine learningGradient-boosted credit models · Language models in operations · Anomaly detectionApplied here
  3. Probability and statisticsCalibration testing · Drift statistics · Group outcome comparisonApplied here
11منسلک

یہ صفحہ پورے کنسورشیم میں کہاں لے جاتا ہے۔

12سوالات

سوالات و جوابات۔

What is automation bias?
The tendency of people to over-rely on automated recommendations, accepting them without sufficient scrutiny even when they are wrong.
Does a human reviewer make a decision lawful?
Not by itself. Review must be meaningful, with time, information and authority to change the outcome, and legal duties still apply to the firm.
Why analyse overrides?
Because their frequency and outcomes show where models are weak and whether human judgement improves decisions or adds inconsistency.
Is human review always necessary?
Not for every decision. The appropriate level of oversight depends on consequences, model risk and the law.
Does the platform set oversight standards?
No. Standards are set by legislators, regulators and institutions. GRA publishes sector-level research.
رپورٹیںحصہ ڈالیں

اس صفحے سے شواہد فراہم کریں۔

Reports کنسورشیم کی سہ ماہی اشاعت ہے۔ اس موضوع کے بارے میں آپ کا علم اگلے شمارے میں شواہد، کیس یا تصحیح بن سکتا ہے، جو آپ کے نام سے منسوب ہوگا اور اگر یہ آپ کا اپنا کام ہو تو CC BY 4.0 کے تحت شائع ہوگا۔

Global Risks AllianceAI and Financial Model RiskMethods and TechnicalExponential Technology

کبھی بھی ذاتی ڈیٹا، حساس بنیادی ڈھانچے کی تفصیل یا کوئی ایسی چیز نہ بھیجیں جسے شیئر کرنے کا آپ کو حق نہ ہو۔ ہر شراکت استعمال سے پہلے جانچی جاتی ہے؛ اسے بھیجنے سے کوئی تصنیفی یا ادارتی حق حاصل نہیں ہوتا، اور سرپرستی کبھی کوئی نتیجہ نہیں خریدتی۔

AI اور مالیاتی ماڈل کا خطرہ

Bring an exposure or capital question to the Alliance.

معلومات اور استفسارات آج کھلے ہیں۔ استفسار کو ایک حوالہ نمبر ملتا ہے اور اگلے مرحلے کی نشاندہی ہوتی ہے؛ درخواستیں، ادائیگیاں، کمپیوٹنگ کی تخصیص اور رجسٹریشن صرف اس وقت کھلتی ہیں جب ان کی ٹیم، شرائط اور جائزے کی استعداد موجود ہو۔