The Global Risks Alliance (GRA)
Human oversight is meant to catch model errors, handle exceptions and keep accountability with people. In practice reviewers can defer to models, lack time or information, or be removed as automation grows. This page sets out how human-in-the-loop controls work and fail in financial services, how their effectiveness is measured, and who decides where human judgement must stay.
GRA · In development · service level R0
Human oversight of AI in finance is the set of controls through which people review, override and stay accountable for model-driven decisions. It fails through automation bias, lack of context and removal of review as automation grows. Boards, operations leaders, regulators and model risk functions decide where judgement must stay human.
How satellite radar turns a flood into an exposure update. Synthetic scene; no real event or place.
Authority and approvals. The dossier below sets out how the risk develops, what is measured, where the data come from, how it is modelled and who decides.
Reviewers presented with a model recommendation tend to accept it, especially under time pressure or when the model is usually right. Oversight becomes approval in name only.
Without explanations, confidence scores and access to underlying data, reviewers cannot judge when to override, so they rarely do.
Regulations and governance assign responsibility to named individuals and boards. Delegating decisions to models does not delegate accountability.
Recorded overrides, and their outcomes, show where models fail. Unrecorded overrides lose that signal and hide inconsistent human judgement.
Each term as this page uses it. Where a standard-setting body defines the term, the wording follows that body.
| Term | Definition |
|---|---|
| Human in the loop | A design in which a person reviews or approves model outputs before they take effect. |
| Automation bias | Over-reliance on automated recommendations, accepting them without adequate scrutiny even when they are wrong. |
| Override | A human decision that departs from a model's recommendation, ideally recorded with reasons. |
Each indicator carries its unit, its cadence, the kind of source behind it and the evidence label a value would carry. No values are shown: a programme supplies them under its own authority.
| Indicator | Unit | Cadence | Source type | Label |
|---|---|---|---|---|
| Override rate | share of model recommendations overridden | monthly | Decision records | OBSERVED |
| Outcome of overrides | performance of overridden against non-overridden decisions | quarterly | Decision and outcome records | ESTIMATED |
| Review time per case | minutes | monthly | Workflow data | OBSERVED |
| Decisions fully automated | share of decisions | quarterly | Process records | DECLARED |
| Named accountable owner per model use | share of uses | annual | Governance records | DECLARED |
| Customer appeals of automated decisions | count and share upheld | quarterly | Complaints data | REPORTED |
The topic as a computational graph: named sources feed the methods, an evidence review gates the outputs, and the decisions stay with the authorities that hold them.
Fig. 02Nodes carry their type, two metric rows and a state; connectors show what feeds what; dashed frames name the owners; the timeline is the cadence. Design of the pipeline, not a record of a run.
Named public datasets and institutional records, each with its licence class. Sources by agreement are used only under their terms and never republished.
| Source | What it gives | Licence class |
|---|---|---|
| EU AI Act human oversight requirements for high-risk systems | Legal expectations for oversight design | Free, EU reuse terms |
| GDPR provisions on automated individual decision-making | Rights concerning solely automated decisions | Free, EU reuse terms |
| UK Senior Managers and Certification Regime documents | Individual accountability in UK financial firms | Free, FCA and PRA terms |
| Research on automation bias in decision support | Evidence on human reliance on automated advice | Free to read, terms per publisher |
| IAIS Application Paper on the supervision of AI | Accountability expectations for insurers | Free, IAIS terms |
AI and Financial Model Risk prepares evidence. The decisions below are taken by the authorities named, under their own rules, and never by the platform.
| Who decides | The question | Evidence this page serves |
|---|---|---|
| Boards and senior managers | Which decisions must keep human judgement and who is accountable | Use risk, legal duties and override evidence |
| Operations leaders | How reviewers are resourced and informed | Workflow, time and override data |
| Regulators | Whether oversight arrangements meet legal requirements | Process design and outcome evidence |
| Model risk functions | How override data feed model review | Override and outcome analysis |
The questions ask whether human oversight is real or nominal.
Automation bias is the tendency to accept automated recommendations without sufficient scrutiny, especially under time pressure or when the system is usually right. It can turn human review into a formality. Conversely, inconsistent overrides can introduce bias. Effective oversight requires clear decision rights, meaningful information, training, workload that permits judgement and analysis of override patterns.
Oversight must be designed to the risk: fully automated processing may be acceptable for low-consequence decisions, while decisions with significant effects on people, or large financial exposures, call for stronger human involvement. Some legal frameworks give individuals rights in relation to solely automated decisions.
| Method or data | What it shows | Limits |
|---|---|---|
| Process and decision-rights mapping | Where humans sit and what they can do | Formal design may differ from practice |
| Override analysis | How and why humans disagree with models | Requires good logging |
| Human factors research | Automation bias and workload effects | Laboratory findings may not transfer |
| Incident review | Failures of oversight | Incidents are under-reported |
The current service level is R0: information and enquiries. An enquiry receives a reference and names the next step; applications, payments, compute allocations and registrations open only when their team, terms and review capacity exist.