Research opportunity · Discretion curtailed, relocated or encoded?
Does telling reviewers a case was picked by a risk model change what they decide?
Earlier version (v1)
This is RO-LD05-02 as it was published at v1, kept unchanged. The current version is v2; its changes are listed there.
Question
When public officials review a case selected for extra scrutiny, does knowing that a risk model selected it, or seeing its risk score, change how often they decline the claim, how hard they look, and whether they catch errors planted upstream in the case file, and do these effects differ by the claimant's nationality?
Causal mechanism not isolatedOne PhD chapter or paperLow to medium costPartly addressed
Why it matters
Agencies make opposite design choices about what reviewers see. One UK benefits system deliberately hides the risk rating and the fact that a referral came from a model; elsewhere, for example in some US pretrial courts, the decision-maker is given the risk score. Both are defended as safeguards, but there is no evidence on which review design produces more independent judgement or catches more upstream mistakes. The answer bears directly on whether a final human decision amounts to meaningful oversight.
The debate lists this among what we still do not know: Under which organisational conditions does human review genuinely correct upstream model errors?
Evidence that motivates it
- DWP’s Universal Credit Advances model selects claims for additional fraud review while the final decision remains human; the reviewer is not shown the risk rating or told that the referral was model-generated.
The DWP Universal Credit Advances reviewer is not shown the risk rating or told the referral was model-generated; this is one of the review designs the experiment compares. The claim does not say whether final decisions are biased, and neither will this study for that system. - The presence of a human reviewer does not establish meaningful oversight; effective discretion depends on what the human can see, whether they can depart, how often they do so, and the organisational cost of disagreement.
Effective discretion depends on what the reviewer can see and whether they can depart; varying what the reviewer sees tests that dependence directly. - For AI-mediated administrative processes, effective contestability depends partly on process provenance—knowing whether a system selected the case, influenced routing, or generated part of the evidentiary record.
Says contestability depends partly on knowing whether a system selected the case; the experiment measures one consequence of that knowledge (reviewer behaviour), not the legal question of disclosure.
A possible design
Randomised survey experiment with practising benefits, fraud or eligibility officers (recruited through professional associations or a partner agency), with a smaller replication on a general-population sample. Each participant reviews several realistic case files. Arms: (a) no reason given for selection; (b) told the case was selected by a risk model; (c) shown a numeric risk score; (d) told the case was randomly selected for quality control. Within files, randomise a claimant nationality cue and plant one factual inconsistency (for example a date or income figure that contradicts another document). Outcomes: decision to approve or decline, request for further evidence, time spent, and whether the planted error is found. Pre-register hypotheses. Interpret effects as evidence about review designs in a simulated task, not as estimates for any named agency.
Data needed
A sample of public officials (or a partner agency willing to circulate the survey); realistic synthetic case files built with practitioner input; a general-population panel for replication; ethics approval.
Feasibility
One PhD chapter or paper. Low to medium: survey software is cheap; a general-population replication costs a few thousand pounds or euros; recruiting officials is the main effort. Officials may be hard to recruit in large numbers; a design with repeated within-person vignettes helps statistical power. Results from a simulated task will need field follow-up.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026; not yet checked in depth).
Field experiments have randomised whether decision-makers get algorithmic support at all (child protection screening, pretrial hearings), and survey experiments have compared algorithmic with human-expert advice. No study was found that randomises disclosure of model selection or score visibility in a selection-for-scrutiny setting, or that tests whether review designs differ in catching errors planted upstream.
How strong this evidence is
- Who searched and checked: an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: none of the 3 nearest works below is recorded as judged from an abstract only. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown; Google Scholar, SSRN and paywalled journals were not systematically searched, and the human-computer interaction literature on decision aids outside the public sector was only partly covered. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Grimon & Mills, "Better Together? A Field Experiment on Human-Algorithm Interaction in Child Protection," arXiv:2502.08501 (Feb 2025; v3 Feb 2026)
Randomised trial (4,681 cases) giving child protection workers real-time algorithmic support; workers shifted investigations toward higher-risk children without mechanically following predictions. Randomises access to the tool, not what the reviewer is told about how a case was selected. - Imai, Jiang, Greiner, Halen & Shin, "Experimental evaluation of algorithm-assisted human decision-making: application to pretrial public safety assessment," Journal of the Royal Statistical Society Series A 186(2), 167–189 (2023)
Field trial in Dane County, Wisconsin, in which whether the judge was given the Public Safety Assessment (risk scores and a recommendation) was essentially randomised by case number; providing it had little overall impact on judges' decisions. Varies presence of a score, in a court setting, not referral disclosure in administrative fraud review. - Alon-Barkat & Busuioc, "Human–AI Interactions in Public Sector Decision Making: 'Automation Bias' and 'Selective Adherence' to Algorithmic Advice," Journal of Public Administration Research and Theory 33(1), 153–169 (2023)
Three survey experiments comparing algorithmic with human-expert advice, including 1,345 Dutch civil servants; found no automation bias. Closest in design, but tests advice source in a contract-renewal task, not selection disclosure or upstream error detection.
Search queries used
experiment showing vs hiding risk score to caseworkers fraud referral disclosure that case was algorithmically selected effect on reviewer decisionsfield experiment human review design algorithmic decision support public agency randomized interface caseworkers welfare"blind" review algorithm-flagged cases anchoring effect investigators knowing case was selected by risk model experiment civil servants
Provenance
RO-LD05-02 v1. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-05 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.