Research opportunity · Discretion curtailed, relocated or encoded?
Does telling reviewers a case was picked by a risk model change what they decide?
Question
When public officials review a case selected for extra scrutiny, does knowing that a risk model selected it, or seeing its risk score, change how often they decline the claim, how hard they look, and whether they catch errors planted upstream in the case file, and do these effects differ by the claimant's nationality?
Causal mechanism not isolatedOne PhD chapter or paperLow to medium costPartly addressed
Why it matters
Agencies make opposite design choices about what reviewers see. One UK benefits system deliberately hides the risk rating and the fact that a referral came from a model; elsewhere, for example in some US pretrial courts, the decision-maker is given the risk score. Both are defended as safeguards, but there is no evidence on which review design produces more independent judgement or catches more upstream mistakes. The answer bears directly on whether a final human decision amounts to meaningful oversight.
The debate lists this among what we still do not know: Under which organisational conditions does human review genuinely correct upstream model errors?
Evidence that motivates it
- DWP’s Universal Credit Advances model selects claims for additional fraud review while the final decision remains human; the reviewer is not shown the risk rating or told that the referral was model-generated.
The DWP Universal Credit Advances reviewer is not shown the risk rating or told the referral was model-generated; this is one of the review designs the experiment compares. The claim does not say whether final decisions are biased, and neither will this study for that system. - The presence of a human reviewer does not establish meaningful oversight; effective discretion depends on what the human can see, whether they can depart, how often they do so, and the organisational cost of disagreement.
Effective discretion depends on what the reviewer can see and whether they can depart; varying what the reviewer sees tests that dependence directly. - For AI-mediated administrative processes, effective contestability depends partly on process provenance—knowing whether a system selected the case, influenced routing, or generated part of the evidentiary record.
Says contestability depends partly on knowing whether a system selected the case; the experiment measures one consequence of that knowledge (reviewer behaviour), not the legal question of disclosure.
A possible design
Randomised survey experiment with practising benefits, fraud or eligibility officers (recruited through professional associations or a partner agency), with a smaller replication on a general-population sample. Each participant reviews several realistic case files. Arms: (a) no reason given for selection; (b) told the case was selected by a risk model; (c) shown a numeric risk score; (d) told the case was randomly selected for quality control. Within files, randomise a claimant nationality cue and plant one factual inconsistency (for example a date or income figure that contradicts another document). Outcomes: decision to approve or decline, request for further evidence, time spent, and whether the planted error is found. Pre-register hypotheses. Interpret effects as evidence about review designs in a simulated task, not as estimates for any named agency.
Data needed
A sample of public officials (or a partner agency willing to circulate the survey); realistic synthetic case files built with practitioner input; a general-population panel for replication; ethics approval.
Feasibility
One PhD chapter or paper. Low to medium: survey software is cheap; a general-population replication costs a few thousand pounds or euros; recruiting officials is the main effort. Officials may be hard to recruit in large numbers; a design with repeated within-person vignettes helps statistical power. Results from a simulated task will need field follow-up.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026, checked again on 13 October 2026).
Field experiments have randomised whether decision-makers get algorithmic support at all (child protection screening, pretrial hearings), survey experiments have compared algorithmic with human-expert advice, and a lay-participant experiment shows a displayed risk score shifts government-aid decisions. A Senegal tax-audit field experiment found that inspectors who knew a case was algorithm-selected executed it less often and with less effort, but disclosure was not randomised separately from selection. No study was found that randomises disclosure of model selection, score visibility or random-selection framing for officials reviewing benefit or eligibility cases, or that tests whether these review designs differ in catching errors planted upstream or by claimant nationality.
How strong this evidence is
- Who searched and checked: Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert; an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: 2 of the 7 nearest works below were judged from an abstract, summary or search listing only, and 1 has details that could not be confirmed; each is marked. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown; Google Scholar, SSRN and paywalled journals were not systematically searched, and the human-computer interaction literature on decision aids outside the public sector was only partly covered. Absence of a hit is not proof that the work does not exist. Literature check of 2026-10-13: Wiley Online Library and Taylor & Francis pages returned errors, so the PAR article by Wang et al. and the 2026 International Journal of Public Administration article 'Let the Machine Impose the Fine!' could not be read; the latter's authors and design were not confirmed and it is not listed. Human-computer interaction work on decision aids outside the public sector was not reviewed.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Grimon & Mills, "Better Together? A Field Experiment on Human-Algorithm Interaction in Child Protection," arXiv:2502.08501 (Feb 2025; v3 Feb 2026)
Randomised trial (4,681 cases) giving child protection workers real-time algorithmic support; workers shifted investigations toward higher-risk children without mechanically following predictions. Randomises access to the tool, not what the reviewer is told about how a case was selected. - Imai, Jiang, Greiner, Halen & Shin, "Experimental evaluation of algorithm-assisted human decision-making: application to pretrial public safety assessment," Journal of the Royal Statistical Society Series A 186(2), 167–189 (2023)
Field trial in Dane County, Wisconsin, in which whether the judge was given the Public Safety Assessment (risk scores and a recommendation) was essentially randomised by case number; providing it had little overall impact on judges' decisions. Varies presence of a score, in a court setting, not referral disclosure in administrative fraud review. - Alon-Barkat & Busuioc, "Human–AI Interactions in Public Sector Decision Making: 'Automation Bias' and 'Selective Adherence' to Algorithmic Advice," Journal of Public Administration Research and Theory 33(1), 153–169 (2023)
Three survey experiments comparing algorithmic with human-expert advice, including 1,345 Dutch civil servants; found no automation bias. Closest in design, but tests advice source in a contract-renewal task, not selection disclosure or upstream error detection. - Bachas, Brockmeyer, Ferreira & Sarr, "Algorithms and Bureaucrats: Evidence from Tax Audit Selection in Senegal," working paper (24 Jan 2025)
Opened. Field experiment in Senegal's tax offices (2018-2020) where each audit list was half inspector-selected and half algorithm-selected; inspectors knew how each case was selected, and randomly selected benchmark desk-audit cases were labelled as chosen by the 'new method'. Algorithm-selected full audits were 18 percentage points less likely to be executed and used smaller, more junior teams. This is the closest evidence that knowing a case was model-selected changes reviewer effort, but disclosure is not randomised separately from selection, the setting is corporate tax rather than benefits, and there is no planted-error or claimant-nationality manipulation. - Green & Chen, "Algorithmic Risk Assessments Can Alter Human Decision-Making Processes in High-Stakes Government Contexts," Proceedings of the ACM on Human-Computer Interaction 5 (CSCW2) (2021); arXiv:2012.05370 Judged from an abstract, summary or search listing only
Opened (abstract). Online experiment with 2,140 lay participants in simulated pretrial-detention and government-loan decisions: showing a risk assessment made participants more risk-averse, cutting government aid by 8.3% and raising racial disparity in detention by 1.9%. Relevant to the score-visibility arm, but with lay participants, no selection-for-scrutiny framing and no error-detection outcome. - Alon-Barkat & Busuioc, "Public Administration Meets Artificial Intelligence: Towards a Meaningful Behavioral Research Agenda on Algorithmic Decision-Making in Government," Journal of Behavioral Public Administration 7 (2024)
Opened. Research agenda reviewing behavioural studies of algorithmic advice in government. It describes no study that manipulates disclosure of algorithmic selection to decision-makers or measures error detection directly, which supports the opportunity's gap claim (as of 2024). - Wang, Zhang, Xie & Guo, "Province of Origin, Decision-Making Bias, and Responses to Bureaucratic Versus Algorithmic Decision-Making," Public Administration Review (2025, early access), doi:10.1111/puar.13928 Judged from an abstract, summary or search listing onlySome details not confirmed
Read via a university newsletter summary; the Wiley page could not be opened. Two survey experiments with 4,816 Chinese residents on traffic enforcement; tests how origin cues shape citizens' acceptance of algorithmic versus bureaucratic decisions. Uses an origin cue like the proposed nationality cue, but participants are citizens, not reviewing officials.
Literature check
Stands as first published. No study was found that randomises whether reviewers are told a case was selected by a risk model, shown a score, or told it was randomly selected, or that plants upstream errors to test detection. The closest new work is a Senegal tax-audit field experiment (Bachas, Brockmeyer, Ferreira & Sarr, 2025) in which inspectors knew whether a case was algorithm- or inspector-selected and executed algorithm-selected audits less often and with less effort; selection method there is not randomised as a disclosure and is confounded with case composition. Green & Chen (2021) show a displayed risk score shifts lay decisions in a simulated government-aid task, but with lay participants and no selection framing.
Searched: web (extended); web (standard); arxiv.org; osf.io; ssrn.com; academic.oup.com; onlinelibrary.wiley.com; tandfonline.com; journals.sagepub.com; cambridge.org; journal-bpa.org; thedocs.worldbank.org; Dutch-language web. Checked by Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert.
The screen before this check
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026).
Field experiments have randomised whether decision-makers get algorithmic support at all (child protection screening, pretrial hearings), and survey experiments have compared algorithmic with human-expert advice. No study was found that randomises disclosure of model selection or score visibility in a selection-for-scrutiny setting, or that tests whether review designs differ in catching errors planted upstream.
Search queries used
experiment showing vs hiding risk score to caseworkers fraud referral disclosure that case was algorithmically selected effect on reviewer decisionsfield experiment human review design algorithmic decision support public agency randomized interface caseworkers welfare"blind" review algorithm-flagged cases anchoring effect investigators knowing case was selected by risk model experiment civil servants
Literature check, 13 October 2026:
experiment civil servants told case flagged by algorithm versus random selection effect on scrutiny fraud investigation decisionrisk score anchoring caseworkers vignette experiment welfare eligibility decision algorithm displayed score nationality discrimination [allowed_domains: arxiv.org, osf.io, ssrn.com, papers.ssrn.com]automation bias public sector survey experiment algorithmic flag error detection case file officials [allowed_domains: academic.oup.com, onlinelibrary.wiley.com, tandfonline.com, journals.sagepub.com, cambridge.org]"Better Together? A Field Experiment on Human-Algorithm Interaction in Child Protection" citedexperiment ambtenaren algoritme risicoscore selectie controle effect op beoordeling fraude"Alon-Barkat" "Busuioc" automation bias replication 2025 algorithmic advice civil servants follow-up experimenttax auditors experiment knowing case was selected by predictive model versus random audit affects audit effort findings"Let the Machine Impose the Fine" algorithmic discretion Germany experiment authorsWang 2025 "Province of Origin, Decision-Making Bias, and Responses to Bureaucratic Versus Algorithmic Decision-Making" Public Administration Review
Provenance
RO-LD05-02 v2. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-05 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.
Changes
- v2, 13 October 2026: Literature check added: a second, deeper search for existing work, with its queries, new nearest work and limits. The question and verdict stand. See v1 as published.