Research opportunity · Discretion curtailed, relocated or encoded?
Who first detects failures in public-sector algorithmic systems, and how late?
Question
For documented failures of public-sector algorithmic systems since 2020, who first detected the problem (internal monitoring, frontline staff, auditors, regulators, courts, journalists, advocacy groups, affected people), how long after deployment, and did detection lead to one-case correction or to system-level redesign?
Not yet measuredOne PhD chapter or paperLow costPartly addressed
Why it matters
Formal review and monitoring are offered as the reason algorithmic administration is safe. If failures are mostly found from outside, years after deployment, that reliance is misplaced and oversight design should change. A census of cancelled systems already shows that media, civil society and courts often play a role in ending them, but it does not establish who noticed first, how long problems ran, or what happens with systems that were repaired rather than cancelled.
The debate lists this among what we still do not know: How often do internal monitoring and frontline staff catch system failures before regulators, courts, journalists or advocacy groups do?
Evidence that motivates it
- Robodebt shows that formal review institutions can exist without reliable system-level repair; the Royal Commission recommended stronger escalation, review visibility, debt-review protections and automated-decision governance.
Robodebt shows that formal review institutions can exist without reliable system-level repair; the claim does not say all appeals are ineffective, so the study measures how often detection leads to redesign across cases. - A 2020 judicial-review challenge alleging race discrimination and feedback-loop effects was followed by the Home Office’s withdrawal of its original visa Streaming Tool before a merits ruling; successor routing systems later remained in production.
The Home Office withdrew its original visa Streaming Tool after a judicial-review challenge and before any merits ruling, an example of an external trigger ending a system without a court finding; the study codes such triggers without treating withdrawal as proof of unlawfulness. - Försäkringskassan withdrew an ML risk profile for parental-benefit controls during Swedish data-protection supervision in 2025; the regulator did not make a discrimination finding.
Försäkringskassan withdrew a risk profile during data-protection supervision without a discrimination finding, an example of a regulator-linked detection channel to be coded as distinct from a finding of harm.
A possible design
Build an event dataset of documented failures, suspensions, withdrawals and major redesigns of public-sector algorithmic systems from 2020 onward, starting from existing inventories (the Data Justice Lab census of cancelled systems, AlgorithmWatch reporting, court dockets, regulator decisions, audit office reports, national algorithm registers) and extending to systems that were repaired rather than cancelled. For each event, reconstruct a dated timeline from primary documents: deployment date, first documented sign of the problem, who raised it, when the agency acknowledged it, and the outcome (individual remedies only, parameter change, redesign, withdrawal). Code 'first detector' only where documents support it and mark the rest unknown. Analyse time from deployment to detection and to response by detector type, sector and country. Where possible, ask agencies by FOI whether internal monitoring had flagged the issue earlier.
Data needed
Public documents (court records, regulator decisions, audit and inspection reports, parliamentary records, news archives); FOI requests; existing inventories of cancelled systems.
Feasibility
One PhD chapter or paper. Low: desk research and FOI; costs are mainly researcher time. The design over-represents failures that became public, so it can describe how known failures surfaced but cannot estimate how many failures are caught quietly inside agencies; FOI questions on internal monitoring partly address this. Dating the 'first sign' will often be uncertain and should be reported with that uncertainty.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026, checked again on 13 October 2026).
Redden et al. coded the factors behind 61 paused or cancelled systems (2020 fieldwork), including civil society critique, media investigation, legal action and internal effectiveness decisions, as non-exclusive factors. Johnson et al. (FAccT 2024) traced 40 abandoned or contested algorithms through phases starting with discovery and typed outcomes as repaired, unchanged or abandoned, but without counts by discoverer, timing or a public-sector split. Gules-Guctas analysed 71 US court dockets on benefits algorithms. A preprint on 480 general AI incidents from 2020 to 2026 finds only 5% were detected through internal monitoring, but does not split external detection by detector type, does not report public-sector results or time to detection, and does not track redesign. None establishes who detected a problem first, how long it ran before detection, or whether detection led to system-level redesign in systems that stayed in use, and no comparable public-sector dataset was found for 2020 onward.
How strong this evidence is
- Who searched and checked: Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert; an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: 4 of the 6 nearest works below were judged from an abstract, summary or search listing only; each is marked. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown; general AI incident databases were not examined directly for public-sector coding (one preprint analysing such a database was found and its abstract read), the Information, Communication & Society article on litigation against automated decision-making could not be opened, and Google Scholar, SSRN and paywalled journals were not systematically searched. Absence of a hit is not proof that the work does not exist. Literature check of 2026-10-13: The last 9,000 characters of Johnson et al. (including any appendix case list) were not read, so the public-sector share of its 40 cases is unknown. Richards, Benn & Zilka was read at abstract level only. Forward citations of Gules-Guctas could not be listed with web search. Dutch and French-language searches and national algorithm-register based studies were not run; the German query found only AlgorithmWatch explainers. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Redden, Brand, Sander & Warne, "Automating Public Services - Learning from Cancelled Systems," Data Justice Lab with Carnegie UK (Sept 2022)
Identifies 61 paused or cancelled systems across Australia, Canada, Europe, New Zealand, the UK and the US and codes cancellation factors (agency effectiveness decisions 31, civil society critique 26, media investigation 24, legal action 19, among others), plus 12 case studies. Covers cancelled systems only, codes contributing factors rather than first detection or time to detection, and fieldwork ended in 2020. - Gules-Guctas, "How Do Algorithmic Decision-Making Systems Used in Public Benefits Determinations Fail? Insights From Legal Challenges," Public Administration Review 86(2), 371–382 (2026; published online 25 Sept 2025), doi:10.1111/puar.70043 Judged from an abstract, summary or search listing only
Analyses 71 US federal and state court dockets on disability, unemployment and nutrition benefit algorithms and argues errors are systemic. Only the abstract was read (via Crossref); court cases are one detection channel, and the study does not compare channels. - Mumtaz, Noor & Mumtaz, "Post-Deployment Accountability in AI Governance - A Cross-Regulatory Empirical Analysis of AI Incidents," arXiv:2605.16281 (v1 Apr 2026; v3 Aug 2026) Judged from an abstract, summary or search listing only
Checks AI Incident Database incidents from 2020 to 2026 against post-deployment provisions of the EU AI Act, the NIST framework and the GDPR, and distinguishes incidents detected through internal monitoring from externally detected ones. Only the abstract was read; it is not specific to the public sector, does not break external detection down by detector type, and reports no time to detection or redesign outcomes. - Johnson, Moharana, Harrington, Andalibi, Heidari & Eslami, "The Fall of an Algorithm: Characterizing the Dynamics Toward Abandonment," Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024), doi:10.1145/3630106.3658910; arXiv:2404.13802
Full text read (first 100,000 characters): thematic analysis of 40 real-world cases (24 abandoned, 16 US-based) from web searches, incident databases and expert consultation; six phases (discovery, diagnosis, dissemination, dialogue, decision, death); outcomes typed as repaired, unchanged, abandoned by choice or abandoned through law or policy. Gives examples of discoverers (a Right to Know request, social media users) but no counts by discoverer type, no systematic time to discovery, no public/private split and no counts of repair versus abandonment. Closest conceptual frame for the design; a dated, coded dataset is what remains. - Richards, Benn & Zilka, "From Incidents to Insights: Patterns of Responsibility following AI Harms," arXiv:2505.04291 (7 May 2025) Judged from an abstract, summary or search listing only
Abstract read: mixed-methods analysis of 962 AI Incident Database incidents and 4,743 reports on how developers, deployers, victims and lawmakers respond after AI harms. All sectors; the abstract mentions no detector coding, time to detection or public-sector breakdown. - Kaun, "Temporalities of welfare automation: On timing, belatedness, and perpetual emergence," Time & Society (2025), doi:10.1177/0961463X251313572 Judged from an abstract, summary or search listing only
Seen in search results only (not opened). Title indicates a conceptual analysis of timing and belatedness in welfare automation, relevant to the time-to-detection question; not known to code a dataset.
Literature check
Stands as first published. No study was found that codes first detector, time from deployment to detection and system-level response for public-sector algorithmic failures since 2020. The closest addition is Johnson et al. (FAccT 2024), a thematic analysis of 40 abandoned or contested algorithms that names discovery as the first of six phases and sorts outcomes into repaired, unchanged or abandoned, but it reports no counts by discoverer type, does not measure time to discovery and does not separate public-sector cases. Richards, Benn & Zilka (2025) code responses after 962 AI Incident Database incidents without detector or timing, and the 480-incident preprint already listed splits only internal versus external detection.
Searched: web (extended and standard); arxiv.org; ssrn.com; osf.io; socarxiv.org; sciencedirect.com; journals.sagepub.com; tandfonline.com; onlinelibrary.wiley.com; academic.oup.com; cambridge.org; German-language web; direct fetches: arxiv.org/abs/2605.16281, arxiv.org/html/2605.16281v3, datajusticelab.org/publications, arxiv.org/abs/2404.13802, arxiv.org/html/2404.13802v2, arxiv.org/abs/2505.04291. Checked by Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert.
The screen before this check
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026).
Redden et al. coded the factors behind 61 paused or cancelled systems (2020 fieldwork), including civil society critique, media investigation, legal action and internal effectiveness decisions, as non-exclusive factors. Gules-Guctas analysed 71 US court dockets on benefits algorithms. A preprint on general AI incidents from 2020 onward separates internal from external detection, but not by detector type or for the public sector. None establishes who detected a problem first, how long it ran before detection, or whether detection led to system-level redesign in systems that stayed in use, and no comparable public-sector dataset was found for 2020 onward.
Search queries used
who discovers public sector algorithm failures journalists courts regulators internal audit database of algorithmic incidents government analysis discovery sourcedataset public sector automated decision-making controversies Europe coded how problems came to light suspension termination of government algorithms"automated decision" government systems cancelled paused study Data Justice Lab "Automating Public Services" cancellation reasonswho detected failures government algorithms dataset discovery source journalists courts regulators internal "time to detection" public sector automated decision-making incidents study
Literature check, 13 October 2026:
"Automating Public Services" "Learning from Cancelled Systems" Redden follow-up dataset 2025 2026public sector algorithm failures dataset how problems were discovered whistleblowers journalists courts suspension government automated decision-making [allowed_domains: arxiv.org, ssrn.com, papers.ssrn.com, osf.io, socarxiv.org]algorithmic scandals public administration comparative analysis detection accountability Robodebt toeslagenaffaire [allowed_domains: sciencedirect.com, journals.sagepub.com, tandfonline.com, onlinelibrary.wiley.com, academic.oup.com, cambridge.org]"The Fall of an Algorithm" abandonment discovery public sector cases who discovered harms journalistsAlgorithmen Verwaltung Fehler aufgedeckt Journalisten Gerichte Datenbank Fälle Studie öffentliche Verwaltung automatisierte Entscheidungen Europa AlgorithmWatchGules-Guctas "How Do Algorithmic Decision-Making Systems Used in Public Benefits Determinations Fail" cited
Provenance
RO-LD05-06 v2. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-05 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.
Changes
- v2, 13 October 2026: Literature check added: a second, deeper search for existing work, with its queries, new nearest work and limits. The question and verdict stand. See v1 as published.