Research opportunity · Discretion curtailed, relocated or encoded?
Who first detects failures in public-sector algorithmic systems, and how late?
Earlier version (v1)
This is RO-LD05-06 as it was published at v1, kept unchanged. The current version is v2; its changes are listed there.
Question
For documented failures of public-sector algorithmic systems since 2020, who first detected the problem (internal monitoring, frontline staff, auditors, regulators, courts, journalists, advocacy groups, affected people), how long after deployment, and did detection lead to one-case correction or to system-level redesign?
Not yet measuredOne PhD chapter or paperLow costPartly addressed
Why it matters
Formal review and monitoring are offered as the reason algorithmic administration is safe. If failures are mostly found from outside, years after deployment, that reliance is misplaced and oversight design should change. A census of cancelled systems already shows that media, civil society and courts often play a role in ending them, but it does not establish who noticed first, how long problems ran, or what happens with systems that were repaired rather than cancelled.
The debate lists this among what we still do not know: How often do internal monitoring and frontline staff catch system failures before regulators, courts, journalists or advocacy groups do?
Evidence that motivates it
- Robodebt shows that formal review institutions can exist without reliable system-level repair; the Royal Commission recommended stronger escalation, review visibility, debt-review protections and automated-decision governance.
Robodebt shows that formal review institutions can exist without reliable system-level repair; the claim does not say all appeals are ineffective, so the study measures how often detection leads to redesign across cases. - A 2020 judicial-review challenge alleging race discrimination and feedback-loop effects was followed by the Home Office’s withdrawal of its original visa Streaming Tool before a merits ruling; successor routing systems later remained in production.
The Home Office withdrew its original visa Streaming Tool after a judicial-review challenge and before any merits ruling, an example of an external trigger ending a system without a court finding; the study codes such triggers without treating withdrawal as proof of unlawfulness. - Försäkringskassan withdrew an ML risk profile for parental-benefit controls during Swedish data-protection supervision in 2025; the regulator did not make a discrimination finding.
Försäkringskassan withdrew a risk profile during data-protection supervision without a discrimination finding, an example of a regulator-linked detection channel to be coded as distinct from a finding of harm.
A possible design
Build an event dataset of documented failures, suspensions, withdrawals and major redesigns of public-sector algorithmic systems from 2020 onward, starting from existing inventories (the Data Justice Lab census of cancelled systems, AlgorithmWatch reporting, court dockets, regulator decisions, audit office reports, national algorithm registers) and extending to systems that were repaired rather than cancelled. For each event, reconstruct a dated timeline from primary documents: deployment date, first documented sign of the problem, who raised it, when the agency acknowledged it, and the outcome (individual remedies only, parameter change, redesign, withdrawal). Code 'first detector' only where documents support it and mark the rest unknown. Analyse time from deployment to detection and to response by detector type, sector and country. Where possible, ask agencies by FOI whether internal monitoring had flagged the issue earlier.
Data needed
Public documents (court records, regulator decisions, audit and inspection reports, parliamentary records, news archives); FOI requests; existing inventories of cancelled systems.
Feasibility
One PhD chapter or paper. Low: desk research and FOI; costs are mainly researcher time. The design over-represents failures that became public, so it can describe how known failures surfaced but cannot estimate how many failures are caught quietly inside agencies; FOI questions on internal monitoring partly address this. Dating the 'first sign' will often be uncertain and should be reported with that uncertainty.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026; not yet checked in depth).
Redden et al. coded the factors behind 61 paused or cancelled systems (2020 fieldwork), including civil society critique, media investigation, legal action and internal effectiveness decisions, as non-exclusive factors. Gules-Guctas analysed 71 US court dockets on benefits algorithms. A preprint on general AI incidents from 2020 onward separates internal from external detection, but not by detector type or for the public sector. None establishes who detected a problem first, how long it ran before detection, or whether detection led to system-level redesign in systems that stayed in use, and no comparable public-sector dataset was found for 2020 onward.
How strong this evidence is
- Who searched and checked: an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: 2 of the 3 nearest works below were judged from an abstract, summary or search listing only; each is marked. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown; general AI incident databases were not examined directly for public-sector coding (one preprint analysing such a database was found and its abstract read), the Information, Communication & Society article on litigation against automated decision-making could not be opened, and Google Scholar, SSRN and paywalled journals were not systematically searched. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Redden, Brand, Sander & Warne, "Automating Public Services - Learning from Cancelled Systems," Data Justice Lab with Carnegie UK (Sept 2022)
Identifies 61 paused or cancelled systems across Australia, Canada, Europe, New Zealand, the UK and the US and codes cancellation factors (agency effectiveness decisions 31, civil society critique 26, media investigation 24, legal action 19, among others), plus 12 case studies. Covers cancelled systems only, codes contributing factors rather than first detection or time to detection, and fieldwork ended in 2020. - Gules-Guctas, "How Do Algorithmic Decision-Making Systems Used in Public Benefits Determinations Fail? Insights From Legal Challenges," Public Administration Review 86(2), 371–382 (2026; published online 25 Sept 2025), doi:10.1111/puar.70043 Judged from an abstract, summary or search listing only
Analyses 71 US federal and state court dockets on disability, unemployment and nutrition benefit algorithms and argues errors are systemic. Only the abstract was read (via Crossref); court cases are one detection channel, and the study does not compare channels. - Mumtaz, Noor & Mumtaz, "Post-Deployment Accountability in AI Governance - A Cross-Regulatory Empirical Analysis of AI Incidents," arXiv:2605.16281 (v1 Apr 2026; v3 Aug 2026) Judged from an abstract, summary or search listing only
Checks AI Incident Database incidents from 2020 to 2026 against post-deployment provisions of the EU AI Act, the NIST framework and the GDPR, and distinguishes incidents detected through internal monitoring from externally detected ones. Only the abstract was read; it is not specific to the public sector, does not break external detection down by detector type, and reports no time to detection or redesign outcomes.
Search queries used
who discovers public sector algorithm failures journalists courts regulators internal audit database of algorithmic incidents government analysis discovery sourcedataset public sector automated decision-making controversies Europe coded how problems came to light suspension termination of government algorithms"automated decision" government systems cancelled paused study Data Justice Lab "Automating Public Services" cancellation reasonswho detected failures government algorithms dataset discovery source journalists courts regulators internal "time to detection" public sector automated decision-making incidents study
Provenance
RO-LD05-06 v1. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-05 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.