Research opportunity · Organizational absorption: Why micro gains do not become macro outcomes
Do generative-AI deployments in government improve services, not just staff time?
Question
When a government agency deploys generative AI to caseworkers, adjudicators or reviewers, do service outcomes recorded in administrative data (processing time, error and appeal rates, outcomes for citizens) improve against a credible comparison group, beyond staff time savings?
Wrong or missing validation targetFunded team projectNeeds a budget (medium or higher)Partly addressed
Why it matters
Public managers in the EU already use generative AI, often without approval or policy, and in US state agencies what drove adoption of earlier chatbots differed from what made implementation succeed. The UK and Australian government evaluations of assistant trials could not be read here, and the comparison-group evaluations the screen found rely on staff self-reports. Time saved by staff is not the public value an agency exists to deliver, so the measure that would justify or stop wider deployment is missing.
The debate lists this among what we still do not know: Do public-sector deployments improve service outcomes? The UK and Australian government trials could not be read for this version.
Evidence that motivates it
- The authors of a survey of 576 public managers in seven EU member states describe public servants using generative AI to draft reports, refine policy proposals and generate content, often without explicit approval or institutional policies, which they call 'shadow use'; the abstract does not say how much of this the survey itself measured.
Public managers in seven EU member states use generative AI to draft reports and policy proposals, often without explicit approval or institutional policy. - In 22 US state agencies using chatbots, the factors that drove adoption (ease of use, leadership, external shocks such as COVID-19) overlapped only partly with those that shaped implementation (building and maintaining the knowledge base, skills, resources, cross-agency coordination, external shocks and citizens' expectations).
In 22 US state agencies using chatbots, adoption and implementation success had different drivers; outcomes for services were not the measure.
A possible design
Randomized or staggered rollout across offices or caseloads in one agency, randomizing cases within reviewer and case-type strata where possible. Outcomes come from administrative records: time to decision, error rates found in audit samples, appeal and reversal rates, and, where measurable, outcomes for the people served. Pre-register the outcomes and the audit sampling, and report staff time alongside service outcomes so the two can be compared.
Data needed
An agency partner with case-level administrative records and an audit process; legal basis for research use of the records; enough offices or cases for a comparison group.
Feasibility
Funded team project. Medium; the deployment is the agency's own, but audit sampling, data access agreements and an independent evaluation team cost money and time. A registered randomized trial of AI in building-permit review in Chilean municipalities, due to end in December 2026, may report the core design soon, so check it before starting. Appeals arrive months after decisions, so follow-up must be long enough to see them.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 15 October 2026; not yet checked in depth).
Controlled evaluations exist, but they use historical or benchmark cases, constructed tasks or staff self-reports. No evaluation was found of a live generative-AI deployment in a government agency that uses a randomized or quasi-experimental comparison and measures service outcomes from administrative records. A registered randomized trial with processing time as its outcome is under way in Chilean municipalities, with no results yet; the gap may close soon.
How strong this evidence is
- Who searched and checked: an independent AI reviewer with fresh context (not a person), 15 October 2026 (review of PR #77): verdicts, query counts and 13 abstract-read labels checked; label slips corrected. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: 6 of the 6 nearest works below were judged from an abstract, summary or search listing only, and 2 have details that could not be confirmed; each is marked. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search (8 queries) and arXiv abstract pages on the date shown; government sites (gov.uk, Australian government evaluation pages) and the Stanford PDF could not be opened from this environment. No native-language searches were run for Korea, Singapore, Estonia or China, and public-administration journals were not searched directly. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Magesh, Martin, Surani, Perez, Rodolfa & Ho, "Evaluating Generative AI in Benefits Administration: A Demonstration Project," Stanford RegLab and Yale Tobin Center working paper (2026) Judged from an abstract, summary or search listing onlySome details not confirmed
Judged from search listings; not read directly (the PDF could not be opened). Reportedly a randomized trial, with the US Department of Labor and Colorado's labour department, of an AI fact-finding assistant on real historical unemployment-insurance cases, measuring decision quality and adjudicator behaviour. The closest design, but on historical cases rather than a live deployment. - Gosciak, Giannella, Guo, Chen & Koenecke, "LLMs in social services: How does chatbot accuracy affect human accuracy?" arXiv:2603.11213 (2026) Judged from an abstract, summary or search listing only
Abstract read. Randomized experiment with Los Angeles nonprofit caseworkers on a benefits-question benchmark: accurate chatbots raised caseworkers' accuracy and wrong suggestions lowered it. Nonprofit staff on benchmark questions, not agency staff on live cases. - Fitzpatrick, Kelly, Carey, Walsh & Nugent, "Assessing Generative AI value in a public sector context: evidence from a field experiment," arXiv:2502.09479 (2025) Judged from an abstract, summary or search listing only
Abstract read. Pre-registered experiment with a control group in a public-sector body: quality and completion time improved on document tasks and quality fell on data tasks. Constructed tasks, not service outcomes. - UK Department for Work and Pensions (DWP), "An Evaluation of DWP's Microsoft 365 Copilot Trial" (January 2026) Judged from an abstract, summary or search listing onlySome details not confirmed
Judged from search listings; not read directly (gov.uk could not be opened). Reportedly surveys of licensed users and a non-user comparison group with regression adjustment, finding self-reported time savings. Non-random allocation and no service-outcome measures, as far as the listings show. - AEA RCT Registry, "Experimental Evaluation of the Use of Artificial Intelligence in Building Permit Review," trial AEARCTR-0017575 (Chilean municipalities) Judged from an abstract, summary or search listing only
Judged from search listings; not read directly. Reportedly randomizes permit applications within reviewer and permit-type strata with processing time as the outcome; listed as in development, ending 31 December 2026. The design this opportunity describes, with no results yet. - Commonwealth of Pennsylvania, ChatGPT Enterprise pilot report (2025) Judged from an abstract, summary or search listing only
Judged from search listings; not read directly. Reportedly a pilot across 14 state agencies with self-reported time savings and no comparison group.
Search queries used
generative AI government agency randomized evaluation case processing time outcomesMagesh Ho "Evaluating Generative AI in Benefits Administration" unemployment insurance Coloradogenerative AI caseworkers social services field experiment administrative data error rates difference-in-differences public agencylarge language model deployment government civil servants quasi-experimental evaluation service outcomes Korea OR Singapore OR Estonia OR ChinaPennsylvania ChatGPT Enterprise pilot state employees evaluation OR DWP Microsoft 365 Copilot evaluation comparison group"Evaluating Generative AI in Benefits Administration" findings adjudicators decision quality timebuilding permit review AI randomized trial results processing time city governmentDWP Microsoft 365 Copilot evaluation 2026 comparison group findings
Provenance
RO-LD03-05 v1. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-03 debate on 2026-10-15. Checked by an independent AI reviewer with fresh context (not a person), 15 October 2026 (review of PR #77): verdicts, query counts and 13 abstract-read labels checked; label slips corrected. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.