Research opportunity · Levelling or hollowing? Skill compression and the reproduction of expertise
Does skill compression reverse when AI acts as an agent rather than an assistant?
Question
On the same tasks, does the advantage that AI gives less skilled workers when it assists them (a copilot) shrink or reverse when it carries out the work autonomously (an agent), as theory predicts?
Untested beyond the original settingOne PhD chapter or paperLow to medium costPartly addressed
Why it matters
Most evidence that AI helps novices most comes from 2023–2025 assistants. A model of knowledge work predicts that autonomous AI benefits the most knowledgeable instead, and agentic tools are now spreading. If the prediction holds, the levelling evidence describes a passing phase; if not, it may hold for agents too. Field data compare tool generations without splitting by skill, and the one direct agent-versus-chatbot experiment used students.
The debate lists this among what we still do not know: Will compression reverse as AI agents act autonomously rather than assist?
Evidence that motivates it
- A model of knowledge work predicts that AI which only assists (a copilot) benefits the least knowledgeable workers, while autonomous AI benefits the most knowledgeable, so the direction of skill compression can reverse as AI becomes more autonomous.
The model predicts that a copilot benefits the least knowledgeable and autonomous AI the most knowledgeable, so the direction of compression can reverse. - In a 2025 randomized trial with 16 experienced open-source developers working on 246 tasks in their own projects, allowing AI tools increased completion time by 19%, although the developers estimated it had cut their time by 20%; METR later called its 2026 follow-up data an unreliable signal because of selection.
Highly experienced developers were slower with early-2025 AI tools while believing they were faster, and METR's later data were too selected to read: expert gains from newer tools are unmeasured. - A 2026 meta-analysis of 23 programming studies found an overall productivity effect of g = 0.33, large in controlled experiments (g = 0.73) but not distinguishable from zero in enterprise or open-source settings, and no significant effect on learning (g = 0.14).
Programming gains are large in controlled experiments but not distinguishable from zero in enterprise and open-source settings, and the meta-analysis does not test skill as a moderator.
A possible design
Randomized experiment with three arms on the same task bank: no AI, an AI assistant that suggests and explains, and an AI agent that completes tasks for the participant to review. Recruit participants across a measured range of baseline skill (a pre-test on the task domain), for example professional developers or data analysts. Outcomes are task quality graded blind, time, and errors that reach the final output. The estimand is how the gain from each arm changes with baseline skill.
Data needed
A task bank with gradable outcomes; access to current assistant and agent tools with fixed versions; a participant panel with a validated skill pre-test; graders blind to arm.
Feasibility
One PhD chapter or paper. Low to medium; participant payments and tool subscriptions; tool versions change quickly, so the study must fix and report them. Agents work best on some tasks and fail on others, so the task bank must include tasks outside the agent's competence, where review skill matters. Selection of willing participants biased METR's 2026 data, so recruitment must not depend on willingness to work without AI.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 14 October 2026; not yet checked in depth).
No experiment was found that gives the same tasks a no-AI arm, an assistant arm and an autonomous-agent arm and estimates gains by baseline skill. Observational studies compare autocomplete, interactive agents and autonomous agents across hundreds of thousands of developers without a skill split in their abstracts, or find experienced programmers capturing most gains without separating agents from assistants. A small experiment compares a coding agent with a chatbot in students and measures comprehension. Two field experiments in the same customer-service operation found the least skilled gaining most from an assistant but report no skill split for the agentic version.
How strong this evidence is
- Who searched and checked: an independent AI reviewer with fresh context (not a person), 14 October 2026 (review of PR #72): verdicts, nearest-work reading labels and limits checked; query lists corrected to the 8 queries run where they differed. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: 4 of the 5 nearest works below were judged from an abstract, summary or search listing only; each is marked. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search (8 queries), arXiv abstract pages and one NBER abstract page on the date shown; the bodies of the observational studies were not read and may contain skill splits. SSRN and Google Scholar were not searched directly, and experiments outside software may be under-covered. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Demirer, Musolff & Yang, "Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools," NBER Working Paper 35275 (2026, revised September 2026) Judged from an abstract, summary or search listing only
Abstract page read; the paper itself was not read. Matched event study using AI-usage telemetry for more than 500,000 GitHub developers: autocomplete, interactive agents and autonomous agents raise commits by 30%, 180% and 240%, with smaller effects on projects and releases. Not randomized; the abstract reports no split by developer skill. Figures differ from an earlier version. - Daniotti, Wachs, Feng & Neffke, "Who is using AI to code? Global diffusion and impact of generative AI," arXiv:2506.08945 (2025) Judged from an abstract, summary or search listing only
Abstract read. AI-written code detected in tens of millions of commits: experienced programmers capture nearly all the productivity and exploration gains. Observational; does not separate agentic from assistant use in the abstract. - Balepur, Baumler, Chen, Choi, Rudinger & Boyd-Graber, "(Im)Paired Programming: Coding Agents Improve Productivity but Harm Understanding," arXiv:2607.26375 (2026) Judged from an abstract, summary or search listing only
Abstract read. Experiment with 54 students building a website with a code-editing agent or a chatbot: the agent aids completion but harms comprehension and the ability to extend the code without it. Direct mode comparison, but students, no no-AI arm and no skill-gradient analysis in the abstract. - Ni, Wang, Feng, Lu, Wang & Zhou, "Generative AI in Action: Field Experimental Evidence from Alibaba's Customer Service Operations," arXiv:2603.29888 (2026) Judged from an abstract, summary or search listing only
Abstract read. Randomized field experiment with an AI assistant: lower-performing agents benefit most and top performers decline in quality. A companion experiment in the same operation (arXiv:2605.14830, abstract read) tests an agentic set-up and reports no skill split in its abstract; the two are separate experiments, not arms of one design. - Becker, Rush, Barnes & Rein, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity," arXiv:2507.09089 (2025)
Full text read for the LD-04 debate. Randomized AI-allowed against AI-disallowed tasks for experienced developers only; no assistant-versus-agent arm and no novice arm.
Search queries used
experiment AI agent versus copilot assistant productivity heterogeneity worker skill experience randomized 2025 2026autonomous AI agents benefit high-skilled workers more experiment "agentic" coding agents experienced developers gains heterogeneityMETR developer productivity study 2026 results AI agents Claude Code experienced developersautocomplete interactive coding agents autonomous coding agents GitHub developers coding activity 40% 140% 180% paperCursor agent adoption experienced developers productivity planning supervising preprint arXivDaniotti 2026 AI coding 160,000 developers experienced developers gains early-careerrandomized experiment chat assistant versus autonomous agent arm knowledge workers high-skill low-skill benefit "agent" treatment 2026coding agents productivity effect by developer experience junior versus senior heterogeneity 2026 study Claude Code Codex tenure
Provenance
RO-LD04-05 v1. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-04 debate on 2026-10-14. Checked by an independent AI reviewer with fresh context (not a person), 14 October 2026 (review of PR #72): verdicts, nearest-work reading labels and limits checked; query lists corrected to the 8 queries run where they differed. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.