Research opportunity · Levelling or hollowing? Skill compression and the reproduction of expertise
Does retained skill depend on how routine the task is and who is learning?
Question
After several weeks of AI-assisted practice, does performance without the tool depend jointly on how routine the task is and on the worker's prior expertise, as the split between existing tool-removed studies suggests?
Causal mechanism not isolatedOne PhD chapter or paperLow to medium costNo study found
Why it matters
Tool-removed studies disagree: routine customer-support agents kept their gains, senior patent lawyers gained unaided skill while juniors did not, and novices learning new material without guardrails lost ground. These studies differ in task, horizon and population at once, so the pattern could come from any of them. A single design that crosses task type with expertise would show whether "routine tasks are safe, judgement tasks are risky for novices" is real, which is what employers and educators need to know before deciding where AI help is harmless.
The debate lists this among what we still do not know: Does retained skill depend on how routine the task is and on prior expertise, as the split in tool-removed studies suggests?
Evidence that motivates it
- In the customer-support firm where an AI assistant raised productivity most for less-skilled agents, agents who had used it still outperformed their own pre-AI baseline during software outages when it gave no suggestions, more so after longer exposure and closer adherence to its suggestions.
In routine customer support, agents still beat their pre-AI baseline when the assistant was briefly unavailable: retained gains in a routine task. - In a three-month randomized trial with 133 patent lawyers, AI assistance raised graded drafting quality by 0.38 standard deviations at 90 days; on a later task done without AI, treated seniors scored 0.45 standard deviations higher than control seniors, treated juniors no higher (−0.03), and the senior–junior difference was statistically significant.
In judgement-heavy patent drafting, three months of AI help improved unaided skill for seniors but not juniors: the expertise contrast within one task. - In a randomized experiment with 52 Python programmers learning an unfamiliar library, those given an AI assistant scored 17% lower on a later quiz of concepts, code reading and debugging, with no significant speed gain; participants who delegated code to the AI scored lowest.
Novices learning a new programming library with AI scored lower without it, and delegation predicted the lowest scores. - In two preregistered lab experiments and a field study, LLM access had no overall effect on learning: students who used it to generate solutions covered more topics but understood each less, students who asked it for explanations understood more, and the gap between students with low and high prior knowledge widened.
Students who used an LLM to generate solutions understood less, and the gap by prior knowledge widened.
A possible design
Two-by-two-by-two experiment: routine versus judgement-heavy tasks from the same domain (for example, standard contract clauses versus clause negotiation advice, or bug fixes versus design decisions), novice versus experienced participants recruited from the same profession, and AI-assisted versus unassisted practice. Participants practise over four to six weeks on a platform that logs AI use, with an unassisted baseline test and unassisted tests at the end and four weeks later. The estimand is the three-way interaction on the unassisted score.
Data needed
A participant pool split by experience (an online professional panel or partner firm); task banks with parallel forms for each task type; blind grading rubrics; AI-use logs.
Feasibility
One PhD chapter or paper. Low to medium; participant payments for several weeks of practice are the main cost; the platform can be built on existing survey and coding tools. Defining "routine" in advance is the hard part; pre-register the task classification and validate it with practitioners. Attrition over several weeks must be checked for differences by arm.
Has this been done?
The search found no study that does the core design (first-pass screen of 14 October 2026; not yet checked in depth).
No study was found that crosses task routineness with prior expertise and measures performance after the tool is removed following weeks of assisted practice. Each existing study varies one factor at most in one task: the patent-drafting trial varies experience on one judgement-heavy task; the customer-support outage analysis covers one routine task; laboratory experiments vary how AI is used over minutes to a day and do not cross task type. The meta-analyses found pool outcomes without coding routineness or expertise.
How strong this evidence is
- Who searched and checked: an independent AI reviewer with fresh context (not a person), 14 October 2026 (review of PR #72): verdicts, nearest-work reading labels and limits checked; query lists corrected to the 8 queries run where they differed. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: 4 of the 6 nearest works below were judged from an abstract, summary or search listing only; each is marked. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search (5 queries) and arXiv abstract pages on the date shown. Several other meta-analyses in the listings were not opened, so whether any of them codes retention against task type is not confirmed. One ChatGPT meta-analysis seen in the listings has been retracted and was not used. PsycINFO and management journals were not searched directly. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Autor, Rodchenko, Martin, Iscenko, Strand, Pearl & Ferere, "Does AI Assistance Enhance or Erode Expertise? Evidence from a Three-Month Field Experiment in Patent Drafting," NBER Working Paper 35720 (2026)
Full text read for the LD-04 debate. Varies experience and tests unaided skill at 90 days, but on a single judgement-heavy task with no routine arm and no baseline test: one half of the proposed design. - Liu, Christian, Dumbalska, Bakker & Dubey, "AI Assistance Reduces Persistence and Hurts Independent Performance," arXiv:2604.04721 (2026) Judged from an abstract, summary or search listing only
Abstract read. Randomized experiments with 1,222 participants on maths reasoning and reading comprehension: about ten minutes of AI help raises immediate performance, then lowers unassisted performance and persistence. Several task types but no expertise factor and no delay of weeks. - Wu, Belem, Fu, Steyvers & Smyth, "How AI Assistance Affects Human Skill Development: A Study of Learning with Logic Puzzles," arXiv:2608.23543 (2026) Judged from an abstract, summary or search listing only
Abstract read. Logic-puzzle experiment with phases before, during and after AI, varying the cost of asking the AI; asking predicts worse performance after removal, and a latent-ability model separates initial ability from skill change. One session; no task-type or professional-expertise factor. - Lira, Rogers, Goldstein, Ungar & Duckworth, "Learning not cheating: AI assistance can enhance rather than hinder skill development," arXiv:2502.02880 Judged from an abstract, summary or search listing only
Seen in search results only; not read directly. Reported experiments in which practising cover letters with AI improves an unaided writing test a day later, no more than viewing an AI example. A retained-gain counter-case; one task type, no expertise factor. - Maier, Gunzenhäuser, Schweisthal, Schneider & Feuerriegel, "A meta-analysis of the effect of generative AI on productivity and learning in programming," arXiv:2605.04779 (2026)
Full text read for the LD-04 debate. Learning effect not significant (g = 0.14); does not code task routineness or prior expertise as moderators. - Strohmaier, Bödefeld, Straser & Reinhold, "LLAMA LIMA: A Living Meta-Analysis on the Effects of Generative AI on Learning Mathematics," arXiv:2601.18685 (2026) Judged from an abstract, summary or search listing only
Abstract read. 34 studies; moderators are integration mode and intervention duration. Does not separate tool-removed retention from assisted performance or cross task type with expertise, according to the abstract.
Search queries used
experiment AI assistance skill retention after tool removed novices versus experts task complexity routine heterogeneitymeta-analysis generative AI ChatGPT learning performance retention transfer effect size 2025 2026arXiv 2026 AI-assisted practice then unassisted test prior expertise moderates learning loss factorial experiment knowledge workers"Learning not cheating" AI assistance enhance skill development writing cover letters persisted after AI removedexperiment generative AI skill retention varies by task type creative versus analytical versus routine, novice and experienced participants, performance without AI afterwards
Provenance
RO-LD04-02 v1. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-04 debate on 2026-10-14. Checked by an independent AI reviewer with fresh context (not a person), 14 October 2026 (review of PR #72): verdicts, nearest-work reading labels and limits checked; query lists corrected to the 8 queries run where they differed. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.