Research opportunity · AI in the research apparatus
Does "LLMs can replace coders" depend on which humans you compare with?
Question
Holding items and codebook fixed, how often does a formal replacement-test verdict flip depending on whether the comparator pool is crowdworkers, trained research assistants, the original study's coders or domain experts, and does the flip rate track construct ambiguity?
Wrong or missing validation targetMaster's thesisLow costPartly addressed
Why it matters
Claims that an LLM "matches humans" are used to justify dropping human coding. If the verdict depends on an unreported choice of comparator, replacement claims in the literature are not comparable, and methods guidance should require reporting the comparator.
Evidence that motivates it
- Whether an LLM can replace human annotation is a task-specific empirical proposition that can sometimes be justified with an explicit statistical replacement test; it is not a general property of LLMs.
Replacement is a task-specific statistical test result. - Recent LLMs can be observationally equivalent to expert coders on some standardized binary political-text classification tasks, particularly when the codebook is clear; this does not imply that substituting LLM labels preserves historical published analyses.
LLM equivalence with experts holds on some clear-codebook tasks. - Human–LLM agreement is not a sufficient decision rule for qualitative coding quality, and current evidence does not establish LLM replacement of researcher interpretation in inductive qualitative analysis.
Human agreement is not a sufficient quality rule, so the choice of human reference matters.
A possible design
Choose 6–10 coding tasks of varying ambiguity with existing expert or original-coder labels. Collect labels from crowdworkers and trained research assistants on the same items. Run the Alternative Annotator Test, or an equivalent, against each comparator pool and model, and model verdict flips as a function of construct ambiguity.
Data needed
Coding tasks with existing expert labels; a modest crowdworker budget; research-assistant coding time; API access.
Feasibility
Master's thesis. Low: a few hundred US dollars for crowd labels.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026, checked again on 13 October 2026).
Indirect evidence suggests the verdict depends on the comparator: LLMs beat outsourced student coders but only match experts (Bermejo et al.), a multi-study political-science comparison finds LLMs descriptively closer to experts than crowd workers are, with disagreement tied to text ambiguity for both (Nakamura et al., crowd labels on 20 texts per study), and expert and crowd pools disagree on a quarter or more of items in preference benchmarks (Jha 2026). The Alternative Annotator Test weights annotator types differently. No study found runs a formal replacement test against several comparator pools (crowd, trained assistants, original coders, experts) on fixed items and codebooks, or models how often the verdict flips as a function of construct ambiguity.
How strong this evidence is
- Who searched and checked: Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert; an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: 1 of the 6 nearest works below was judged from an abstract, summary or search listing only; it is marked. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist. Literature check of 2026-10-13: The PSRM article 'Fine-tuned large language models can replicate expert coding better than trained coders' (doi:10.1017/psrm.2025.10086) appeared in results but its page could not be opened and its authors were not confirmed, so it is not listed; it compares fine-tuned LLMs with trained coders against experts and may be a further indirect data point. Jha (2026) and He et al. (2026) were read from abstracts only. Google Scholar was not searched. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Calderon, Reichart & Dror, "The Alternative Annotator Test for LLM-as-a-Judge," ACL 2025
Sets different cost parameters by annotator type and recommends experts; annotator type is confounded with dataset. - Bermejo, Gago, Gálvez, Harari et al., "LLMs outperform outsourced human coders on complex textual analysis," Scientific Reports 15:40122 (2025)
LLMs agree with expert gold labels more closely than outsourced student coders do; experts are the reference rather than a comparator, and there is no formal replacement test. - Nakamura, Tan & Yean, "Observational Equivalence of LLM and Human Annotation," arXiv:2609.22133 (2026)
Ten LLMs, three experts and 165 crowd workers code texts from 14 political-science studies. Full text read in the literature check: it runs no formal replacement or equivalence test; its comparisons are descriptive, judging LLMs comparable to experts and more aligned with experts than crowd workers are; crowd labels cover only 20 texts per paper; and disagreement is linked to text ambiguity for both comparators. Also posted on SSRN (7389359). - Jha, "Whose Gold? Annotator-Pool Disagreement Is Large at the Item Level, and Hidden by Small Leaderboards," arXiv:2608.15980 (Aug 2026) Judged from an abstract, summary or search listing only
Compares expert and crowd pools (MultiPref) and benchmark authors versus recruited experts (MT-Bench) on internally unanimous items: pools give different majority labels on 23.6% and 30.5% of items; six-model leaderboards do not change but simulations show larger ones would; an LLM judge agrees more with the crowd pool than the expert pool. Shows that the choice of human reference matters, but for preference benchmarks rather than social-science coding, with no Alternative Annotator Test and no explicit ambiguity analysis (abstract only). - Carlson & Burbano, "The Use of LLMs to Annotate Data in Management Research: Foundational Guidelines and Warnings," working paper (c. late 2025; no version date on the PDF)
Validates 35 LLM model-prompt combinations against one expert-consensus gold standard (250 projects) and shows downstream regression results vary with LLM configuration. Crowd and RA labels are not evaluated as separate comparators on the same items, and no formal replacement test is used. - He, Zhou, Vanacore, Kizilcec & Huang, "Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation," arXiv:2609.26926 (22 Sep 2026)
Uses cross-LLM disagreement to target expert codebook revision on tutoring transcripts; evaluates against expert labels only. Related to the ambiguity angle, not to comparator-pool choice or replacement tests.
Literature check
Stands as first published. The closest work, Nakamura, Tan & Yean (arXiv:2609.22133; also SSRN 7389359), read in full in an earlier pass, has experts and crowd workers code the same political-science texts but runs no formal replacement or equivalence test against each pool; it judges LLMs comparable to experts and more aligned with experts than crowd workers descriptively, analyses crowd labels on only 20 texts per paper, and links disagreement to text ambiguity for both comparators. A new preprint (Jha, "Whose Gold?", Aug 2026) shows expert and crowd pools disagree on 24-31% of items in preference benchmarks and that an LLM judge sides more with the crowd pool, but it concerns model leaderboards, not social-science coding, and uses no Alternative Annotator Test. No study runs a formal replacement test against several comparator pools on fixed items and models verdict flips by construct ambiguity.
Searched: web (extended and standard); arxiv.org; osf.io; ssrn.com; aclanthology.org; dl.acm.org; cambridge.org; journals.sagepub.com; pnas.org; nature.com; tandfonline.com; faculty.wharton.upenn.edu. Checked by Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert.
The screen before this check
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026).
Indirect evidence suggests the verdict depends on the comparator (LLMs beat student coders but only match experts), and the Alternative Annotator Test weights annotator types differently. No study found runs a formal replacement test against several comparator pools on fixed items.
Search queries used
Alternative Annotator Test comparator crowdworkers vs experts choice of human baseline changes conclusion LLM annotationLLM annotation evaluation depends on reference annotators expert vs crowd gold standard benchmark social science "which humans"GPT incivility toxicity classification political comments validation against expert coders measurement error downstream inference
Literature check, 13 October 2026:
LLM annotation replacement test results depend on human annotator pool crowd workers vs experts vs research assistants same items"Alternative Annotator Test" LLM annotators crowdworkers experts comparison 2026 [allowed_domains: arxiv.org, osf.io, ssrn.com, papers.ssrn.com]LLM text classification political science benchmark against expert coders and crowd coders agreement varies with task ambiguity [allowed_domains: cambridge.org, journals.sagepub.com, aclanthology.org, pnas.org, nature.com, tandfonline.com]"Observational Equivalence of LLM and Human Annotation" OR "Alternative Annotator Test" cites extends equivalence testing LLM annotation human baseline choicefine-tuned large language models replicate expert coding better than trained coders informative signals interest groups Political Science Research and Methods authors (standard)which human reference matters LLM annotation evaluation expert versus crowd gold labels change model ranking conclusions subjective tasks disagreement [allowed_domains: aclanthology.org, arxiv.org, dl.acm.org]
Provenance
RO-LD07-10 v2. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-07 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.
Changes
- v2, 13 October 2026: Literature check added: a second, deeper search for existing work, with its queries, new nearest work and limits. The question and verdict stand. See v1 as published.