Research opportunity · AI in the research apparatus
Does "LLMs can replace coders" depend on which humans you compare with?
Earlier version (v1)
This is RO-LD07-10 as it was published at v1, kept unchanged. The current version is v2; its changes are listed there.
Question
Holding items and codebook fixed, how often does a formal replacement-test verdict flip depending on whether the comparator pool is crowdworkers, trained research assistants, the original study's coders or domain experts, and does the flip rate track construct ambiguity?
Wrong or missing validation targetMaster's thesisLow costPartly addressed
Why it matters
Claims that an LLM "matches humans" are used to justify dropping human coding. If the verdict depends on an unreported choice of comparator, replacement claims in the literature are not comparable, and methods guidance should require reporting the comparator.
Evidence that motivates it
- Whether an LLM can replace human annotation is a task-specific empirical proposition that can sometimes be justified with an explicit statistical replacement test; it is not a general property of LLMs.
Replacement is a task-specific statistical test result. - Recent LLMs can be observationally equivalent to expert coders on some standardized binary political-text classification tasks, particularly when the codebook is clear; this does not imply that substituting LLM labels preserves historical published analyses.
LLM equivalence with experts holds on some clear-codebook tasks. - Human–LLM agreement is not a sufficient decision rule for qualitative coding quality, and current evidence does not establish LLM replacement of researcher interpretation in inductive qualitative analysis.
Human agreement is not a sufficient quality rule, so the choice of human reference matters.
A possible design
Choose 6–10 coding tasks of varying ambiguity with existing expert or original-coder labels. Collect labels from crowdworkers and trained research assistants on the same items. Run the Alternative Annotator Test, or an equivalent, against each comparator pool and model, and model verdict flips as a function of construct ambiguity.
Data needed
Coding tasks with existing expert labels; a modest crowdworker budget; research-assistant coding time; API access.
Feasibility
Master's thesis. Low: a few hundred US dollars for crowd labels. Check the full text of Nakamura, Tan & Yean first: it has expert and crowd comparators and may partly answer this.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026; not yet checked in depth).
Indirect evidence suggests the verdict depends on the comparator (LLMs beat student coders but only match experts), and the Alternative Annotator Test weights annotator types differently. No study found runs a formal replacement test against several comparator pools on fixed items.
How strong this evidence is
- Who searched and checked: an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: none of the 3 nearest works below is recorded as judged from an abstract only. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Calderon, Reichart & Dror, "The Alternative Annotator Test for LLM-as-a-Judge," ACL 2025
Sets different cost parameters by annotator type and recommends experts; annotator type is confounded with dataset. - Bermejo, Gago, Gálvez, Harari et al., "LLMs outperform outsourced human coders on complex textual analysis," Scientific Reports 15:40122 (2025)
LLMs agree with expert gold labels more closely than outsourced student coders do; experts are the reference rather than a comparator, and there is no formal replacement test. - Nakamura, Tan & Yean, "Observational Equivalence of LLM and Human Annotation," arXiv:2609.22133 (2026)
Ten LLMs, three experts and 165 crowd workers code texts from 14 political-science studies; the abstract reports LLM equivalence with experts but no crowd-specific verdict.
Search queries used
Alternative Annotator Test comparator crowdworkers vs experts choice of human baseline changes conclusion LLM annotationLLM annotation evaluation depends on reference annotators expert vs crowd gold standard benchmark social science "which humans"GPT incivility toxicity classification political comments validation against expert coders measurement error downstream inference
Provenance
RO-LD07-10 v1. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-07 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.