Does "LLMs can replace coders" depend on which humans you compare with?

Research question

Holding items and codebook fixed, how often does a formal replacement-test verdict flip depending on whether the comparator pool is crowdworkers, trained research assistants, the original study's coders or domain experts, and does the flip rate track construct ambiguity?

Why it matters

Claims that an LLM "matches humans" are used to justify dropping human coding. If the verdict depends on an unreported choice of comparator, replacement claims in the literature are not comparable, and methods guidance should require reporting the comparator.

Evidence that motivates it

1. Whether an LLM can replace human annotation is a task-specific empirical proposition that can sometimes be justified with an explicit statistical replacement test; it is not a general property of LLMs. (LD07-HQ-C03 v1.0)
   Replacement is a task-specific statistical test result.
   Cites: Calderon et al. (2025), “The Alternative Annotator Test…”
2. Recent LLMs can be observationally equivalent to expert coders on some standardized binary political-text classification tasks, particularly when the codebook is clear; this does not imply that substituting LLM labels preserves historical published analyses. (LD07-HQ-C04 v1.0)
   LLM equivalence with experts holds on some clear-codebook tasks.
   Cites: Nakamura et al., “Observational Equivalence of LLM and…”
3. Human–LLM agreement is not a sufficient decision rule for qualitative coding quality, and current evidence does not establish LLM replacement of researcher interpretation in inductive qualitative analysis. (LD07-HQ-C15 v1.0)
   Human agreement is not a sufficient quality rule, so the choice of human reference matters.
   Cites: Liu et al., “Agreement Is Not Quality: Blind Expert…”; “Large language models for thematic analysis in healthcare…”

Claims are shown as they read on the day this pack was built; claim pages keep every earlier version.

A possible design

Choose 6–10 coding tasks of varying ambiguity with existing expert or original-coder labels. Collect labels from crowdworkers and trained research assistants on the same items. Run the Alternative Annotator Test, or an equivalent, against each comparator pool and model, and model verdict flips as a function of construct ambiguity.

Data needed

Coding tasks with existing expert labels; a modest crowdworker budget; research-assistant coding time; API access.

Feasibility

Master's thesis. Low: a few hundred US dollars for crowd labels.

Has this been done?

Partly done already: related work covers some of this question (first-pass screen of 12 October 2026, checked again on 13 October 2026).

Indirect evidence suggests the verdict depends on the comparator: LLMs beat outsourced student coders but only match experts (Bermejo et al.), a multi-study political-science comparison finds LLMs descriptively closer to experts than crowd workers are, with disagreement tied to text ambiguity for both (Nakamura et al., crowd labels on 20 texts per study), and expert and crowd pools disagree on a quarter or more of items in preference benchmarks (Jha 2026). The Alternative Annotator Test weights annotator types differently. No study found runs a formal replacement test against several comparator pools (crowd, trained assistants, original coders, experts) on fixed items and codebooks, or models how often the verdict flips as a function of construct ambiguity.

Literature check. The closest work, Nakamura, Tan & Yean (arXiv:2609.22133; also SSRN 7389359), read in full in an earlier pass, has experts and crowd workers code the same political-science texts but runs no formal replacement or equivalence test against each pool; it judges LLMs comparable to experts and more aligned with experts than crowd workers descriptively, analyses crowd labels on only 20 texts per paper, and links disagreement to text ambiguity for both comparators. A new preprint (Jha, "Whose Gold?", Aug 2026) shows expert and crowd pools disagree on 24-31% of items in preference benchmarks and that an LLM judge sides more with the crowd pool, but it concerns model leaderboards, not social-science coding, and uses no Alternative Annotator Test. No study runs a formal replacement test against several comparator pools on fixed items and models verdict flips by construct ambiguity. Checked by Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert.

Where the search could not reach. Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist. Literature check of 2026-10-13: The PSRM article 'Fine-tuned large language models can replicate expert coding better than trained coders' (doi:10.1017/psrm.2025.10086) appeared in results but its page could not be opened and its authors were not confirmed, so it is not listed; it compares fine-tuned LLMs with trained coders against experts and may be a further indirect data point. Jha (2026) and He et al. (2026) were read from abstracts only. Google Scholar was not searched. Absence of a hit is not proof that the work does not exist.

Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.

Nearest work found

• Calderon, Reichart & Dror, "The Alternative Annotator Test for LLM-as-a-Judge," ACL 2025
  Sets different cost parameters by annotator type and recommends experts; annotator type is confounded with dataset.
• Bermejo, Gago, Gálvez, Harari et al., "LLMs outperform outsourced human coders on complex textual analysis," Scientific Reports 15:40122 (2025)
  LLMs agree with expert gold labels more closely than outsourced student coders do; experts are the reference rather than a comparator, and there is no formal replacement test.
• Nakamura, Tan & Yean, "Observational Equivalence of LLM and Human Annotation," arXiv:2609.22133 (2026)
  Ten LLMs, three experts and 165 crowd workers code texts from 14 political-science studies. Full text read in the literature check: it runs no formal replacement or equivalence test; its comparisons are descriptive, judging LLMs comparable to experts and more aligned with experts than crowd workers are; crowd labels cover only 20 texts per paper; and disagreement is linked to text ambiguity for both comparators. Also posted on SSRN (7389359).
• Jha, "Whose Gold? Annotator-Pool Disagreement Is Large at the Item Level, and Hidden by Small Leaderboards," arXiv:2608.15980 (Aug 2026) Judged from an abstract, summary or search listing only
  Compares expert and crowd pools (MultiPref) and benchmark authors versus recruited experts (MT-Bench) on internally unanimous items: pools give different majority labels on 23.6% and 30.5% of items; six-model leaderboards do not change but simulations show larger ones would; an LLM judge agrees more with the crowd pool than the expert pool. Shows that the choice of human reference matters, but for preference benchmarks rather than social-science coding, with no Alternative Annotator Test and no explicit ambiguity analysis (abstract only).
• Carlson & Burbano, "The Use of LLMs to Annotate Data in Management Research: Foundational Guidelines and Warnings," working paper (c. late 2025; no version date on the PDF)
  Validates 35 LLM model-prompt combinations against one expert-consensus gold standard (250 projects) and shows downstream regression results vary with LLM configuration. Crowd and RA labels are not evaluated as separate comparators on the same items, and no formal replacement test is used.
• He, Zhou, Vanacore, Kizilcec & Huang, "Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation," arXiv:2609.26926 (22 Sep 2026)
  Uses cross-LLM disagreement to target expert codebook revision on tutoring transcripts; evaluates against expert labels only. Related to the ambiguity angle, not to comparator-pool choice or replacement tests.

References

Sources behind the motivating claims:

• Calderon, Reichart & Dror, “The Alternative Annotator Test for LLM-as-a-Judge,” ACL 2025 (SRC-LD07-CALDERON-ALTERNATIVE-ANNOTATOR-2025, original)
• Nakamura, Tan & Yean, “Observational Equivalence of LLM and Human Annotation,” arXiv:2609.22133 (SRC-LD07-NAKAMURA-OBSERVATIONAL-EQUIVALENCE-2026, original)
• Liu et al., “Agreement Is Not Quality: Blind Expert Verification of Human and LLM Qualitative Coding When Human Consensus Is Not Ground Truth,” arXiv:2607.28890 (SRC-LD07-LIU-AGREEMENT-IS-NOT-QUALITY-2026, original)
• “Large language models for thematic analysis in healthcare research: A blinded mixed-methods comparison with human analysts,” PLOS Digital Health 2026 (SRC-LD07-PLOS-THEMATIC-ANALYSIS-2026, original)

BibTeX

The first entry cites this opportunity; then its nearest work; then the sources above. Nearest work and sources are recorded as reference strings, so each goes in whole as the title; add authors and year from the original before you cite it.

@misc{sociolens_ro_ld07_10_v2,
  author = {{SocioLens project}},
  title = {Does "LLMs can replace coders" depend on which humans you compare with?},
  year = {2026},
  date = {2026-10-13},
  howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld07-10/v2/}},
  note = {Research opportunity RO-LD07-10, version v2}
}

@misc{ro_ld07_10_nearest_1,
  title = {Calderon, Reichart \& Dror, "The Alternative Annotator Test for LLM-as-a-Judge," ACL 2025},
  url = {https://doi.org/10.18653/v1/2025.acl-long.782},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_10_nearest_2,
  title = {Bermejo, Gago, Gálvez, Harari et al., "LLMs outperform outsourced human coders on complex textual analysis," Scientific Reports 15:40122 (2025)},
  url = {https://www.nature.com/articles/s41598-025-23798-y},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_10_nearest_3,
  title = {Nakamura, Tan \& Yean, "Observational Equivalence of LLM and Human Annotation," arXiv:2609.22133 (2026)},
  url = {https://arxiv.org/abs/2609.22133},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_10_nearest_4,
  title = {Jha, "Whose Gold? Annotator-Pool Disagreement Is Large at the Item Level, and Hidden by Small Leaderboards," arXiv:2608.15980 (Aug 2026)},
  url = {https://arxiv.org/abs/2608.15980},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld07_10_nearest_5,
  title = {Carlson \& Burbano, "The Use of LLMs to Annotate Data in Management Research: Foundational Guidelines and Warnings," working paper (c. late 2025; no version date on the PDF)},
  url = {https://faculty.wharton.upenn.edu/wp-content/uploads/2025/09/The-use-of-LLMs-to-Annotate-Data.pdf},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_10_nearest_6,
  title = {He, Zhou, Vanacore, Kizilcec \& Huang, "Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation," arXiv:2609.26926 (22 Sep 2026)},
  url = {https://arxiv.org/abs/2609.26926},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_calderon_alternative_annotator_2025,
  title = {Calderon, Reichart \& Dror, “The Alternative Annotator Test for LLM-as-a-Judge,” ACL 2025},
  url = {https://doi.org/10.18653/v1/2025.acl-long.782},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_nakamura_observational_equivalence_2026,
  title = {Nakamura, Tan \& Yean, “Observational Equivalence of LLM and Human Annotation,” arXiv:2609.22133},
  url = {https://arxiv.org/abs/2609.22133},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_liu_agreement_is_not_quality_2026,
  title = {Liu et al., “Agreement Is Not Quality: Blind Expert Verification of Human and LLM Qualitative Coding When Human Consensus Is Not Ground Truth,” arXiv:2607.28890},
  url = {https://arxiv.org/abs/2607.28890},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_plos_thematic_analysis_2026,
  title = {“Large language models for thematic analysis in healthcare research: A blinded mixed-methods comparison with human analysts,” PLOS Digital Health 2026},
  url = {https://doi.org/10.1371/journal.pdig.0001189},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}