Research opportunity · AI in the research apparatus
Do LLM judges inflate the quality of AI-written deliberation?
Question
On a corpus mixing human-written and LLM-written deliberative contributions, do LLM judges assign higher construct scores (for example on the Discourse Quality Index or argument quality) to LLM-authored text than expert coders blind to authorship, and is the gap larger when the judge model also wrote the text?
Wrong or missing validation targetOne PhD chapter or paperLow to medium costPartly addressed
Why it matters
LLM judges are moving into social science to score deliberation, civility and argument quality, at the same time as online discussion fills with AI-written text. A judge that favours machine prose would quietly bias measures of public deliberation.
Evidence that motivates it
- LLM judges can correlate with human evaluations while exhibiting systematic evaluator-specific biases, including position bias and self-preference; task-specific validation is therefore required.
LLM judges show position bias and self-preference, so construct-level validation is required. - Whether an LLM can replace human annotation is a task-specific empirical proposition that can sometimes be justified with an explicit statistical replacement test; it is not a general property of LLMs.
Replacing human judgement is a task-specific claim that has to be tested, not assumed.
A possible design
Build a corpus of matched deliberative contributions on the same prompts written by people and by several LLMs. Have expert coders score them with an established instrument while blind to authorship. Score the same items with several LLM judges, including each generating model. Estimate the authorship gap between judges and experts, and the extra gap for self-authored text.
Data needed
Existing deliberation corpora (for example from online deliberation platforms or citizens' assemblies); generated counterparts; expert coder time.
Feasibility
One PhD chapter or paper. Low to medium: coder time is the main cost.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026, checked again on 13 October 2026).
LLM scoring of deliberation quality has been validated on human-written text (Gelauff et al.), and LLMs prefer LLM-written product, paper and film descriptions over human judges' choices (Laurito et al.), though a reanalysis finds a shared taste for model-written text rather than a preference for a model's own text, and blind evaluation removes self-preference in another task (Chae et al.). No study found tests judge-mode scoring of an established deliberation construct on mixed human and LLM text against authorship-blind experts, or estimates the extra gap for self-authored deliberative text.
How strong this evidence is
- Who searched and checked: Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert; an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: none of the 8 nearest works below is recorded as judged from an abstract only. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist. Literature check of 2026-10-13: Six searches were run for this opportunity under the shared budget; full texts of Chae et al. and Flechtner were not read, so whether either uses human scorers is unconfirmed from abstracts. Google Scholar and political-communication journals beyond those listed were not searched. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Gelauff, Goyal, Dindukurthi, Goel & Siu, "Estimating Contribution Quality in Online Deliberations Using a Large Language Model," arXiv:2408.11936 (2024)
LLM scores deliberation contributions well against human averages; all texts human-written. - Laurito, Davis, Grietzer, Gavenčiak, Böhm & Kulveit, "AI–AI bias: large language models favor communications generated by large language models," PNAS 122 (2025)
LLMs prefer LLM-written descriptions of products, papers and films; not social-science constructs. - Chehbouni et al., "Neither Valid nor Reliable? Investigating the Use of LLMs as Judges," arXiv:2508.18076 (2025)
Conceptual critique using measurement theory; no empirical construct test. - Żatuchin, "A Shared Taste for Model-Written Text: The Generator-by-Selector Matrices of 'AI-AI Bias' Show No Detectable Own-Model Premium," arXiv:2610.00369 (30 Sep 2026)
Reanalyses the public Laurito et al. data: no own-model premium (pooled +0.019, p = 0.14; design could detect about 0.04 or more), but a robust shared preference for model-written text; contrasts with a sizeable own-model term in Tan et al. (ACL 2024) data. Product, abstract and film descriptions only; no new human judges and no deliberation construct. Directly bears on the opportunity's 'extra gap for self-authored text' sub-question. - Chae, Kim, Jung, Choi & Jung, "Self- and Other-Labels Induce Bidirectional Bias in LLM Judges," arXiv:2608.18091 (v2, Sep 2026)
Ten LLMs judge sets of narrative constraints; under blind evaluation self-preference disappears, while self/other authorship labels inflate or deflate scores regardless of true source. Not deliberative text and no human baseline mentioned in the abstract; shows the blind versus labelled condition matters for the proposed design. - Chen, Wang, Zhang, Hu & Lin, "Beyond the Surface: Measuring Self-Preference in LLM Judgments," EMNLP 2025, pp. 1653-1672
Proposes measuring self-preference against gold judgments of quality rather than raw own-versus-other scores, the same logic as the opportunity's expert benchmark. General NLP tasks; the abstract does not say the gold judgments are human, and there is no deliberation construct. - Panickssery, Bowman & Feng, "LLM Evaluators Recognize and Favor Their Own Generations," NeurIPS 2024
Authors and title checked on the arXiv page (arXiv:2404.13076); the NeurIPS 2024 venue is from search results. Summarisation tasks: self-recognition correlates with self-preference. Not social-science constructs or expert coders. - Flechtner, "The Deliberative Deficit: An Empirical Critique of LLMs in Democratic Discourse," arXiv:2608.10186 (10 Aug 2026)
1,980 runs of five-agent LLM groups on 12 citizen-assembly topics scored with the Deliberative Reason Index and compared with human assemblies at group level. The abstract does not say who scored the text or test whether LLM judges inflate AI-written contributions.
Literature check
Stands as first published. Six new queries found no study that scores an established deliberation construct (DQI, argument quality) on a mixed human and LLM corpus with LLM judges against authorship-blind expert coders. New work sharpens rather than settles the question: a reanalysis of the Laurito et al. data finds a shared preference for model-written text but no own-model premium (Żatuchin, Sept 2026), a blind-evaluation study finds self-preference vanishes when authorship labels are removed but labels alone shift scores (Chae et al., 2026), and a deliberation study compares LLM agent groups with citizen assemblies on the Deliberative Reason Index at group level without testing judge bias (Flechtner, Aug 2026). None of these uses deliberative text with expert coders as the benchmark.
Searched: web (extended); arxiv.org; osf.io; ssrn.com; aclanthology.org; dl.acm.org; openreview.net; proceedings.neurips.cc; cambridge.org; journals.sagepub.com; tandfonline.com; academic.oup.com; sciencedirect.com; pmc.ncbi.nlm.nih.gov. Checked by Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert.
The screen before this check
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026).
LLM scoring of deliberation quality has been validated on human-written text, and AI–AI preference has been shown on product and paper descriptions. No study found tests judge-mode scoring of an established deliberation construct on mixed human and LLM text against blinded experts.
Search queries used
LLM-as-a-judge deliberative quality index measurement validity human codersLLM rating argument quality incivility construct validity self-preference AI-generated vs human-written comments political communicationautomated measurement Discourse Quality Index large language models deliberation coding validation arXiv OR Political AnalysisLLM raters inflate quality scores of AI-generated deliberative contributions compared with human experts mixed corpus
Literature check, 13 October 2026:
LLM judge self-preference bias evaluating AI-generated vs human-written arguments argument quality annotation experts blindAI-generated deliberation comments quality rated by LLM versus human coders discourse quality index mixed corpus [allowed_domains: arxiv.org, osf.io, ssrn.com, papers.ssrn.com, osf.io/preprints/socarxiv]self-preference bias LLM evaluators favor own generations argument persuasiveness essays human raters [allowed_domains: aclanthology.org, dl.acm.org, openreview.net, proceedings.neurips.cc]"AI-AI bias" Laurito extends OR "we build on" LLM preference for LLM-generated text 2026 human evaluatorslarge language models measure deliberative quality justification respect validation human coders AI-written contributions [allowed_domains: cambridge.org, journals.sagepub.com, tandfonline.com, academic.oup.com, delibdemjournal.org, sciencedirect.com]LLM raters score AI-written political arguments or comments higher than human raters do, authorship gap, citizen deliberation, Europolis OR Stanford Online Deliberation
Provenance
RO-LD07-09 v2. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-07 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.
Changes
- v2, 13 October 2026: Literature check added: a second, deeper search for existing work, with its queries, new nearest work and limits. The question and verdict stand. See v1 as published.