Research opportunity · AI in the research apparatus
Do LLM judges inflate the quality of AI-written deliberation?
Earlier version (v1)
This is RO-LD07-09 as it was published at v1, kept unchanged. The current version is v2; its changes are listed there.
Question
On a corpus mixing human-written and LLM-written deliberative contributions, do LLM judges assign higher construct scores (for example on the Discourse Quality Index or argument quality) to LLM-authored text than expert coders blind to authorship, and is the gap larger when the judge model also wrote the text?
Wrong or missing validation targetOne PhD chapter or paperLow to medium costPartly addressed
Why it matters
LLM judges are moving into social science to score deliberation, civility and argument quality, at the same time as online discussion fills with AI-written text. A judge that favours machine prose would quietly bias measures of public deliberation.
Evidence that motivates it
- LLM judges can correlate with human evaluations while exhibiting systematic evaluator-specific biases, including position bias and self-preference; task-specific validation is therefore required.
LLM judges show position bias and self-preference, so construct-level validation is required. - Whether an LLM can replace human annotation is a task-specific empirical proposition that can sometimes be justified with an explicit statistical replacement test; it is not a general property of LLMs.
Replacing human judgement is a task-specific claim that has to be tested, not assumed.
A possible design
Build a corpus of matched deliberative contributions on the same prompts written by people and by several LLMs. Have expert coders score them with an established instrument while blind to authorship. Score the same items with several LLM judges, including each generating model. Estimate the authorship gap between judges and experts, and the extra gap for self-authored text.
Data needed
Existing deliberation corpora (for example from online deliberation platforms or citizens' assemblies); generated counterparts; expert coder time.
Feasibility
One PhD chapter or paper. Low to medium: coder time is the main cost.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026; not yet checked in depth).
LLM scoring of deliberation quality has been validated on human-written text, and AI–AI preference has been shown on product and paper descriptions. No study found tests judge-mode scoring of an established deliberation construct on mixed human and LLM text against blinded experts.
How strong this evidence is
- Who searched and checked: an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: none of the 3 nearest works below is recorded as judged from an abstract only. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Gelauff, Goyal, Dindukurthi, Goel & Siu, "Estimating Contribution Quality in Online Deliberations Using a Large Language Model," arXiv:2408.11936 (2024)
LLM scores deliberation contributions well against human averages; all texts human-written. - Laurito, Davis, Grietzer, Gavenčiak, Böhm & Kulveit, "AI–AI bias: large language models favor communications generated by large language models," PNAS 122 (2025)
LLMs prefer LLM-written descriptions of products, papers and films; not social-science constructs. - Chehbouni et al., "Neither Valid nor Reliable? Investigating the Use of LLMs as Judges," arXiv:2508.18076 (2025)
Conceptual critique using measurement theory; no empirical construct test.
Search queries used
LLM-as-a-judge deliberative quality index measurement validity human codersLLM rating argument quality incivility construct validity self-preference AI-generated vs human-written comments political communicationautomated measurement Discourse Quality Index large language models deliberation coding validation arXiv OR Political AnalysisLLM raters inflate quality scores of AI-generated deliberative contributions compared with human experts mixed corpus
Provenance
RO-LD07-09 v1. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-07 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.