Claim LD06-SL-C18 (v1.0)

Across 20 datasets with human annotations, the best language-model judge (GPT-4o) reached an average Spearman correlation of only 0.50 with human judgements on the datasets with graded annotations (categorical ones were scored by Cohen’s kappa), and agreement varied widely by dataset, by the property judged and by whether humans or models wrote the text; the authors advise validating judges against task-specific human judgements before use.

Provenance and status

Claim type: ESTIMATE (as printed: “ESTIMATE”)
Publication status: ELIGIBLE (QUALIFIED) (as printed: “ELIGIBLE WITH QUALIFIER”)
Verification state: CONTENT_VERIFIED_PRIMARY (as printed: “CONTENT VERIFIED”; verified by: UNKNOWN)
Does not license: Judge reliability for tasks not tested
Source (field 6, as printed): Bavaresco, Bernardi, Bertolazzi, Elliott, Fernández, Gatt et al., “LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks,” ACL 2025, arXiv:2406.18403
Source type (field 7): PEER_REVIEWED_CONFERENCE_PAPER (as printed: “Peer-reviewed conference paper”)
Institutional position: Independent academic (mostly European universities)
Maturity (claim evidence / publication maturity): RAPIDLY_MATURING (as printed: “RAPIDLY MATURING”)
Recorded date: 2026-10-15
Currentness / correction: This is the current version. No correction or concern relation is recorded on its Sources.

Version

Current version v1.0, resolved through the explicit current pointer CLAIM--LD06-SL-C18.

Versions:v1.0 · current

Pinned Sources (field 6, version-specific)

“Bavaresco, Bernardi, Bertolazzi, Elliott, Fernández, Gatt et al., “LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks,” ACL 2025, arXiv:2406.18403” → SRC-SL-BAVARESCO-ET-AL-JUDGE-BENCH-ACL-2025--v1.0 — Bavaresco, Bernardi, Bertolazzi, Elliott, Fernández, Gatt et al., “LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks,” ACL 2025, arXiv:2406.18403; class PEER_REVIEWED_CONFERENCE_PAPER (explicit public provenance pin)

All 25 canonical Claim fields (raw values and normalized values)

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD06-SL-C18	—
2	PUBLIC-FACING CLAIM	Across 20 datasets with human annotations, the best language-model judge (GPT-4o) reached an average Spearman correlation of only 0.50 with human judgements on the datasets with graded annotations (categorical ones were scored by Cohen’s kappa), and agreement varied widely by dataset, by the property judged and by whether humans or models wrote the text; the authors advise validating judges against task-specific human judgements before use.	—
3	CLAIM TYPE	ESTIMATE	base_types: ESTIMATE
4	RESEARCH STRAND	B — Leaderboards, judges and disclosure	—
5	MATURITY	RAPIDLY MATURING	states: RAPIDLY_MATURING
6	SOURCE	Bavaresco, Bernardi, Bertolazzi, Elliott, Fernández, Gatt et al., “LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks,” ACL 2025, arXiv:2406.18403	source_ids: SRC-SL-BAVARESCO-ET-AL-JUDGE-BENCH-ACL-2025
7	SOURCE TYPE	Peer-reviewed conference paper	classes: PEER_REVIEWED_CONFERENCE_PAPER
8	INSTITUTIONAL POSITION	Independent academic (mostly European universities)	—
9	SCOPE	20 NLP evaluation tasks	—
10	DESIGN	Comparison of model judgements with existing human annotations	—
11	SAMPLE / DATASET	20 datasets; 11 models as judges	—
12	MODEL / TOOL / VERSION	GPT-4o, Llama-3.1-70B, Mixtral-8x22B, Command-R+ and others	—
13	COMPARATOR	Human annotators	—
14	MAIN RESULT	GPT-4o average Spearman 0.50 (±0.21) on graded-annotation datasets; Llama-3.1-70B 0.43; Mixtral-8x22B 0.44; Command-R+ 0.30; chain-of-thought did not consistently improve agreement	—
15	INFERENTIAL TARGET	Whether language models can replace human judges in evaluation	—
16	DOES NOT LICENSE	Judge reliability for tasks not tested	—
17	VERIFICATION STATE	CONTENT VERIFIED	state: CONTENT_VERIFIED_PRIMARY · verified_by: UNKNOWN
18	LIMITATIONS	Agreement with humans can reflect shared biases, as the authors note	—
19	CONTRADICTORY / SUPPORTING SOURCES	Compare LD06-SL-C19, where the headline is over 80% agreement	—
20	GEOGRAPHY / LANGUAGE	Not checked / English	—
21	DISTRIBUTIONAL LEDGER	Agreement depends on the expertise of the human judges	—
22	R01 / FIELD-MAP LINKS	LD-06, D4b	—
23	WHAT WOULD CHANGE THIS CLAIM?	Evidence of stable high agreement across tasks for newer judges	—
24	PUBLICATION STATUS	ELIGIBLE WITH QUALIFIER	state: ELIGIBLE · qualifiers: QUALIFIED
25	RECORDED DATE	2026-10-15	—

Field 22 references resolved against Field Map v0.2: LD-06 (LIVING_DEBATE); D4b (SUB_AREA)

Cited by Living Debate versions

LD-06 v0.1

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 caccdbb7bc3ce95507e0476f2047018906ce7fe81ad3932946d911962d37ae2e.

Repository-authored inputs (not Drive exports)

SocioLens claim — LD06-SL-C18 v1.0 (v1.0) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-15; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts. Block “ENTRY 18 — LD06-SL-C18” SHA-256 f9e2b7fcd3eaf219c1ccd35f42cd1330ba8cefd7f8f5a0540433c37815937012; fixture fixtures/authored/claims/LD06-SL-C18--v1.0.

Builder decisions (not Drive data)

VERSION = v1.0 — basis: Claim version taken from the repository-authored claim fixture version (SocioLens claim — LD06-SL-C18 v1.0); the first version of a new Claim of the LD-06 debate
CURRENT_POINTER = true — basis: repository-authored claim "SocioLens claim — LD06-SL-C18 v1.0" (Project Lead (Claude, AI), 2026-10-15); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts
FIELD_6_SOURCE_REFERENCE = SRC-SL-BAVARESCO-ET-AL-JUDGE-BENCH-ACL-2025--v1.0 — basis: field 6 of ENTRY 18 — LD06-SL-C18 cites "Bavaresco, Bernardi, Bertolazzi, Elliott, Fernández, Gatt et al., “LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks,” ACL 2025, arXiv:2406.18403"; the only field-6 segment of this ClaimVersion's repository-authored ledger block.
SOURCE_PROVENANCE_PIN = SRC-SL-BAVARESCO-ET-AL-JUDGE-BENCH-ACL-2025--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Bavaresco, Bernardi, Bertolazzi, Elliott, Fernández, Gatt et al., “LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks,” ACL 2025, arXiv:2406.18403".