Claim LD07-HQ-C13 (v1.0)

LLM judges can correlate with human evaluations while exhibiting systematic evaluator-specific biases, including position bias and self-preference; task-specific validation is therefore required.

Provenance and status

Claim type: FACT + INTERPRETATION (as printed: “FACT + INTERPRETATION”)
Publication status: ELIGIBLE (as printed: “ELIGIBLE”)
Verification state: CONTENT_VERIFIED_PRIMARY (as printed: “CLEARED”; verified by: UNKNOWN)
Does not license: LLM judge as sole unvalidated outcome for social constructs
Source (field 6, as printed): Shi et al. IJCNLP-AACL 2025; Panickssery, Bowman & Feng NeurIPS 2024; Calderon et al. ACL 2025
Source type (field 7): PEER_REVIEWED_CONFERENCE_PAPER (as printed: “Peer-reviewed conference studies”)
Institutional position: Independent academic / mixed academic-industry author affiliations
Maturity (claim evidence / publication maturity): RAPIDLY_MATURING (as printed: “RAPIDLY MATURING”)
Recorded date: 2026-09-25
Currentness / correction: This is the current version. No correction or concern relation is recorded on its Sources.

Version

Pinned version v1.0 (entry LD07-HQ-C13--v1.0) — this page never floats to newer evidence.

This pinned version is also the current version.

Versions:v1.0 · current

Pinned Sources (field 6, version-specific)

“Shi et al. IJCNLP-AACL 2025” → SRC-LD07-SHI-JUDGING-THE-JUDGES-2025--v1.0 — Shi et al., “Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge,” IJCNLP-AACL 2025; class PEER_REVIEWED_CONFERENCE_PAPER (explicit public provenance pin)
“Panickssery, Bowman & Feng NeurIPS 2024” → SRC-LD07-PANICKSSERY-SELF-PREFERENCE-2024--v1.0 — Panickssery, Bowman & Feng, “LLM Evaluators Recognize and Favor Their Own Generations,” NeurIPS 2024; class PEER_REVIEWED_CONFERENCE_PAPER (explicit public provenance pin)
“Calderon et al. ACL 2025” → SRC-LD07-CALDERON-ALTERNATIVE-ANNOTATOR-2025--v1.0 — Calderon, Reichart & Dror, “The Alternative Annotator Test for LLM-as-a-Judge,” ACL 2025; class PEER_REVIEWED_CONFERENCE_PAPER (explicit public provenance pin)

Read the originals

SRC-LD07-SHI-JUDGING-THE-JUDGES-2025: open the original
SRC-LD07-PANICKSSERY-SELF-PREFERENCE-2024: open the original
SRC-LD07-CALDERON-ALTERNATIVE-ANNOTATOR-2025: open the original

All 25 canonical Claim fields (raw values and normalized values)

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD07-HQ-C13	—
2	PUBLIC-FACING CLAIM	LLM judges can correlate with human evaluations while exhibiting systematic evaluator-specific biases, including position bias and self-preference; task-specific validation is therefore required.	—
3	CLAIM TYPE	FACT + INTERPRETATION	base_types: FACT, INTERPRETATION
4	RESEARCH STRAND	B	—
5	MATURITY	RAPIDLY MATURING	states: RAPIDLY_MATURING
6	SOURCE	Shi et al. IJCNLP-AACL 2025; Panickssery, Bowman & Feng NeurIPS 2024; Calderon et al. ACL 2025	source_ids: SRC-LD07-SHI-JUDGING-THE-JUDGES-2025, SRC-LD07-PANICKSSERY-SELF-PREFERENCE-2024, SRC-LD07-CALDERON-ALTERNATIVE-ANNOTATOR-2025
7	SOURCE TYPE	Peer-reviewed conference studies	classes: PEER_REVIEWED_CONFERENCE_PAPER
8	INSTITUTIONAL POSITION	Independent academic / mixed academic-industry author affiliations	—
9	SCOPE	LLM-as-judge NLP evaluation tasks	—
10	DESIGN	Large multi-judge benchmark audits; self-recognition experiments; replacement test	—
11	SAMPLE / DATASET	Shi >150,000 evaluations, 15 judges, MTBench/DevBench; Panickssery controlled recognition/preference tests	—
12	MODEL / TOOL / VERSION	Multiple families incl GPT-4/Llama 2 and later judges	—
13	COMPARATOR	Human preferences; swapped presentation; other model outputs	—
14	MAIN RESULT	Position affects judgments; self-recognition contributes to self-preference; some judge replacement can pass task-specific tests	—
15	INFERENTIAL TARGET	Validity/reliability of model-based evaluation	—
16	DOES NOT LICENSE	LLM judge as sole unvalidated outcome for social constructs	—
17	VERIFICATION STATE	CLEARED	state: CONTENT_VERIFIED_PRIMARY · verified_by: UNKNOWN
18	LIMITATIONS	Mostly engineering/NLP constructs; social-science construct validation sparse	—
19	CONTRADICTORY / SUPPORTING SOURCES	2026 judge studies may update magnitude	—
20	GEOGRAPHY / LANGUAGE	Mostly English	—
21	DISTRIBUTIONAL LEDGER	Language/group-specific judge bias remains under-studied	—
22	R01 / FIELD-MAP LINKS	D4b, D9, R-1	—
23	WHAT WOULD CHANGE THIS CLAIM?	Construct-level validation showing biases immaterial under specified protocol	—
24	PUBLICATION STATUS	ELIGIBLE	state: ELIGIBLE
25	RECORDED DATE	2026-09-25	—

Field 22 references resolved against Field Map v0.2: D4b (SUB_AREA); D9 (DOMAIN); R-1 (LENS)

Cited by Living Debate versions

LD-07 v0.1

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 3bb4566dc3b52d50595581cae1788566ad1986f635b7301b48593b7af4e516f9.

Frozen Drive-derived inputs

R03 — LD-07 Canonical Claim Ledger v1.0 — LOCKED (v1.0) — Drive ID 1xVz_LrQAA8EA1tPhVWIrDgOUU3Zidlg9; revision: NOT_EXPOSED_BY_DRIVE_TEXT_EXPORT; Drive modified 2026-09-29T04:31:14.120Z; block “ENTRY 13 — LD07-HQ-C13” SHA-256 3ff0bb8b809271926087a1c2088fd36c90d851d10d1291d42390397c0f3a9d6a; SocioLens — Programme Status & Authority Register v0.6 status “CURRENT.”; frozen fixture fixtures/frozen/r03-ld07-claim-ledger-v1.0-canonical.

Builder decisions (not Drive data)

VERSION = v1.0 — basis: Claim version taken from the canonical ledger artifact version (R03 — LD-07 Canonical Claim Ledger v1.0); Implementation Architecture Selection §5
CURRENT_POINTER = true — basis: SocioLens — Programme Status & Authority Register v0.6 status for "R03 — LD-07 Canonical Claim Ledger v1.0 — LOCKED": "CURRENT."
FIELD_6_SOURCE_REFERENCE = SRC-LD07-SHI-JUDGING-THE-JUDGES-2025--v1.0 — basis: field 6 of ENTRY 13 — LD07-HQ-C13 cites "Shi et al. IJCNLP-AACL 2025"; the frozen field-6 segment of this ClaimVersion's ledger block; Drive-derived identity authority (Issue #39 Source inventory): Batch 04 §4; ledger C13; KEY EVIDENCE.
FIELD_6_SOURCE_REFERENCE = SRC-LD07-PANICKSSERY-SELF-PREFERENCE-2024--v1.0 — basis: field 6 of ENTRY 13 — LD07-HQ-C13 cites "Panickssery, Bowman & Feng NeurIPS 2024"; the frozen field-6 segment of this ClaimVersion's ledger block; Drive-derived identity authority (Issue #39 Source inventory): Batch 04 §5; ledger C13; KEY EVIDENCE.
FIELD_6_SOURCE_REFERENCE = SRC-LD07-CALDERON-ALTERNATIVE-ANNOTATOR-2025--v1.0 — basis: field 6 of ENTRY 13 — LD07-HQ-C13 cites "Calderon et al. ACL 2025"; the frozen field-6 segment of this ClaimVersion's ledger block; Drive-derived identity authority (Issue #39 Source inventory): Batch 04 §3; ledger C03/C13; KEY EVIDENCE.
SOURCE_PROVENANCE_PIN = SRC-LD07-SHI-JUDGING-THE-JUDGES-2025--v1.0 — basis: Issue #5 proof rendering (Architecture Proof Protocol v0.1 AP-08 / Amendment v1.2 §6): explicit version-pinned public provenance of this ClaimVersion — the pinned field-6 Source record "Shi et al. IJCNLP-AACL 2025"; Drive-derived identity authority (Issue #39 Source inventory): Batch 04 §4; ledger C13; KEY EVIDENCE.
SOURCE_PROVENANCE_PIN = SRC-LD07-PANICKSSERY-SELF-PREFERENCE-2024--v1.0 — basis: Issue #5 proof rendering (Architecture Proof Protocol v0.1 AP-08 / Amendment v1.2 §6): explicit version-pinned public provenance of this ClaimVersion — the pinned field-6 Source record "Panickssery, Bowman & Feng NeurIPS 2024"; Drive-derived identity authority (Issue #39 Source inventory): Batch 04 §5; ledger C13; KEY EVIDENCE.
SOURCE_PROVENANCE_PIN = SRC-LD07-CALDERON-ALTERNATIVE-ANNOTATOR-2025--v1.0 — basis: Issue #5 proof rendering (Architecture Proof Protocol v0.1 AP-08 / Amendment v1.2 §6): explicit version-pinned public provenance of this ClaimVersion — the pinned field-6 Source record "Calderon et al. ACL 2025"; Drive-derived identity authority (Issue #39 Source inventory): Batch 04 §3; ledger C03/C13; KEY EVIDENCE.