Claim LD07-HQ-C02 (v1.0)

In a large 2025 preprint spanning 37 annotation tasks from 21 published studies, plausible choices of LLM and annotation configuration produced incorrect inferential conclusions in about one third of tested hypotheses for state-of-the-art models, with higher rates for smaller models and near conventional significance thresholds.

Provenance and status

Claim type: ESTIMATE (as printed: “ESTIMATE”)
Publication status: ELIGIBLE (PREPRINT, CORPUS) (as printed: “ELIGIBLE WITH PREPRINT/CORPUS QUALIFIER”)
Verification state: CONTENT_VERIFIED_PRIMARY (PREPRINT, SOURCE_SPECIFIC) (as printed: “CLEARED WITH QUALIFIER”; verified by: UNKNOWN)
Does not license: “31% of published findings are wrong”; future-model extrapolation
Source (field 6, as printed): Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber & Hovy, “Large Language Model Hacking,” arXiv:2509.08825v2
Source type (field 7): PREPRINT (as printed: “Preprint”)
Institutional position: Independent academic
Maturity (claim evidence / publication maturity): RAPIDLY_MATURING (as printed: “RAPIDLY MATURING”)
Recorded date: 2026-09-25
Currentness / correction: This is the current version. No correction or concern relation is recorded on its Sources.

Version

Pinned version v1.0 (entry LD07-HQ-C02--v1.0) — this page never floats to newer evidence.

This pinned version is also the current version.

Versions:v1.0 · current

Pinned Sources (field 6, version-specific)

“Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber & Hovy, “Large Language Model Hacking,” arXiv:2509.08825v2” → SRC-LD07-BAUMANN-LLM-HACKING-2025--v1.0 — Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber & Hovy, “Large Language Model Hacking,” arXiv:2509.08825v2; class PREPRINT (explicit public provenance pin)

Read the originals

SRC-LD07-BAUMANN-LLM-HACKING-2025: open the original

All 25 canonical Claim fields (raw values and normalized values)

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD07-HQ-C02	—
2	PUBLIC-FACING CLAIM	In a large 2025 preprint spanning 37 annotation tasks from 21 published studies, plausible choices of LLM and annotation configuration produced incorrect inferential conclusions in about one third of tested hypotheses for state-of-the-art models, with higher rates for smaller models and near conventional significance thresholds.	—
3	CLAIM TYPE	ESTIMATE	base_types: ESTIMATE
4	RESEARCH STRAND	A	—
5	MATURITY	RAPIDLY MATURING	states: RAPIDLY_MATURING
6	SOURCE	Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber & Hovy, “Large Language Model Hacking,” arXiv:2509.08825v2	source_ids: SRC-LD07-BAUMANN-LLM-HACKING-2025
7	SOURCE TYPE	Preprint	classes: PREPRINT
8	INSTITUTIONAL POSITION	Independent academic	—
9	SCOPE	37 CSS annotation tasks from 21 studies	—
10	DESIGN	Large multi-study re-annotation/configuration audit	—
11	SAMPLE / DATASET	~13M labels; 2,361 realistic hypotheses	—
12	MODEL / TOOL / VERSION	18 LLMs, 2025-era model set	—
13	COMPARATOR	Study-specific ground-truth/reference annotation and hypothesis conclusions	—
14	MAIN RESULT	~31% incorrect conclusions for SOTA; ~50% smaller models; risk higher near p=.05	—
15	INFERENTIAL TARGET	Preservation of statistical conclusions under annotation configuration choice	—
16	DOES NOT LICENSE	“31% of published findings are wrong”; future-model extrapolation	—
17	VERIFICATION STATE	CLEARED WITH QUALIFIER	state: CONTENT_VERIFIED_PRIMARY · qualifiers: PREPRINT, SOURCE_SPECIFIC · verified_by: UNKNOWN
18	LIMITATIONS	Preprint; corpus/model-generation specific; configuration space defined by authors	—
19	CONTRADICTORY / SUPPORTING SOURCES	Egami; Yang; Li; Nakamura qualifies task-level agreement interpretation	—
20	GEOGRAPHY / LANGUAGE	Mostly English CSS	—
21	DISTRIBUTIONAL LEDGER	No broad subgroup guarantee	—
22	R01 / FIELD-MAP LINKS	T1, D9, R-1, R-2	—
23	WHAT WOULD CHANGE THIS CLAIM?	Peer-reviewed replication with current models and substantially lower error	—
24	PUBLICATION STATUS	ELIGIBLE WITH PREPRINT/CORPUS QUALIFIER	state: ELIGIBLE · qualifiers: PREPRINT, CORPUS
25	RECORDED DATE	2026-09-25	—

Field 22 references resolved against Field Map v0.2: T1 (TENSION); D9 (DOMAIN); R-1 (LENS); R-2 (LENS)

Cited by Living Debate versions

LD-07 v0.1

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 305c71041ba2d8d1231b52858ca7825ff07748b3081aa34b3c739dcfce4d0145.

Frozen Drive-derived inputs

R03 — LD-07 Canonical Claim Ledger v1.0 — LOCKED (v1.0) — Drive ID 1xVz_LrQAA8EA1tPhVWIrDgOUU3Zidlg9; revision: NOT_EXPOSED_BY_DRIVE_TEXT_EXPORT; Drive modified 2026-09-29T04:31:14.120Z; block “ENTRY 02 — LD07-HQ-C02” SHA-256 d7b3ad85094278da9806fd9b7ff86c521940f9960c462480c6b32801419f55b1; SocioLens — Programme Status & Authority Register v0.6 status “CURRENT.”; frozen fixture fixtures/frozen/r03-ld07-claim-ledger-v1.0-canonical.

Builder decisions (not Drive data)

VERSION = v1.0 — basis: Claim version taken from the canonical ledger artifact version (R03 — LD-07 Canonical Claim Ledger v1.0); Implementation Architecture Selection §5
CURRENT_POINTER = true — basis: SocioLens — Programme Status & Authority Register v0.6 status for "R03 — LD-07 Canonical Claim Ledger v1.0 — LOCKED": "CURRENT."
FIELD_17_CONTEXT_QUALIFIERS = PREPRINT, SOURCE_SPECIFIC — basis: R03 — HQ Cross-Batch Synthesis & Adjudication (v0.1, Drive 1V-HOVaPFwF8BTf60-LGPsJXf59V3kRlQ)
FIELD_6_SOURCE_REFERENCE = SRC-LD07-BAUMANN-LLM-HACKING-2025--v1.0 — basis: field 6 of ENTRY 02 — LD07-HQ-C02 cites "Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber & Hovy, “Large Language Model Hacking,” arXiv:2509.08825v2"; the frozen field-6 segment of this ClaimVersion's ledger block; Drive-derived identity authority (Issue #39 Source inventory): Batch 01 §2 ("PREPRINT", v2); ledger C02; KEY EVIDENCE.
SOURCE_PROVENANCE_PIN = SRC-LD07-BAUMANN-LLM-HACKING-2025--v1.0 — basis: Issue #5 proof rendering (Architecture Proof Protocol v0.1 AP-08 / Amendment v1.2 §6): explicit version-pinned public provenance of this ClaimVersion — the pinned field-6 Source record "Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber & Hovy, “Large Language Model Hacking,” arXiv:2509.08825v2"; Drive-derived identity authority (Issue #39 Source inventory): Batch 01 §2 ("PREPRINT", v2); ledger C02; KEY EVIDENCE.