Claim LD06-SL-C01 (v1.0)

The 2026 AI Index reports that a review found invalid-question rates in nine benchmarks “ranging from 2% on MMLU Math to 42% on GSM8K”, citing Truong et al. (2025); in that paper’s figure, 2 and 42 are the numbers of confirmed invalid questions among the 50 that a statistical screen flagged in each benchmark (“2% corresponds to one question”, so each question is 2 points on that scale), and the paper puts GSM8K’s benchmark-wide share of invalid questions at about 5%, citing Vendrow et al., while its own Platinum-based labelling marks 88 of 997 (8.8%) invalid.

Provenance and status

Claim type: FACT (as printed: “FACT”)
Publication status: ELIGIBLE (SOURCE_POSITION) (as printed: “ELIGIBLE WITH SOURCE-POSITION QUALIFIER”)
Verification state: CONTENT_VERIFIED_PRIMARY (as printed: “CONTENT VERIFIED”; verified by: UNKNOWN)
Does not license: A claim about how the figure came to be misreported, or that other figures in the index are wrong
Source (field 6, as printed): Truong, Tu, Hardy, Reuel, Tang, Burapacheep, Perera, Uwakwe, Domingue, Haber & Koyejo, “Fantastic Bugs and Where to Find Them in AI Benchmarks,” arXiv:2511.16842, November 2025; Sajadieh, Fattorini, Perrault, Gil, Parli et al., “The AI Index 2026 Annual Report,” AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, April 2026
Source type (field 7): PREPRINT, INDEPENDENT_RESEARCH_REPORT (as printed: “Preprint + annual index report”)
Institutional position: Independent academic (Stanford); the index is also Stanford-hosted, and two of the paper’s authors are listed among its contributors
Maturity (claim evidence / publication maturity): EMERGING (as printed: “EMERGING”)
Recorded date: 2026-10-15
Currentness / correction: This is the current version. No correction or concern relation is recorded on its Sources.

Version

Pinned version v1.0 (entry LD06-SL-C01--v1.0) — this page never floats to newer evidence.

This pinned version is also the current version.

Versions:v1.0 · current

Pinned Sources (field 6, version-specific)

“Truong, Tu, Hardy, Reuel, Tang, Burapacheep, Perera, Uwakwe, Domingue, Haber & Koyejo, “Fantastic Bugs and Where to Find Them in AI Benchmarks,” arXiv:2511.16842, November 2025” → SRC-SL-TRUONG-ET-AL-ARXIV-2511-16842--v1.0 — Truong, Tu, Hardy, Reuel, Tang, Burapacheep, Perera, Uwakwe, Domingue, Haber & Koyejo, “Fantastic Bugs and Where to Find Them in AI Benchmarks,” arXiv:2511.16842, November 2025; class PREPRINT (explicit public provenance pin)
“Sajadieh, Fattorini, Perrault, Gil, Parli et al., “The AI Index 2026 Annual Report,” AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, April 2026” → SRC-SL-AI-INDEX-2026-ANNUAL-REPORT--v1.0 — Sajadieh, Fattorini, Perrault, Gil, Parli et al., “The AI Index 2026 Annual Report,” AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, April 2026; class INDEPENDENT_RESEARCH_REPORT (explicit public provenance pin)

All 25 canonical Claim fields (raw values and normalized values)

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD06-SL-C01	—
2	PUBLIC-FACING CLAIM	The 2026 AI Index reports that a review found invalid-question rates in nine benchmarks “ranging from 2% on MMLU Math to 42% on GSM8K”, citing Truong et al. (2025); in that paper’s figure, 2 and 42 are the numbers of confirmed invalid questions among the 50 that a statistical screen flagged in each benchmark (“2% corresponds to one question”, so each question is 2 points on that scale), and the paper puts GSM8K’s benchmark-wide share of invalid questions at about 5%, citing Vendrow et al., while its own Platinum-based labelling marks 88 of 997 (8.8%) invalid.	—
3	CLAIM TYPE	FACT	base_types: FACT
4	RESEARCH STRAND	A — Benchmark validity and measurement	—
5	MATURITY	EMERGING	states: EMERGING
6	SOURCE	Truong, Tu, Hardy, Reuel, Tang, Burapacheep, Perera, Uwakwe, Domingue, Haber & Koyejo, “Fantastic Bugs and Where to Find Them in AI Benchmarks,” arXiv:2511.16842, November 2025; Sajadieh, Fattorini, Perrault, Gil, Parli et al., “The AI Index 2026 Annual Report,” AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, April 2026	source_ids: SRC-SL-TRUONG-ET-AL-ARXIV-2511-16842, SRC-SL-AI-INDEX-2026-ANNUAL-REPORT
7	SOURCE TYPE	Preprint + annual index report	classes: PREPRINT, INDEPENDENT_RESEARCH_REPORT
8	INSTITUTIONAL POSITION	Independent academic (Stanford); the index is also Stanford-hosted, and two of the paper’s authors are listed among its contributors	—
9	SCOPE	Nine benchmarks reviewed by experts in the paper; one chart and two passages in the index	—
10	DESIGN	Comparison of the index’s text and chart with the cited paper’s figure, caption and text	—
11	SAMPLE / DATASET	Per benchmark, the 50 items a statistical screen ranked most suspect, reviewed by experts	—
12	MODEL / TOOL / VERSION	40 to 91 language models used as test takers in the paper	—
13	COMPARATOR	The paper’s own figure caption and its benchmark-wide estimate	—
14	MAIN RESULT	Index: “error rates ranging from 2% on MMLU Math to 42% on GSM8K”, chart axis “Precision@50”; paper: 42 of 50 flagged GSM8K items invalid (precision 84%), 2 of 50 for MMLU Math (4%); GSM8K overall about 5% invalid (citing Vendrow et al.); the paper’s own GSM8K-Platinum labelling marks 88 of 997 (8.8%); the index text beside the chart also says “up to 84% precision”	—
15	INFERENTIAL TARGET	How faithfully an evaluation index relays a primary audit	—
16	DOES NOT LICENSE	A claim about how the figure came to be misreported, or that other figures in the index are wrong	—
17	VERIFICATION STATE	CONTENT VERIFIED	state: CONTENT_VERIFIED_PRIMARY · verified_by: UNKNOWN
18	LIMITATIONS	The authors’ own blog post about the paper could not be read, so whether the percentage framing started there is unknown; one search found no erratum as of 15 October 2026; the paper is a preprint (its PDF footer names the NeurIPS 2025 Datasets and Benchmarks Track)	—
19	CONTRADICTORY / SUPPORTING SOURCES	Benchmark-wide error estimates of a few percent: LD06-SL-C03, LD06-SL-C04	—
20	GEOGRAPHY / LANGUAGE	Global / English	—
21	DISTRIBUTIONAL LEDGER	ThaiExam, one of the nine benchmarks, showed cultural biases and linguistic ambiguities	—
22	R01 / FIELD-MAP LINKS	LD-06, D4b	—
23	WHAT WOULD CHANGE THIS CLAIM?	A correction by the index, or evidence that the source itself presented the counts as benchmark-wide rates	—
24	PUBLICATION STATUS	ELIGIBLE WITH SOURCE-POSITION QUALIFIER	state: ELIGIBLE · qualifiers: SOURCE_POSITION
25	RECORDED DATE	2026-10-15	—

Field 22 references resolved against Field Map v0.2: LD-06 (LIVING_DEBATE); D4b (SUB_AREA)

Cited by Living Debate versions

LD-06 v0.1

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 00f6cf2654aeb8b433310f0f796697e8e590e7f955fa15652be36af67da59636.

Repository-authored inputs (not Drive exports)

SocioLens claim — LD06-SL-C01 v1.0 (v1.0) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-15; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts. Block “ENTRY 01 — LD06-SL-C01” SHA-256 8bc0db37ec55d23f3b86a964133747e54b91e73f675e37098f1d572dadaaa9ac; fixture fixtures/authored/claims/LD06-SL-C01--v1.0.

Builder decisions (not Drive data)

VERSION = v1.0 — basis: Claim version taken from the repository-authored claim fixture version (SocioLens claim — LD06-SL-C01 v1.0); the first version of a new Claim of the LD-06 debate
CURRENT_POINTER = true — basis: repository-authored claim "SocioLens claim — LD06-SL-C01 v1.0" (Project Lead (Claude, AI), 2026-10-15); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts
FIELD_6_SOURCE_REFERENCE = SRC-SL-TRUONG-ET-AL-ARXIV-2511-16842--v1.0 — basis: field 6 of ENTRY 01 — LD06-SL-C01 cites "Truong, Tu, Hardy, Reuel, Tang, Burapacheep, Perera, Uwakwe, Domingue, Haber & Koyejo, “Fantastic Bugs and Where to Find Them in AI Benchmarks,” arXiv:2511.16842, November 2025"; the first field-6 segment of this ClaimVersion's repository-authored ledger block.
FIELD_6_SOURCE_REFERENCE = SRC-SL-AI-INDEX-2026-ANNUAL-REPORT--v1.0 — basis: field 6 of ENTRY 01 — LD06-SL-C01 cites "Sajadieh, Fattorini, Perrault, Gil, Parli et al., “The AI Index 2026 Annual Report,” AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, April 2026"; the second field-6 segment of this ClaimVersion's repository-authored ledger block.
SOURCE_PROVENANCE_PIN = SRC-SL-TRUONG-ET-AL-ARXIV-2511-16842--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Truong, Tu, Hardy, Reuel, Tang, Burapacheep, Perera, Uwakwe, Domingue, Haber & Koyejo, “Fantastic Bugs and Where to Find Them in AI Benchmarks,” arXiv:2511.16842, November 2025".
SOURCE_PROVENANCE_PIN = SRC-SL-AI-INDEX-2026-ANNUAL-REPORT--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Sajadieh, Fattorini, Perrault, Gil, Parli et al., “The AI Index 2026 Annual Report,” AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, April 2026".