Research opportunity · The evaluation apparatus: Can benchmarks, leaderboards and audits carry the weight placed on them?
How faithfully do indices and policy documents relay AI evaluation results?
Question
When annual indices, industry reports and government or procurement documents cite benchmark and evaluation figures, what share come from the developers themselves, what share are independently measured, and how often is the figure, its unit or its scope changed between the primary source and the relay?
Institutional practice unknownMaster's thesisLow costNo study found
Why it matters
Most readers meet evaluation results second-hand. The 2026 AI Index says it assumes company-reported results are accurate, and it presented counts of flagged items from an audit as benchmark-wide invalid rates. One case shows the risk, not its size. A systematic trace would show whether relay error and reliance on self-reports are rare slips or a structural feature of how evaluation numbers travel (P2).
The debate lists this among what we still do not know: How often do indices, procurers and regulators rely on developer-reported scores, and how often do relays distort them?
Evidence that motivates it
- The 2026 AI Index reports that a review found invalid-question rates in nine benchmarks “ranging from 2% on MMLU Math to 42% on GSM8K”, citing Truong et al. (2025); in that paper’s figure, 2 and 42 are the numbers of confirmed invalid questions among the 50 that a statistical screen flagged in each benchmark (“2% corresponds to one question”, so each question is 2 points on that scale), and the paper puts GSM8K’s benchmark-wide share of invalid questions at about 5%, citing Vendrow et al., while its own Platinum-based labelling marks 88 of 997 (8.8%) invalid.
The AI Index 2026 relayed counts of invalid items among 50 flagged per benchmark as invalid-question rates of 2% to 42%. - The 2026 AI Index states that the benchmark scores in its technical-performance chapter come from leaderboards, public repositories and company disclosures, including papers, blog posts and product releases, and that it “assumes that company-reported results are accurate”, while its introduction says that “independent testing does not always confirm what developers report”.
The same index states that it assumes company-reported results are accurate. - The 2025 Foundation Model Transparency Index found that the average transparency score of major AI developers fell from 58 in 2024 to 40 out of 100, and that no company adequately disclosed how far its training data overlap with the evaluations it is tested on; the authors note that changed indicators and companies may contribute to the decline.
Developer transparency, including on evaluation methods and train-test overlap, fell between the 2024 and 2025 transparency indices.
A possible design
Draw a sample of documents from three genres (annual indices and industry reports; government strategies and regulator guidance; public procurement documents) for 2023-2026. Extract every benchmark or evaluation figure, trace it to its primary source, and code: developer-reported or independent; whether the number, its unit (share, count, points), its denominator and its scope match the source; and whether limitations stated in the source survive. Two coders with a reliability check; report error rates by genre and by source type.
Data needed
Public documents only; access to primary papers, model cards and leaderboards as of the relay date (archived copies where they have changed).
Feasibility
Master's thesis. Low; public documents and coder time. Leaderboards change over time, so the trace must use archived versions from the relay date. Results concerning named organizations should be shared with them before publication.
Has this been done?
The search found no study that does the core design (first-pass screen of 15 October 2026; not yet checked in depth).
No study was found that samples secondary documents, traces each benchmark or evaluation figure to its primary source and codes both whether it is developer-reported and whether it was relayed accurately. Related work audits academic papers for evaluating outdated models, proposes provenance labels for scores, or compares developers' self-reported scores with independent runs on one benchmark.
How strong this evidence is
- Who searched and checked: .
- What was read: 4 of the 4 nearest works below were judged from an abstract, summary or search listing only, and 1 has details that could not be confirmed; each is marked. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search (3 queries) and one arXiv query on the date shown; the general literature on citation and quotation errors was not searched, and HELM or Open LLM Leaderboard discrepancy studies were not looked for directly. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Gringras & Salahshoor, "Frontier Lag: A Bibliometric Audit of Capability Misrepresentation in Academic AI Evaluation," arXiv:2605.04135 (2026) Judged from an abstract, summary or search listing only
Abstract read. Measures how far evaluated models lag the frontier in academic papers; the claim that results spread to policy was seen in search results only. - Ghosh, Reuel, Chim et al., "Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting," arXiv:2606.09809 (2026) Judged from an abstract, summary or search listing only
Abstract read. Labels scores as first-party, third-party or collaborative; reporting infrastructure, not a relay audit. - Epoch AI, "Self-reported GPQA Diamond scores vs. independent evaluations," data insight (date not confirmed) Judged from an abstract, summary or search listing onlySome details not confirmed
Judged from search listings; not read directly (the page was blocked). Developer-versus-independent comparison for one benchmark. - Wang, Bianchi, Zhu, Nie, Kwon, Dhingra & Zou, "Automated Benchmark Auditing for AI Agents and Large Language Models," arXiv:2605.26079 (2026) Judged from an abstract, summary or search listing only
Abstract read. Audits 168 benchmarks for item issues; benchmark validity, not relay accuracy.
Search queries used
self-reported benchmark results discrepancy independent evaluation model developers reproduce scoresaudit of AI Index report benchmark figures provenance developer-reported scores policy documents citation accuracypolicy documents misreport AI benchmark results citation distortion relayed scores government reports studyarXiv: abs:"self-reported" AND abs:benchmark AND abs:scores
Provenance
RO-LD06-02 v1. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-06 debate on 2026-10-15. Not yet reviewed. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.