Claim LD06-SL-C15 (v1.0)
An audit of Chatbot Arena’s data from January 2024 to April 2025 found undisclosed private testing (27 private Meta variants before the Llama 4 release), data skewed towards proprietary providers (Google and OpenAI about 20% each), and 205 of 243 public models silently deprecated, and concluded that rankings partly reflect adaptation to the Arena rather than general model quality.
Provenance and status
- Claim type
- ESTIMATE (as printed: “ESTIMATE”)
- Publication status
- ELIGIBLE (SOURCE_POSITION) (as printed: “ELIGIBLE WITH SOURCE-POSITION QUALIFIER”)
- Verification state
- CONTENT_VERIFIED_PRIMARY (PREPRINT) (as printed: “CONTENT VERIFIED PREPRINT”; verified by: UNKNOWN)
- Does not license
- That any provider broke the Arena’s rules; the size of the private-testing effect on public scores, which the platform disputes
- Source (field 6, as printed)
- Singh, Nan, Wang, D’Souza, Kapoor, Üstün, Koyejo, Deng, Longpre, Smith, Ermis, Fadaee & Hooker, “The Leaderboard Illusion,” arXiv:2504.20879 (v2), May 2025
- Source type (field 7)
- PREPRINT (as printed: “Preprint”)
- Institutional position
- AI developer-led (Cohere, an Arena participant) with academic co-authors
- Maturity (claim evidence / publication maturity)
- CONTESTED (as printed: “CONTESTED”)
- Recorded date
- 2026-10-15
- Currentness / correction
- This is the current version. No correction or concern relation is recorded on its Sources.
Version
Current version v1.0, resolved through the explicit current pointer CLAIM--LD06-SL-C15.
Pinned Sources (field 6, version-specific)
- “Singh, Nan, Wang, D’Souza, Kapoor, Üstün, Koyejo, Deng, Longpre, Smith, Ermis, Fadaee & Hooker, “The Leaderboard Illusion,” arXiv:2504.20879 (v2), May 2025” → SRC-SL-SINGH-ET-AL-LEADERBOARD-ILLUSION-ARXIV-2504-20879--v1.0 — Singh, Nan, Wang, D’Souza, Kapoor, Üstün, Koyejo, Deng, Longpre, Smith, Ermis, Fadaee & Hooker, “The Leaderboard Illusion,” arXiv:2504.20879 (v2), May 2025; class PREPRINT (explicit public provenance pin)
All 25 canonical Claim fields (raw values and normalized values)
| # | Canonical field | Canonical value (verbatim) | Normalized (Claim Value Vocabulary v1.0.2) |
|---|---|---|---|
| 1 | CLAIM ID | LD06-SL-C15 | — |
| 2 | PUBLIC-FACING CLAIM | An audit of Chatbot Arena’s data from January 2024 to April 2025 found undisclosed private testing (27 private Meta variants before the Llama 4 release), data skewed towards proprietary providers (Google and OpenAI about 20% each), and 205 of 243 public models silently deprecated, and concluded that rankings partly reflect adaptation to the Arena rather than general model quality. | — |
| 3 | CLAIM TYPE | ESTIMATE | base_types: ESTIMATE |
| 4 | RESEARCH STRAND | B — Leaderboards, judges and disclosure | — |
| 5 | MATURITY | CONTESTED | states: CONTESTED |
| 6 | SOURCE | Singh, Nan, Wang, D’Souza, Kapoor, Üstün, Koyejo, Deng, Longpre, Smith, Ermis, Fadaee & Hooker, “The Leaderboard Illusion,” arXiv:2504.20879 (v2), May 2025 | source_ids: SRC-SL-SINGH-ET-AL-LEADERBOARD-ILLUSION-ARXIV-2504-20879 |
| 7 | SOURCE TYPE | Preprint | classes: PREPRINT |
| 8 | INSTITUTIONAL POSITION | AI developer-led (Cohere, an Arena participant) with academic co-authors | — |
| 9 | SCOPE | Chatbot Arena, January 2024 to April 2025 | — |
| 10 | DESIGN | Analysis of battle data, scraped records and simulations, plus the authors’ own private-variant experiment | — |
| 11 | SAMPLE / DATASET | 2 million battles, 42 providers, 243 models | — |
| 12 | MODEL / TOOL / VERSION | Models ranked on the Arena | — |
| 13 | COMPARATOR | Simulated leaderboards; training with and without Arena data | — |
| 14 | MAIN RESULT | 27 private Meta variants; Google 19.2% and OpenAI 20.4% of Arena data; 41 fully open-source models received an estimated 8.9% of data (v2); 205 of 243 models silently deprecated; in the authors’ own fine-tuning experiment, adding Arena data raised win rates on ArenaHard from 23.5% to 49.9% | — |
| 15 | INFERENTIAL TARGET | Whether a preference leaderboard ranks general model quality | — |
| 16 | DOES NOT LICENSE | That any provider broke the Arena’s rules; the size of the private-testing effect on public scores, which the platform disputes | — |
| 17 | VERIFICATION STATE | CONTENT VERIFIED PREPRINT | state: CONTENT_VERIFIED_PRIMARY · qualifiers: PREPRINT · verified_by: UNKNOWN |
| 18 | LIMITATIONS | Authors compete on the Arena, and Cohere’s own enquiry prompted the work; the private-testing effect rests partly on simulation; the platform’s response disputes several figures (LD06-SL-C16) | — |
| 19 | CONTRADICTORY / SUPPORTING SOURCES | Disputed in part by LD06-SL-C16 | — |
| 20 | GEOGRAPHY / LANGUAGE | Global platform / English | — |
| 21 | DISTRIBUTIONAL LEDGER | Open-weight and open-source models received less data and were deprecated more often | — |
| 22 | R01 / FIELD-MAP LINKS | LD-06, D4b | — |
| 23 | WHAT WOULD CHANGE THIS CLAIM? | Platform data showing that private testing does not move public ranks | — |
| 24 | PUBLICATION STATUS | ELIGIBLE WITH SOURCE-POSITION QUALIFIER | state: ELIGIBLE · qualifiers: SOURCE_POSITION |
| 25 | RECORDED DATE | 2026-10-15 | — |
Field 22 references resolved against Field Map v0.2: LD-06 (LIVING_DEBATE); D4b (SUB_AREA)
Cited by Living Debate versions
Research-of-record provenance
Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 533127625564ce3cf2885f4120d0fac508579accec20925a614c903ec855e9ff.
Repository-authored inputs (not Drive exports)
- SocioLens claim — LD06-SL-C15 v1.0 (v1.0) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-15; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts. Block “ENTRY 15 — LD06-SL-C15” SHA-256
9eb78693321ac2ee3347b541e1139f40077e79112d77755951ff2e19def963e8; fixturefixtures/authored/claims/LD06-SL-C15--v1.0.
Builder decisions (not Drive data)
- VERSION = v1.0 — basis: Claim version taken from the repository-authored claim fixture version (SocioLens claim — LD06-SL-C15 v1.0); the first version of a new Claim of the LD-06 debate
- CURRENT_POINTER = true — basis: repository-authored claim "SocioLens claim — LD06-SL-C15 v1.0" (Project Lead (Claude, AI), 2026-10-15); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts
- FIELD_6_SOURCE_REFERENCE = SRC-SL-SINGH-ET-AL-LEADERBOARD-ILLUSION-ARXIV-2504-20879--v1.0 — basis: field 6 of ENTRY 15 — LD06-SL-C15 cites "Singh, Nan, Wang, D’Souza, Kapoor, Üstün, Koyejo, Deng, Longpre, Smith, Ermis, Fadaee & Hooker, “The Leaderboard Illusion,” arXiv:2504.20879 (v2), May 2025"; the only field-6 segment of this ClaimVersion's repository-authored ledger block.
- SOURCE_PROVENANCE_PIN = SRC-SL-SINGH-ET-AL-LEADERBOARD-ILLUSION-ARXIV-2504-20879--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Singh, Nan, Wang, D’Souza, Kapoor, Üstün, Koyejo, Deng, Longpre, Smith, Ermis, Fadaee & Hooker, “The Leaderboard Illusion,” arXiv:2504.20879 (v2), May 2025".