Claim LD06-SL-C13 (v1.0)
In interviews with 19 people who use or decided against using AI benchmarks, academics generally saw public benchmarks as suitable for tracking research progress, while in product and policy work benchmarks, even internally built ones, were often found inadequate for substantive decisions.
Research opportunities this claim motivates
What this claim leaves open for research. Each is a hypothesis about a gap, not a finding.
- Do benchmark scores predict what models achieve in real deployments?Product and policy practitioners often found benchmarks inadequate for substantive decisions.
Provenance and status
- Claim type
- ESTIMATE (as printed: “ESTIMATE”)
- Publication status
- ELIGIBLE (QUALIFIED) (as printed: “ELIGIBLE WITH QUALIFIER”)
- Verification state
- CONTENT_VERIFIED_PRIMARY (ABSTRACT_ONLY) (as printed: “CONTENT VERIFIED ABSTRACT”; verified by: UNKNOWN)
- Does not license
- A prevalence estimate of benchmark use or distrust
- Source (field 6, as printed)
- Hardy, Reuel, Jafari Meimandi, Soder, Griffith, Asmar, Koyejo, Bernstein & Kochenderfer, “More than Marketing? On the Information Value of AI Benchmarks for Practitioners,” arXiv:2412.05520, December 2024
- Source type (field 7)
- PREPRINT (as printed: “Preprint”)
- Institutional position
- Independent academic (Stanford, with one author at the policy think tank interface)
- Maturity (claim evidence / publication maturity)
- EMERGING (as printed: “EMERGING”)
- Recorded date
- 2026-10-15
- Currentness / correction
- This is the current version. No correction or concern relation is recorded on its Sources.
Version
Current version v1.0, resolved through the explicit current pointer CLAIM--LD06-SL-C13.
Pinned Sources (field 6, version-specific)
- “Hardy, Reuel, Jafari Meimandi, Soder, Griffith, Asmar, Koyejo, Bernstein & Kochenderfer, “More than Marketing? On the Information Value of AI Benchmarks for Practitioners,” arXiv:2412.05520, December 2024” → SRC-SL-HARDY-ET-AL-MORE-THAN-MARKETING-ARXIV-2412-05520--v1.0 — Hardy, Reuel, Jafari Meimandi, Soder, Griffith, Asmar, Koyejo, Bernstein & Kochenderfer, “More than Marketing? On the Information Value of AI Benchmarks for Practitioners,” arXiv:2412.05520, December 2024; class PREPRINT (explicit public provenance pin)
All 25 canonical Claim fields (raw values and normalized values)
| # | Canonical field | Canonical value (verbatim) | Normalized (Claim Value Vocabulary v1.0.2) |
|---|---|---|---|
| 1 | CLAIM ID | LD06-SL-C13 | — |
| 2 | PUBLIC-FACING CLAIM | In interviews with 19 people who use or decided against using AI benchmarks, academics generally saw public benchmarks as suitable for tracking research progress, while in product and policy work benchmarks, even internally built ones, were often found inadequate for substantive decisions. | — |
| 3 | CLAIM TYPE | ESTIMATE | base_types: ESTIMATE |
| 4 | RESEARCH STRAND | A — Benchmark validity and measurement | — |
| 5 | MATURITY | EMERGING | states: EMERGING |
| 6 | SOURCE | Hardy, Reuel, Jafari Meimandi, Soder, Griffith, Asmar, Koyejo, Bernstein & Kochenderfer, “More than Marketing? On the Information Value of AI Benchmarks for Practitioners,” arXiv:2412.05520, December 2024 | source_ids: SRC-SL-HARDY-ET-AL-MORE-THAN-MARKETING-ARXIV-2412-05520 |
| 7 | SOURCE TYPE | Preprint | classes: PREPRINT |
| 8 | INSTITUTIONAL POSITION | Independent academic (Stanford, with one author at the policy think tank interface) | — |
| 9 | SCOPE | Practitioners in academia, product development and policy | — |
| 10 | DESIGN | Qualitative interview study | — |
| 11 | SAMPLE / DATASET | 19 interviewees | — |
| 12 | MODEL / TOOL / VERSION | Not applicable | — |
| 13 | COMPARATOR | Between academic, product and policy settings | — |
| 14 | MAIN RESULT | Benchmarks used as a signal of relative performance; in product and policy, often inadequate for decisions because goals were neither well defined nor reflective of real-world use | — |
| 15 | INFERENTIAL TARGET | Whether benchmark scores carry over to the decisions people make with them | — |
| 16 | DOES NOT LICENSE | A prevalence estimate of benchmark use or distrust | — |
| 17 | VERIFICATION STATE | CONTENT VERIFIED ABSTRACT | state: CONTENT_VERIFIED_PRIMARY · qualifiers: ABSTRACT_ONLY · verified_by: UNKNOWN |
| 18 | LIMITATIONS | Read in abstract and first page only; small qualitative sample | — |
| 19 | CONTRADICTORY / SUPPORTING SOURCES | Compare the evaluation gap named in LD06-SL-C14 | — |
| 20 | GEOGRAPHY / LANGUAGE | Not checked / English | — |
| 21 | DISTRIBUTIONAL LEDGER | Not analysed in this claim | — |
| 22 | R01 / FIELD-MAP LINKS | LD-06, D4b | — |
| 23 | WHAT WOULD CHANGE THIS CLAIM? | Larger surveys of procurement and policy users | — |
| 24 | PUBLICATION STATUS | ELIGIBLE WITH QUALIFIER | state: ELIGIBLE · qualifiers: QUALIFIED |
| 25 | RECORDED DATE | 2026-10-15 | — |
Field 22 references resolved against Field Map v0.2: LD-06 (LIVING_DEBATE); D4b (SUB_AREA)
Cited by Living Debate versions
Research-of-record provenance
Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 f8c385164c90fb01c5c8a212a3b9748679726d59d3954d21311e738703d0eb8a.
Repository-authored inputs (not Drive exports)
- SocioLens claim — LD06-SL-C13 v1.0 (v1.0) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-15; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts. Block “ENTRY 13 — LD06-SL-C13” SHA-256
7e6712e470f9064cc597511763903c97fdc20f4f4dfe4a79faf41cda8f7c6aa2; fixturefixtures/authored/claims/LD06-SL-C13--v1.0.
Builder decisions (not Drive data)
- VERSION = v1.0 — basis: Claim version taken from the repository-authored claim fixture version (SocioLens claim — LD06-SL-C13 v1.0); the first version of a new Claim of the LD-06 debate
- CURRENT_POINTER = true — basis: repository-authored claim "SocioLens claim — LD06-SL-C13 v1.0" (Project Lead (Claude, AI), 2026-10-15); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts
- FIELD_6_SOURCE_REFERENCE = SRC-SL-HARDY-ET-AL-MORE-THAN-MARKETING-ARXIV-2412-05520--v1.0 — basis: field 6 of ENTRY 13 — LD06-SL-C13 cites "Hardy, Reuel, Jafari Meimandi, Soder, Griffith, Asmar, Koyejo, Bernstein & Kochenderfer, “More than Marketing? On the Information Value of AI Benchmarks for Practitioners,” arXiv:2412.05520, December 2024"; the only field-6 segment of this ClaimVersion's repository-authored ledger block.
- SOURCE_PROVENANCE_PIN = SRC-SL-HARDY-ET-AL-MORE-THAN-MARKETING-ARXIV-2412-05520--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Hardy, Reuel, Jafari Meimandi, Soder, Griffith, Asmar, Koyejo, Bernstein & Kochenderfer, “More than Marketing? On the Information Value of AI Benchmarks for Practitioners,” arXiv:2412.05520, December 2024".