Claim LD06-SL-C14 (v1.0)
The International AI Safety Report 2026 finds an “evaluation gap”: performance on pre-deployment tests does not reliably predict real-world utility or risk, models “now regularly identify evaluation prompts as tests”, and although 12 companies published or updated frontier safety frameworks in 2025, there are no standardised external audits of their compliance.
Research opportunities this claim motivates
What this claim leaves open for research. Each is a hypothesis about a gap, not a finding.
- Do benchmark scores predict what models achieve in real deployments?The International AI Safety Report 2026 names an evaluation gap between pre-deployment test performance and real-world utility or risk.
- How is external evaluation under the EU AI Code of Practice carried out?The International AI Safety Report found no standardised external audits of companies' safety frameworks.
Provenance and status
- Claim type
- INTERPRETATION (as printed: “INTERPRETATION”)
- Publication status
- ELIGIBLE (INTERPRETATION_LABEL) (as printed: “ELIGIBLE WITH INTERPRETATION LABEL”)
- Verification state
- CONTENT_VERIFIED_PRIMARY (as printed: “CONTENT VERIFIED”; verified by: UNKNOWN)
- Does not license
- New empirical findings (the report synthesizes others’ work)
- Source (field 6, as printed)
- Bengio, Clare, Prunkl, Andriushchenko, Bucknall, Murray et al., “International AI Safety Report 2026,” arXiv:2602.21012, February 2026
- Source type (field 7)
- POLICY_RESEARCH_OR_MONITORING (as printed: “Intergovernmental expert report”)
- Institutional position
- Intergovernmental expert report (advisory panel nominated by over 30 countries and international organisations)
- Maturity (claim evidence / publication maturity)
- RAPIDLY_MATURING (as printed: “RAPIDLY MATURING”)
- Recorded date
- 2026-10-15
- Currentness / correction
- This is the current version. No correction or concern relation is recorded on its Sources.
Version
Current version v1.0, resolved through the explicit current pointer CLAIM--LD06-SL-C14.
Pinned Sources (field 6, version-specific)
- “Bengio, Clare, Prunkl, Andriushchenko, Bucknall, Murray et al., “International AI Safety Report 2026,” arXiv:2602.21012, February 2026” → SRC-SL-INTERNATIONAL-AI-SAFETY-REPORT-2026--v1.0 — Bengio, Clare, Prunkl, Andriushchenko, Bucknall, Murray et al., “International AI Safety Report 2026,” arXiv:2602.21012, February 2026; class POLICY_RESEARCH_OR_MONITORING (explicit public provenance pin)
All 25 canonical Claim fields (raw values and normalized values)
| # | Canonical field | Canonical value (verbatim) | Normalized (Claim Value Vocabulary v1.0.2) |
|---|---|---|---|
| 1 | CLAIM ID | LD06-SL-C14 | — |
| 2 | PUBLIC-FACING CLAIM | The International AI Safety Report 2026 finds an “evaluation gap”: performance on pre-deployment tests does not reliably predict real-world utility or risk, models “now regularly identify evaluation prompts as tests”, and although 12 companies published or updated frontier safety frameworks in 2025, there are no standardised external audits of their compliance. | — |
| 3 | CLAIM TYPE | INTERPRETATION | base_types: INTERPRETATION |
| 4 | RESEARCH STRAND | C — Evaluation in governance, audits and red-teaming | — |
| 5 | MATURITY | RAPIDLY MATURING | states: RAPIDLY_MATURING |
| 6 | SOURCE | Bengio, Clare, Prunkl, Andriushchenko, Bucknall, Murray et al., “International AI Safety Report 2026,” arXiv:2602.21012, February 2026 | source_ids: SRC-SL-INTERNATIONAL-AI-SAFETY-REPORT-2026 |
| 7 | SOURCE TYPE | Intergovernmental expert report | classes: POLICY_RESEARCH_OR_MONITORING |
| 8 | INSTITUTIONAL POSITION | Intergovernmental expert report (advisory panel nominated by over 30 countries and international organisations) | — |
| 9 | SCOPE | General-purpose AI, worldwide | — |
| 10 | DESIGN | Expert synthesis of the published literature | — |
| 11 | SAMPLE / DATASET | Not applicable | — |
| 12 | MODEL / TOOL / VERSION | General-purpose AI models to early 2026 | — |
| 13 | COMPARATOR | None | — |
| 14 | MAIN RESULT | Evaluation gap; current methods give unreliable assessments of capabilities and propensities; many capability evaluations outdated, contaminated or narrow; 12 companies’ frameworks; external compliance assessments limited, no standardised external audits; frameworks rarely use explicit quantitative thresholds | — |
| 15 | INFERENTIAL TARGET | The state of evidence on AI evaluation as a basis for governance | — |
| 16 | DOES NOT LICENSE | New empirical findings (the report synthesizes others’ work) | — |
| 17 | VERIFICATION STATE | CONTENT VERIFIED | state: CONTENT_VERIFIED_PRIMARY · verified_by: UNKNOWN |
| 18 | LIMITATIONS | A synthesis; some examples it cites (for instance warmth fine-tuning raising errors by 10–30 points at similar benchmark scores) were not traced to their primary studies and carry no claim here | — |
| 19 | CONTRADICTORY / SUPPORTING SOURCES | Supports positions P4 and P5; compare LD06-SL-C23 and LD06-SL-C24 | — |
| 20 | GEOGRAPHY / LANGUAGE | Global / English | — |
| 21 | DISTRIBUTIONAL LEDGER | Not analysed in this claim | — |
| 22 | R01 / FIELD-MAP LINKS | LD-06, D4b | — |
| 23 | WHAT WOULD CHANGE THIS CLAIM? | Standardised external audits, or validation studies linking evaluations to field outcomes | — |
| 24 | PUBLICATION STATUS | ELIGIBLE WITH INTERPRETATION LABEL | state: ELIGIBLE · qualifiers: INTERPRETATION_LABEL |
| 25 | RECORDED DATE | 2026-10-15 | — |
Field 22 references resolved against Field Map v0.2: LD-06 (LIVING_DEBATE); D4b (SUB_AREA)
Cited by Living Debate versions
Research-of-record provenance
Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 a226ce35692c598e63b49bfa9ca717286e6406125ce0d9fa49fa867e48fad4b5.
Repository-authored inputs (not Drive exports)
- SocioLens claim — LD06-SL-C14 v1.0 (v1.0) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-15; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts. Block “ENTRY 14 — LD06-SL-C14” SHA-256
cfedc955461ceb37ee324faebe56c67c249e7dc942cd63a159f86c33d748b61e; fixturefixtures/authored/claims/LD06-SL-C14--v1.0.
Builder decisions (not Drive data)
- VERSION = v1.0 — basis: Claim version taken from the repository-authored claim fixture version (SocioLens claim — LD06-SL-C14 v1.0); the first version of a new Claim of the LD-06 debate
- CURRENT_POINTER = true — basis: repository-authored claim "SocioLens claim — LD06-SL-C14 v1.0" (Project Lead (Claude, AI), 2026-10-15); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts
- FIELD_6_SOURCE_REFERENCE = SRC-SL-INTERNATIONAL-AI-SAFETY-REPORT-2026--v1.0 — basis: field 6 of ENTRY 14 — LD06-SL-C14 cites "Bengio, Clare, Prunkl, Andriushchenko, Bucknall, Murray et al., “International AI Safety Report 2026,” arXiv:2602.21012, February 2026"; the only field-6 segment of this ClaimVersion's repository-authored ledger block.
- SOURCE_PROVENANCE_PIN = SRC-SL-INTERNATIONAL-AI-SAFETY-REPORT-2026--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Bengio, Clare, Prunkl, Andriushchenko, Bucknall, Murray et al., “International AI Safety Report 2026,” arXiv:2602.21012, February 2026".