Claim LD01-SL-C22 (v1.0)
On 11 tasks rewritten with changed premises, such as arithmetic in base 9 or 1-based indexing in code, language models performed above chance but their performance “substantially and consistently degrades” relative to the familiar versions, which the authors attribute partly to overfitting to the default tasks.
Research opportunities this claim motivates
What this claim leaves open for research. Each is a hypothesis about a gap, not a finding.
- Do the failure patterns of 2023–2024 language models persist in reasoning models?Counterfactual task versions degrade performance consistently.
Provenance and status
- Claim type
- ESTIMATE + INTERPRETATION (as printed: “ESTIMATE / INTERPRETATION”)
- Publication status
- ELIGIBLE (QUALIFIED) (as printed: “ELIGIBLE WITH QUALIFIER”)
- Verification state
- CONTENT_VERIFIED_PRIMARY (PREPRINT) (as printed: “CONTENT VERIFIED PREPRINT”; verified by: UNKNOWN)
- Does not license
- A claim that models only recite, or that they reason generally; the authors find evidence of both
- Source (field 6, as printed)
- Wu et al., “Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks,” NAACL 2024
- Source type (field 7)
- PEER_REVIEWED_CONFERENCE_PAPER (as printed: “Peer-reviewed conference paper”)
- Institutional position
- Independent academic (MIT, Boston University)
- Maturity (claim evidence / publication maturity)
- EMERGING (as printed: “EMERGING”)
- Recorded date
- 2026-10-16
- Currentness / correction
- This is the current version. No correction or concern relation is recorded on its Sources.
Version
Current version v1.0, resolved through the explicit current pointer CLAIM--LD01-SL-C22.
Pinned Sources (field 6, version-specific)
- “Wu et al., “Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks,” NAACL 2024” → SRC-SL-WU-ET-AL-COUNTERFACTUAL-ARXIV-2307-02477--v1.0 — Wu et al., “Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks,” NAACL 2024; class PEER_REVIEWED_CONFERENCE_PAPER (explicit public provenance pin)
All 25 canonical Claim fields (raw values and normalized values)
| # | Canonical field | Canonical value (verbatim) | Normalized (Claim Value Vocabulary v1.0.2) |
|---|---|---|---|
| 1 | CLAIM ID | LD01-SL-C22 | — |
| 2 | PUBLIC-FACING CLAIM | On 11 tasks rewritten with changed premises, such as arithmetic in base 9 or 1-based indexing in code, language models performed above chance but their performance “substantially and consistently degrades” relative to the familiar versions, which the authors attribute partly to overfitting to the default tasks. | — |
| 3 | CLAIM TYPE | ESTIMATE / INTERPRETATION | base_types: ESTIMATE, INTERPRETATION |
| 4 | RESEARCH STRAND | B — Evidence that could tell the accounts apart | — |
| 5 | MATURITY | EMERGING | states: EMERGING |
| 6 | SOURCE | Wu et al., “Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks,” NAACL 2024 | source_ids: SRC-SL-WU-ET-AL-COUNTERFACTUAL-ARXIV-2307-02477 |
| 7 | SOURCE TYPE | Peer-reviewed conference paper | classes: PEER_REVIEWED_CONFERENCE_PAPER |
| 8 | INSTITUTIONAL POSITION | Independent academic (MIT, Boston University) | — |
| 9 | SCOPE | Eleven task families with default and counterfactual versions; English; 2023 models | — |
| 10 | DESIGN | Default and counterfactual task pairs, with checks that models understood the changed rules | — |
| 11 | SAMPLE / DATASET | 11 tasks | — |
| 12 | MODEL / TOOL / VERSION | GPT-4, GPT-3.5, Claude (claude-v1.3) and PaLM-2 | — |
| 13 | COMPARATOR | The default version of each task | — |
| 14 | MAIN RESULT | Above-random counterfactual performance on most tasks, but consistently below default performance | — |
| 15 | INFERENTIAL TARGET | How far model abilities are general rather than tied to familiar task forms | — |
| 16 | DOES NOT LICENSE | A claim that models only recite, or that they reason generally; the authors find evidence of both | — |
| 17 | VERIFICATION STATE | CONTENT VERIFIED PREPRINT | state: CONTENT_VERIFIED_PRIMARY · qualifiers: PREPRINT · verified_by: UNKNOWN |
| 18 | LIMITATIONS | Read in arXiv version 3 (28 March 2024), which carries no venue statement; the NAACL 2024 version was not compared; 2023 models; the authors note humans may also do worse in unfamiliar conditions; one model tested is made by Anthropic, the developer of the model that drafted this debate | — |
| 19 | CONTRADICTORY / SUPPORTING SOURCES | LD01-SL-C04, LD01-SL-C21 | — |
| 20 | GEOGRAPHY / LANGUAGE | English task set | — |
| 21 | DISTRIBUTIONAL LEDGER | Not analysed in this claim | — |
| 22 | R01 / FIELD-MAP LINKS | LD-01, T4, D3 | — |
| 23 | WHAT WOULD CHANGE THIS CLAIM? | Models that perform as well on counterfactual as on default versions | — |
| 24 | PUBLICATION STATUS | ELIGIBLE WITH QUALIFIER | state: ELIGIBLE · qualifiers: QUALIFIED |
| 25 | RECORDED DATE | 2026-10-16 | — |
Field 22 references resolved against Field Map v0.2: LD-01 (LIVING_DEBATE); T4 (TENSION); D3 (DOMAIN)
Cited by Living Debate versions
Research-of-record provenance
Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 652e6386fc3f14c05493e2524a1e1c8c78a0702a154fbde562794667e5616f53.
Repository-authored inputs (not Drive exports)
- SocioLens claim — LD01-SL-C22 v1.0 (v1.0) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-16; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 16 October 2026 to build the LD-01 debate; a new Claim, first version, written from the project's LD-01 source notes, checked against full texts, official web pages or official abstracts. Block “ENTRY 22 — LD01-SL-C22” SHA-256
a1eda34206cb224f7fd5dd9ca26a6048633966ff7aadf98d7ffdfef47bfe3b2b; fixturefixtures/authored/claims/LD01-SL-C22--v1.0.
Builder decisions (not Drive data)
- VERSION = v1.0 — basis: Claim version taken from the repository-authored claim fixture version (SocioLens claim — LD01-SL-C22 v1.0); the first version of a new Claim of the LD-01 debate
- CURRENT_POINTER = true — basis: repository-authored claim "SocioLens claim — LD01-SL-C22 v1.0" (Project Lead (Claude, AI), 2026-10-16); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 16 October 2026 to build the LD-01 debate; a new Claim, first version, written from the project's LD-01 source notes, checked against full texts, official web pages or official abstracts
- FIELD_6_SOURCE_REFERENCE = SRC-SL-WU-ET-AL-COUNTERFACTUAL-ARXIV-2307-02477--v1.0 — basis: field 6 of ENTRY 22 — LD01-SL-C22 cites "Wu et al., “Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks,” NAACL 2024"; the only field-6 segment of this ClaimVersion's repository-authored ledger block.
- SOURCE_PROVENANCE_PIN = SRC-SL-WU-ET-AL-COUNTERFACTUAL-ARXIV-2307-02477--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Wu et al., “Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks,” NAACL 2024".