Claim LD01-SL-C22 (v1.0)

On 11 tasks rewritten with changed premises, such as arithmetic in base 9 or 1-based indexing in code, language models performed above chance but their performance “substantially and consistently degrades” relative to the familiar versions, which the authors attribute partly to overfitting to the default tasks.

Provenance and status

Claim type: ESTIMATE + INTERPRETATION (as printed: “ESTIMATE / INTERPRETATION”)
Publication status: ELIGIBLE (QUALIFIED) (as printed: “ELIGIBLE WITH QUALIFIER”)
Verification state: CONTENT_VERIFIED_PRIMARY (PREPRINT) (as printed: “CONTENT VERIFIED PREPRINT”; verified by: UNKNOWN)
Does not license: A claim that models only recite, or that they reason generally; the authors find evidence of both
Source (field 6, as printed): Wu et al., “Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks,” NAACL 2024
Source type (field 7): PEER_REVIEWED_CONFERENCE_PAPER (as printed: “Peer-reviewed conference paper”)
Institutional position: Independent academic (MIT, Boston University)
Maturity (claim evidence / publication maturity): EMERGING (as printed: “EMERGING”)
Recorded date: 2026-10-16
Currentness / correction: This is the current version. No correction or concern relation is recorded on its Sources.

Version

Pinned version v1.0 (entry LD01-SL-C22--v1.0) — this page never floats to newer evidence.

This pinned version is also the current version.

Versions:v1.0 · current

Pinned Sources (field 6, version-specific)

“Wu et al., “Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks,” NAACL 2024” → SRC-SL-WU-ET-AL-COUNTERFACTUAL-ARXIV-2307-02477--v1.0 — Wu et al., “Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks,” NAACL 2024; class PEER_REVIEWED_CONFERENCE_PAPER (explicit public provenance pin)

All 25 canonical Claim fields (raw values and normalized values)

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD01-SL-C22	—
2	PUBLIC-FACING CLAIM	On 11 tasks rewritten with changed premises, such as arithmetic in base 9 or 1-based indexing in code, language models performed above chance but their performance “substantially and consistently degrades” relative to the familiar versions, which the authors attribute partly to overfitting to the default tasks.	—
3	CLAIM TYPE	ESTIMATE / INTERPRETATION	base_types: ESTIMATE, INTERPRETATION
4	RESEARCH STRAND	B — Evidence that could tell the accounts apart	—
5	MATURITY	EMERGING	states: EMERGING
6	SOURCE	Wu et al., “Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks,” NAACL 2024	source_ids: SRC-SL-WU-ET-AL-COUNTERFACTUAL-ARXIV-2307-02477
7	SOURCE TYPE	Peer-reviewed conference paper	classes: PEER_REVIEWED_CONFERENCE_PAPER
8	INSTITUTIONAL POSITION	Independent academic (MIT, Boston University)	—
9	SCOPE	Eleven task families with default and counterfactual versions; English; 2023 models	—
10	DESIGN	Default and counterfactual task pairs, with checks that models understood the changed rules	—
11	SAMPLE / DATASET	11 tasks	—
12	MODEL / TOOL / VERSION	GPT-4, GPT-3.5, Claude (claude-v1.3) and PaLM-2	—
13	COMPARATOR	The default version of each task	—
14	MAIN RESULT	Above-random counterfactual performance on most tasks, but consistently below default performance	—
15	INFERENTIAL TARGET	How far model abilities are general rather than tied to familiar task forms	—
16	DOES NOT LICENSE	A claim that models only recite, or that they reason generally; the authors find evidence of both	—
17	VERIFICATION STATE	CONTENT VERIFIED PREPRINT	state: CONTENT_VERIFIED_PRIMARY · qualifiers: PREPRINT · verified_by: UNKNOWN
18	LIMITATIONS	Read in arXiv version 3 (28 March 2024), which carries no venue statement; the NAACL 2024 version was not compared; 2023 models; the authors note humans may also do worse in unfamiliar conditions; one model tested is made by Anthropic, the developer of the model that drafted this debate	—
19	CONTRADICTORY / SUPPORTING SOURCES	LD01-SL-C04, LD01-SL-C21	—
20	GEOGRAPHY / LANGUAGE	English task set	—
21	DISTRIBUTIONAL LEDGER	Not analysed in this claim	—
22	R01 / FIELD-MAP LINKS	LD-01, T4, D3	—
23	WHAT WOULD CHANGE THIS CLAIM?	Models that perform as well on counterfactual as on default versions	—
24	PUBLICATION STATUS	ELIGIBLE WITH QUALIFIER	state: ELIGIBLE · qualifiers: QUALIFIED
25	RECORDED DATE	2026-10-16	—

Field 22 references resolved against Field Map v0.2: LD-01 (LIVING_DEBATE); T4 (TENSION); D3 (DOMAIN)

Cited by Living Debate versions

LD-01 v0.1

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 652e6386fc3f14c05493e2524a1e1c8c78a0702a154fbde562794667e5616f53.

Repository-authored inputs (not Drive exports)

SocioLens claim — LD01-SL-C22 v1.0 (v1.0) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-16; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 16 October 2026 to build the LD-01 debate; a new Claim, first version, written from the project's LD-01 source notes, checked against full texts, official web pages or official abstracts. Block “ENTRY 22 — LD01-SL-C22” SHA-256 a1eda34206cb224f7fd5dd9ca26a6048633966ff7aadf98d7ffdfef47bfe3b2b; fixture fixtures/authored/claims/LD01-SL-C22--v1.0.

Builder decisions (not Drive data)

VERSION = v1.0 — basis: Claim version taken from the repository-authored claim fixture version (SocioLens claim — LD01-SL-C22 v1.0); the first version of a new Claim of the LD-01 debate
CURRENT_POINTER = true — basis: repository-authored claim "SocioLens claim — LD01-SL-C22 v1.0" (Project Lead (Claude, AI), 2026-10-16); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 16 October 2026 to build the LD-01 debate; a new Claim, first version, written from the project's LD-01 source notes, checked against full texts, official web pages or official abstracts
FIELD_6_SOURCE_REFERENCE = SRC-SL-WU-ET-AL-COUNTERFACTUAL-ARXIV-2307-02477--v1.0 — basis: field 6 of ENTRY 22 — LD01-SL-C22 cites "Wu et al., “Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks,” NAACL 2024"; the only field-6 segment of this ClaimVersion's repository-authored ledger block.
SOURCE_PROVENANCE_PIN = SRC-SL-WU-ET-AL-COUNTERFACTUAL-ARXIV-2307-02477--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Wu et al., “Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks,” NAACL 2024".