Do the failure patterns of 2023–2024 language models persist in reasoning models?

Research question

When the same items that showed sensitivity to output probability, the reversal curse, irrelevant clauses, compositional depth and counterfactual premises are put to successive model generations from 2023 to 2026, including reasoning and agentic models, which failure patterns shrink in size but keep their shape, and which disappear?

Why it matters

The model-level evidence in this debate comes mostly from 2023–2024 systems, and it is read both as mimicry and as a tool with known failure modes. If the patterns keep their shape in newer models, the training objective is still visible in behaviour; if they vanish, the mimicry reading loses its main evidence. One follow-up on a single reasoning model found the same pattern at higher accuracy, and a re-analysis disputes part of the maths result, so the question is open across the battery.

The debate lists this among what we still do not know: Do the failure patterns of 2023–2024 models hold for later reasoning and agentic systems?

Evidence that motivates it

1. McCoy and colleagues predicted that, because language models are trained to predict the next word, their accuracy should depend on how probable the task, input and output are, even where probability should not matter; GPT-4 decoded a simple cipher with 51% accuracy when the output was a high-probability word sequence and 13% when it was low-probability. (LD01-SL-C18 v1.0)
   Accuracy tracks output probability in GPT-4 (51% against 13% on a cipher).
   Cites: McCoy et al. (2023), “Embers of Autoregression…”
2. Testing OpenAI’s o1, a model trained to reason step by step, the authors of “Embers of Autoregression” found that it substantially outperformed earlier models “in many cases” but “still displays the same qualitative trends”: its accuracy at decoding a shift cipher ranged from 47% in the lowest-probability case to 92% in the highest, and it used more reasoning tokens on low-probability examples. (LD01-SL-C37 v1.0)
   OpenAI's o1 shows the same pattern at higher accuracy (47% to 92%).
   Cites: McCoy et al. (2024), “When a language model is optimized…”
3. Models fine-tuned on facts of the form “A is B” failed to answer “B is A”; GPT-4 named the parent of 79% of 1,000 celebrities, but on the 1,573 child–parent pairs this produced it named the celebrity from the parent only 33% of the time. (LD01-SL-C19 v1.0)
   The reversal curse in fine-tuned models and GPT-4.
   Cites: Berglund et al. (2024), “The Reversal Curse: LLMs trained…”
4. Adding a clause that seems relevant but does not change the answer to grade-school maths problems cut accuracy by up to 65 percentage points across 25 models; GPT-4o scored 95.2% on the full GSM8K test set and 63.1% with the clause added, and OpenAI’s o1-preview 94.9% and 77.4%. (LD01-SL-C20 v1.0)
   Irrelevant clauses cut maths accuracy by up to 65 percentage points in 2024 models.
   Cites: Mirzadeh et al. (2025), “GSM-Symbolic: Understanding the…”
5. Off-the-shelf ChatGPT and GPT-4 reached only 55% and 59% accuracy on 3-digit by 3-digit multiplication, and accuracy on compositional tasks fell towards zero as complexity grew; the authors hypothesize that transformers reduce multi-step reasoning to “linearized path matching.” (LD01-SL-C21 v1.0)
   Accuracy on multi-digit multiplication falls towards zero as compositional depth grows.
   Cites: Dziri et al. (2023), “Faith and Fate: Limits of…”
6. On 11 tasks rewritten with changed premises, such as arithmetic in base 9 or 1-based indexing in code, language models performed above chance but their performance “substantially and consistently degrades” relative to the familiar versions, which the authors attribute partly to overfitting to the default tasks. (LD01-SL-C22 v1.0)
   Counterfactual task versions degrade performance consistently.
   Cites: Wu et al. (2024), “Reasoning or Reciting? Exploring the…”

Claims are shown as they read on the day this pack was built; claim pages keep every earlier version.

A possible design

Assemble the published items of the five studies into one battery, with permission where needed, and add fresh items generated by the same procedures to limit contamination. Run it on a fixed panel of open and closed models released from 2023 to 2026 under matched prompting, with and without step-by-step reasoning and tool use. Estimate each pattern's effect size by model generation with mixed-effects models and per-item random effects, as the GSM-Symbolic re-analysis recommends, and report where effects keep their shape as accuracy rises.

Data needed

Published task items; API or weight access to models across generations; compute for evaluation; contamination checks.

Feasibility

One PhD chapter or paper. Medium; mainly API fees for closed models. Retired model versions may no longer be available, which limits the panel; record exact versions and dates. Benchmark items may have entered training data after publication, so fresh items are essential.

Has this been done?

Partly done already: related work covers some of this question (first-pass screen of 16 October 2026; not yet checked in depth).

Single follow-ups exist for one pattern or one model, but no study was found that runs these failure patterns as one battery across model generations including agentic systems.

Where the search could not reach. Screened by one web search, arXiv record lookups and the LD-01 source scan on the date shown; replications posted as blog posts or in model cards were not searched. Absence of a hit is not proof that the work does not exist.

Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.

Nearest work found

• McCoy et al., o1 follow-up (LD01-SL-C37)
  Full text read for this debate. Re-tests one study's tasks on one reasoning model.
• Długosz, Oliveira & Díaz-Rodríguez, "The Importance of Being Statistically Earnest: A Critical Re-evaluation of GSM-Symbolic," arXiv:2605.28700 (2026) Judged from an abstract, summary or search listing only
  Abstract, introduction and one footnote read; not read in full. Re-analyses the template variants on 20 open-weight models with mixed-effects models; leaves the irrelevant-clause variant out of scope.

References

Sources behind the motivating claims:

• McCoy, Yao, Friedman, Hardy & Griffiths, “Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve,” arXiv:2309.13638, 2023 (SRC-SL-MCCOY-ET-AL-EMBERS-ARXIV-2309-13638, original)
• McCoy, Yao, Friedman, Hardy & Griffiths, “When a language model is optimized for reasoning, does it still show embers of autoregression? An analysis of OpenAI o1,” arXiv:2410.01792, 2024 (SRC-SL-MCCOY-ET-AL-O1-ARXIV-2410-01792, original)
• Berglund, Tong, Kaufmann, Balesni, Cooper Stickland, Korbak & Evans, “The Reversal Curse: LLMs trained on ‘A is B’ fail to learn ‘B is A’,” ICLR 2024 (SRC-SL-BERGLUND-ET-AL-REVERSAL-ARXIV-2309-12288, original)
• Mirzadeh, Alizadeh, Shahrokhi, Tuzel, Bengio & Farajtabar, “GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models,” ICLR 2025 (SRC-SL-MIRZADEH-ET-AL-GSM-SYMBOLIC-ARXIV-2410-05229, original)
• Dziri et al., “Faith and Fate: Limits of Transformers on Compositionality,” NeurIPS 2023 (SRC-SL-DZIRI-ET-AL-FAITH-FATE-ARXIV-2305-18654, original)
• Wu et al., “Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks,” NAACL 2024 (SRC-SL-WU-ET-AL-COUNTERFACTUAL-ARXIV-2307-02477, original)

BibTeX

The first entry cites this opportunity; then its nearest work; then the sources above. Nearest work and sources are recorded as reference strings, so each goes in whole as the title; add authors and year from the original before you cite it.

@misc{sociolens_ro_ld01_02_v1,
  author = {{SocioLens project}},
  title = {Do the failure patterns of 2023–2024 language models persist in reasoning models?},
  year = {2026},
  date = {2026-10-16},
  howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld01-02/v1/}},
  note = {Research opportunity RO-LD01-02, version v1}
}

@misc{ro_ld01_02_nearest_1,
  title = {McCoy et al., o1 follow-up (LD01-SL-C37)},
  url = {https://arxiv.org/abs/2410.01792},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld01_02_nearest_2,
  title = {Długosz, Oliveira \& Díaz-Rodríguez, "The Importance of Being Statistically Earnest: A Critical Re-evaluation of GSM-Symbolic," arXiv:2605.28700 (2026)},
  url = {https://arxiv.org/abs/2605.28700},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{src_sl_mccoy_et_al_embers_arxiv_2309_13638,
  title = {McCoy, Yao, Friedman, Hardy \& Griffiths, “Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve,” arXiv:2309.13638, 2023},
  url = {https://arxiv.org/abs/2309.13638},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_sl_mccoy_et_al_o1_arxiv_2410_01792,
  title = {McCoy, Yao, Friedman, Hardy \& Griffiths, “When a language model is optimized for reasoning, does it still show embers of autoregression? An analysis of OpenAI o1,” arXiv:2410.01792, 2024},
  url = {https://arxiv.org/abs/2410.01792},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_sl_berglund_et_al_reversal_arxiv_2309_12288,
  title = {Berglund, Tong, Kaufmann, Balesni, Cooper Stickland, Korbak \& Evans, “The Reversal Curse: LLMs trained on ‘A is B’ fail to learn ‘B is A’,” ICLR 2024},
  url = {https://arxiv.org/abs/2309.12288},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_sl_mirzadeh_et_al_gsm_symbolic_arxiv_2410_05229,
  title = {Mirzadeh, Alizadeh, Shahrokhi, Tuzel, Bengio \& Farajtabar, “GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models,” ICLR 2025},
  url = {https://arxiv.org/abs/2410.05229},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_sl_dziri_et_al_faith_fate_arxiv_2305_18654,
  title = {Dziri et al., “Faith and Fate: Limits of Transformers on Compositionality,” NeurIPS 2023},
  url = {https://arxiv.org/abs/2305.18654},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_sl_wu_et_al_counterfactual_arxiv_2307_02477,
  title = {Wu et al., “Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks,” NAACL 2024},
  url = {https://arxiv.org/abs/2307.02477},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}