How many published AI evaluations outside medicine can no longer be repeated?

Research question

Among published studies in NLP, the social sciences and education research that evaluate closed language models through an API, what share used a model version that has since been retired or is due to retire, and how many of a sample can still be re-run to reproduce their main result?

Why it matters

Independent evaluation depends on others being able to repeat a result. In biomedicine, 42% of mentions of the most used models involved one retired by publication or due within two years. If the same holds in the social sciences, a large part of the published evidence on what language models do, including much of this platform's evidence base, cannot be checked once the model is gone.

The debate lists this among what we still do not know: How far can published evaluations of closed models be repeated after the model is retired? (Field Map: independent evaluation under API retirement.)

Evidence that motivates it

1. Of 8,931 mentions of the 50 most used language models in 5,242 biomedical publications from 2022 to March 2026, 77.7% were of closed commercial models, and 42% involved a model already retired when the paper was published or scheduled to retire within two years. (LD06-SL-C29 v1.0)
   In biomedical publications, 42% of mentions of the 50 most used models involved a model retired by publication or due to retire within two years; median 538 days to retirement.
   Cites: Wolfrath et al. (2026), “Model Retirement Creates…”
2. The 2025 Foundation Model Transparency Index found that the average transparency score of major AI developers fell from 58 in 2024 to 40 out of 100, and that no company adequately disclosed how far its training data overlap with the evaluations it is tested on; the authors note that changed indicators and companies may contribute to the decline. (LD06-SL-C17 v1.0)
   Developers give little detail for external reproduction of their evaluations.
   Cites: Wan et al. (2025), “The 2025 Foundation Model…”

Claims are shown as they read on the day this pack was built; claim pages keep every earlier version.

A possible design

Sample papers from the ACL Anthology and from social-science and education journals (2022-2026) that evaluate API models; extract the model and version reported; match each to providers' deprecation records; estimate the share retired by publication or within two years, by field. For a random subsample with released code and data, attempt re-runs with the original model where still available and with its designated successor otherwise, and report how often the main result holds.

Data needed

Paper metadata and full texts; providers' public deprecation pages and archived copies; API budget for re-runs.

Feasibility

Master's thesis. Low; public papers, archived deprecation pages and a small API budget. Many papers do not report the exact version, which is itself a finding; extraction by a model must be validated by people, as in the biomedical study.

Has this been done?

Partly done already: related work covers some of this question (first-pass screen of 15 October 2026; not yet checked in depth).

The biomedical version of the count exists, and software engineering has a re-run study of conference papers that used OpenAI models. No study was found that counts retirement exposure across NLP, social-science and education papers and re-runs a subsample.

Where the search could not reach. Screened by web search (3 queries) and one arXiv query on the date shown; the ACL Anthology, Scopus, SocArXiv and education venues were not searched directly, and two 2026 workshop papers seen only in listings were not read. Absence of a hit is not proof that the work does not exist.

Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.

Nearest work found

• N. Wolfrath et al., "Model Retirement Creates Reproducibility Risk in Biomedical AI Publications," arXiv:2609.04699 (2026) Judged from an abstract, summary or search listing only
  Abstract read for the LD-06 debate (LD06-SL-C29). The template design, limited to biomedicine and with no re-runs.
• Angermeir, Amougou, Kreitz, Bauer et al., "Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies," arXiv:2510.25506 (2025) Judged from an abstract, summary or search listing only
  Abstract read. Of 18 ICSE and ASE 2024 studies with artefacts using OpenAI models, 5 could be executed and none fully reproduced; the role of deprecated models was seen in search results only.
• Balloccu, Schmidtová, Lango & Dušek, "Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs," arXiv:2402.03927 (2024) Judged from an abstract, summary or search listing only
  Abstract read. Review of NLP papers using closed models; its figures on how often versions are reported were seen in search results only.
• Cheng, Mayya & Sedoc, "To Err Is Human; To Annotate, SILICON? Toward Robust Reproducibility in LLM Annotation," arXiv:2412.14461 (2024) Judged from an abstract, summary or search listing only
  Abstract read. Names model retirement as a core threat to annotation reproducibility; does not count affected papers.

References

Sources behind the motivating claims:

• Wolfrath, Conroy, Kosten, Bell, Neupane, Kindel et al., “Model Retirement Creates Reproducibility Risk in Biomedical AI Publications,” arXiv:2609.04699, September 2026 (SRC-SL-WOLFRATH-ET-AL-MODEL-RETIREMENT-ARXIV-2609-04699, original)
• Wan, Klyman, Kapoor, Maslej, Longpre, Xiong, Liang & Bommasani, “The 2025 Foundation Model Transparency Index,” arXiv:2512.10169, December 2025 (SRC-SL-WAN-ET-AL-FMTI-2025-ARXIV-2512-10169, original)

BibTeX

The first entry cites this opportunity; then its nearest work; then the sources above. Nearest work and sources are recorded as reference strings, so each goes in whole as the title; add authors and year from the original before you cite it.

@misc{sociolens_ro_ld06_06_v1,
  author = {{SocioLens project}},
  title = {How many published AI evaluations outside medicine can no longer be repeated?},
  year = {2026},
  date = {2026-10-15},
  howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld06-06/v1/}},
  note = {Research opportunity RO-LD06-06, version v1}
}

@misc{ro_ld06_06_nearest_1,
  title = {N. Wolfrath et al., "Model Retirement Creates Reproducibility Risk in Biomedical AI Publications," arXiv:2609.04699 (2026)},
  url = {https://arxiv.org/abs/2609.04699},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld06_06_nearest_2,
  title = {Angermeir, Amougou, Kreitz, Bauer et al., "Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies," arXiv:2510.25506 (2025)},
  url = {https://arxiv.org/abs/2510.25506},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld06_06_nearest_3,
  title = {Balloccu, Schmidtová, Lango \& Dušek, "Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs," arXiv:2402.03927 (2024)},
  url = {https://arxiv.org/abs/2402.03927},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld06_06_nearest_4,
  title = {Cheng, Mayya \& Sedoc, "To Err Is Human; To Annotate, SILICON? Toward Robust Reproducibility in LLM Annotation," arXiv:2412.14461 (2024)},
  url = {https://arxiv.org/abs/2412.14461},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{src_sl_wolfrath_et_al_model_retirement_arxiv_2609_04699,
  title = {Wolfrath, Conroy, Kosten, Bell, Neupane, Kindel et al., “Model Retirement Creates Reproducibility Risk in Biomedical AI Publications,” arXiv:2609.04699, September 2026},
  url = {https://arxiv.org/abs/2609.04699},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_sl_wan_et_al_fmti_2025_arxiv_2512_10169,
  title = {Wan, Klyman, Kapoor, Maslej, Longpre, Xiong, Liang \& Bommasani, “The 2025 Foundation Model Transparency Index,” arXiv:2512.10169, December 2025},
  url = {https://arxiv.org/abs/2512.10169},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}