Research opportunity · The evaluation apparatus: Can benchmarks, leaderboards and audits carry the weight placed on them?
How many published AI evaluations outside medicine can no longer be repeated?
Question
Among published studies in NLP, the social sciences and education research that evaluate closed language models through an API, what share used a model version that has since been retired or is due to retire, and how many of a sample can still be re-run to reproduce their main result?
Untested beyond the original settingMaster's thesisLow costPartly addressed
Why it matters
Independent evaluation depends on others being able to repeat a result. In biomedicine, 42% of mentions of the most used models involved one retired by publication or due within two years. If the same holds in the social sciences, a large part of the published evidence on what language models do, including much of this platform's evidence base, cannot be checked once the model is gone.
The debate lists this among what we still do not know: How far can published evaluations of closed models be repeated after the model is retired? (Field Map: independent evaluation under API retirement.)
Evidence that motivates it
- Of 8,931 mentions of the 50 most used language models in 5,242 biomedical publications from 2022 to March 2026, 77.7% were of closed commercial models, and 42% involved a model already retired when the paper was published or scheduled to retire within two years.
In biomedical publications, 42% of mentions of the 50 most used models involved a model retired by publication or due to retire within two years; median 538 days to retirement. - The 2025 Foundation Model Transparency Index found that the average transparency score of major AI developers fell from 58 in 2024 to 40 out of 100, and that no company adequately disclosed how far its training data overlap with the evaluations it is tested on; the authors note that changed indicators and companies may contribute to the decline.
Developers give little detail for external reproduction of their evaluations.
A possible design
Sample papers from the ACL Anthology and from social-science and education journals (2022-2026) that evaluate API models; extract the model and version reported; match each to providers' deprecation records; estimate the share retired by publication or within two years, by field. For a random subsample with released code and data, attempt re-runs with the original model where still available and with its designated successor otherwise, and report how often the main result holds.
Data needed
Paper metadata and full texts; providers' public deprecation pages and archived copies; API budget for re-runs.
Feasibility
Master's thesis. Low; public papers, archived deprecation pages and a small API budget. Many papers do not report the exact version, which is itself a finding; extraction by a model must be validated by people, as in the biomedical study.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 15 October 2026; not yet checked in depth).
The biomedical version of the count exists, and software engineering has a re-run study of conference papers that used OpenAI models. No study was found that counts retirement exposure across NLP, social-science and education papers and re-runs a subsample.
How strong this evidence is
- Who searched and checked: .
- What was read: 4 of the 4 nearest works below were judged from an abstract, summary or search listing only; each is marked. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search (3 queries) and one arXiv query on the date shown; the ACL Anthology, Scopus, SocArXiv and education venues were not searched directly, and two 2026 workshop papers seen only in listings were not read. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- N. Wolfrath et al., "Model Retirement Creates Reproducibility Risk in Biomedical AI Publications," arXiv:2609.04699 (2026) Judged from an abstract, summary or search listing only
Abstract read for the LD-06 debate (LD06-SL-C29). The template design, limited to biomedicine and with no re-runs. - Angermeir, Amougou, Kreitz, Bauer et al., "Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies," arXiv:2510.25506 (2025) Judged from an abstract, summary or search listing only
Abstract read. Of 18 ICSE and ASE 2024 studies with artefacts using OpenAI models, 5 could be executed and none fully reproduced; the role of deprecated models was seen in search results only. - Balloccu, Schmidtová, Lango & Dušek, "Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs," arXiv:2402.03927 (2024) Judged from an abstract, summary or search listing only
Abstract read. Review of NLP papers using closed models; its figures on how often versions are reported were seen in search results only. - Cheng, Mayya & Sedoc, "To Err Is Human; To Annotate, SILICON? Toward Robust Reproducibility in LLM Annotation," arXiv:2412.14461 (2024) Judged from an abstract, summary or search listing only
Abstract read. Names model retirement as a core threat to annotation reproducibility; does not count affected papers.
Search queries used
deprecated OpenAI models reproducibility published NLP papers model versions no longer available studyACL papers closed LLM versions reported snapshot reproducibility audit "model version" GPT-3.5 deprecated survey of paperssocial science research using ChatGPT API reproducibility model deprecation share of studies cannot be replicated retired model versionsarXiv: abs:deprecation AND abs:reproducibility
Provenance
RO-LD06-06 v1. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-06 debate on 2026-10-15. Not yet reviewed. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.