Research opportunity · AI in the research apparatus
Do pinned model snapshots keep social-science conclusions stable over time?
Question
On a fixed battery of published social-science annotation tasks, how often does a substantive conclusion (sign, significance, interval coverage) change between repeated queries of a pinned model snapshot, a moving model alias over 6–12 months, and successive open-weight releases, and can design-based correction absorb those changes?
Instrument stabilityOne PhD chapter or paperLow costPartly addressed
Why it matters
Researchers increasingly cite a model name as if it identified a fixed instrument. If conclusions drift with silent updates or release lines, published LLM-annotated findings may not be reproducible even with the same code and prompt.
The debate lists this among what we still do not know: How stable are 2025–2026 model snapshots over time when used for the same social-science measurement task?
Evidence that motivates it
- Model identity is time-indexed: proprietary services can change materially across deployed snapshots, so exact version and date are part of the measurement protocol.
Model identity is time-indexed; exact version and date belong in the protocol. - In a large 2025 preprint spanning 37 annotation tasks from 21 published studies, plausible choices of LLM and annotation configuration produced incorrect inferential conclusions in about one third of tested hypotheses for state-of-the-art models, with higher rates for smaller models and near conventional significance thresholds.
Configuration choices already flip about a third of tested conclusions at a single point in time. - High annotation accuracy does not by itself guarantee valid downstream inference; treating imperfect predicted labels as observed variables can bias estimates and invalidate confidence intervals, while design-based validation with a probability-sampled human-labelled subset can restore valid inference under stated assumptions.
Design-based correction is the candidate remedy whose robustness to drift is untested.
A possible design
Pre-register a battery of 10–20 replication tasks with public data and published estimates (for example from the Baumann et al. or Yang et al. task sets). Re-run them monthly for 6–12 months on pinned snapshot IDs, on the corresponding moving aliases, and on two or three open-weight release lines. The outcome is the substantive estimate, not label agreement. Then apply design-based supervised learning with a fixed human-labelled subsample and test whether it absorbs the drift.
Data needed
Published replication datasets and codebooks; API access to two providers; one GPU for open-weight runs; a human-labelled validation subsample per task (existing gold labels where available).
Feasibility
One PhD chapter or paper. Low: API spend in the low hundreds of US dollars plus modest GPU time. Needs time rather than money: the value comes from the longitudinal window.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026, checked again on 13 October 2026).
Barrie, Palmer and Spirling already show month-to-month drift for closed models and fragile regression conclusions across model versions, Chen, Zaharia and Zou showed version-to-version behaviour change in 2023, and Tschisgale and Wulff (2026) found periodic variability in GPT-4o over three months on a non-social-science task. What remains open is a pre-registered comparison of pinned snapshots versus aliases versus open-weight release lines, with flips in substantive social-science estimates as the outcome and a test of whether design-based correction absorbs them.
How strong this evidence is
- Who searched and checked: Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert; an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: none of the 7 nearest works below is recorded as judged from an abstract only, and 1 has details that could not be confirmed; it is marked. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist. Literature check of 2026-10-13: The Barrie, Palmer and Spirling full paper and the Political Communication critique by Balluff et al. could not be read in full (only the slide deck and the ITU repository abstract). The Wiley article by Aksoy (2026) on political bias over time could not be opened; it appears to test political-bias benchmarks over GPT versions rather than annotation tasks, but this is unconfirmed. Scholarly APIs were not used. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Barrie, Palmer & Spirling, "Replication for Language Models: Problems, Principles, and Best Practice for Political Science," working paper (Dec 2024 draft)
Monthly re-runs of labelling tasks show variability in closed models (GPT-4 and Gemini, per the companion slides) and changed regression conclusions; includes one locally run open-weight model (Llama-2) as a stable comparator but does not track open-weight release lines or pinned snapshots. - Fan, Li, Zhuang, Sun & Wang, "The Moving Target: A Longitudinal Audit of Trust-Benchmark Score Drift Across Open-Source Chat LLM Release Lines," arXiv:2607.02587 (2026)
Tracks trust-benchmark drift across three generations of four open-weight families; no social-science tasks or downstream estimates. - Coqueret et al., "Randomness in large language models: What researchers need to know (and report)," arXiv:2607.24372 (2026)
Shows sampling randomness changes downstream regressions and names silent updates as a source; does not measure drift over time. - Chen, Zaharia & Zou, "How is ChatGPT's behavior changing over time?," arXiv:2307.09009 (2023) Some details not confirmed
Compares March and June 2023 releases of GPT-3.5 and GPT-4 on seven tasks (including opinion surveys) and finds substantial behaviour changes between versions. Two time points, no social-science annotation battery, no downstream estimates, no open-weight lines or correction. A version also appears in Harvard Data Science Review (search result only; issue details not confirmed). - Tschisgale & Wulff, "Daily and Weekly Periodicity in Large Language Model Performance and Its Implications for Research," arXiv:2602.15889 (2026)
Queries GPT-4o every three hours for about three months on one physics task with fixed settings and finds periodic variability accounting for about 20% of variance. Shows time-indexed instability of a nominally fixed model; abstract does not say whether a dated snapshot was pinned; not a social-science task and no downstream estimates. - Abraham & Bucknall, "Silent Updates: Measuring and Closing the Post-Deployment Disclosure Gap," arXiv:2608.11803 (2026)
Scores post-deployment disclosure practices of nine API providers and seven inference hosts. Documents that model changes are often undisclosed; does not measure output drift or any research conclusion. - Trisovic, "The Shrinking Lifespan of LLMs in Science," arXiv:2604.07530 (2026)
Bibliometric study of 62 LLMs in 108,000 citing papers: model lifespans in science are shortening by about 23% per release year. Motivates the problem (models retired quickly) but does not re-run any task.
Literature check
Stands as first published. No study was found that pre-registers repeated runs of social-science annotation tasks over 6-12 months comparing pinned snapshots, moving aliases and open-weight release lines, with flips in substantive estimates as the outcome and a test of design-based correction. Barrie, Palmer and Spirling remain the closest work (monthly re-runs, closed models plus Llama-2), and Chen, Zaharia and Zou showed version-to-version behaviour change in 2023 on non-social-science tasks. New 2026 work (Tschisgale and Wulff on periodic GPT-4o variability; Abraham and Bucknall on disclosure of silent updates) strengthens the motivation but does not do the core design.
Searched: web (extended); web (standard); arxiv.org; osf.io; ssrn.com; socarxiv.org; journals.sagepub.com; cambridge.org; tandfonline.com; academic.oup.com; aclanthology.org; onlinelibrary.wiley.com; nature.com. Checked by Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert.
The screen before this check
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026).
Barrie, Palmer and Spirling already show month-to-month drift for closed models and fragile regression conclusions across model versions. What remains open is a pre-registered comparison of pinned snapshots versus aliases versus open-weight lines, with flips in substantive estimates as the outcome and a test of whether design-based correction absorbs them.
Search queries used
LLM model version drift annotation social science replication downstream conclusions over timetemporal stability GPT annotation same prompt months later reproducibility text classification political sciencemodel deprecation OR API snapshot LLM social science measurement replicability downstream regression coefficients changelongitudinal re-annotation LLM model updates same model name changes results computational social science replication 2026
Literature check, 13 October 2026:
LLM annotation reproducibility over time model updates political science replication drift GPT-4 snapshot"Replication for Language Models" Barrie Palmer Spirlinglongitudinal stability LLM text classification repeated monthly measurement closed vs open-weight models social scienceBarrie Palmer Spirling replication language models political science 2025 published journal "trust me bro"model deprecation reproducibility LLM measurement social science same prompt different model version estimates change "design-based supervised learning" robustnesstemporal drift LLM annotation political text classification re-run months later version change replicationdated model snapshot versus model alias reproducibility LLM research results changed silently API update study"How is ChatGPT's behavior changing over time" follow-up social science annotation stability 2026Coqueret "Randomness in large language models" researchers reportsuccessive model releases Llama 3 3.1 3.3 Qwen2.5 Qwen3 annotation agreement social science tasks version comparison downstream regression"Newer, Larger, Better? A Critique of the Unreflective LLM Adoption in Communication Research"
Provenance
RO-LD07-01 v2. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-07 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.
Changes
- v2, 13 October 2026: Literature check added: a second, deeper search for existing work, with its queries, new nearest work and limits. The question and verdict stand. See v1 as published.