Research opportunity · AI in the research apparatus
Do pinned model snapshots keep social-science conclusions stable over time?
Earlier version (v1)
This is RO-LD07-01 as it was published at v1, kept unchanged. The current version is v2; its changes are listed there.
Question
On a fixed battery of published social-science annotation tasks, how often does a substantive conclusion (sign, significance, interval coverage) change between repeated queries of a pinned model snapshot, a moving model alias over 6–12 months, and successive open-weight releases, and can design-based correction absorb those changes?
Instrument stabilityOne PhD chapter or paperLow costPartly addressed
Why it matters
Researchers increasingly cite a model name as if it identified a fixed instrument. If conclusions drift with silent updates or release lines, published LLM-annotated findings may not be reproducible even with the same code and prompt.
The debate lists this among what we still do not know: How stable are 2025–2026 model snapshots over time when used for the same social-science measurement task?
Evidence that motivates it
- Model identity is time-indexed: proprietary services can change materially across deployed snapshots, so exact version and date are part of the measurement protocol.
Model identity is time-indexed; exact version and date belong in the protocol. - In a large 2025 preprint spanning 37 annotation tasks from 21 published studies, plausible choices of LLM and annotation configuration produced incorrect inferential conclusions in about one third of tested hypotheses for state-of-the-art models, with higher rates for smaller models and near conventional significance thresholds.
Configuration choices already flip about a third of tested conclusions at a single point in time. - High annotation accuracy does not by itself guarantee valid downstream inference; treating imperfect predicted labels as observed variables can bias estimates and invalidate confidence intervals, while design-based validation with a probability-sampled human-labelled subset can restore valid inference under stated assumptions.
Design-based correction is the candidate remedy whose robustness to drift is untested.
A possible design
Pre-register a battery of 10–20 replication tasks with public data and published estimates (for example from the Baumann et al. or Yang et al. task sets). Re-run them monthly for 6–12 months on pinned snapshot IDs, on the corresponding moving aliases, and on two or three open-weight release lines. The outcome is the substantive estimate, not label agreement. Then apply design-based supervised learning with a fixed human-labelled subsample and test whether it absorbs the drift.
Data needed
Published replication datasets and codebooks; API access to two providers; one GPU for open-weight runs; a human-labelled validation subsample per task (existing gold labels where available).
Feasibility
One PhD chapter or paper. Low: API spend in the low hundreds of US dollars plus modest GPU time. Needs time rather than money: the value comes from the longitudinal window.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026; not yet checked in depth).
Barrie, Palmer and Spirling already show month-to-month drift for closed models and fragile regression conclusions across model versions. What remains open is a pre-registered comparison of pinned snapshots versus aliases versus open-weight lines, with flips in substantive estimates as the outcome and a test of whether design-based correction absorbs them.
How strong this evidence is
- Who searched and checked: an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: none of the 3 nearest works below is recorded as judged from an abstract only. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Barrie, Palmer & Spirling, "Replication for Language Models: Problems, Principles, and Best Practice for Political Science," working paper (Dec 2024 draft)
Monthly re-runs of labelling tasks show closed-model variability and changed regression conclusions; includes one locally run open-weight model (Llama-2) as a stable comparator but does not track open-weight release lines or pinned snapshots. - Fan, Li, Zhuang, Sun & Wang, "The Moving Target: A Longitudinal Audit of Trust-Benchmark Score Drift Across Open-Source Chat LLM Release Lines," arXiv:2607.02587 (2026)
Tracks trust-benchmark drift across three generations of four open-weight families; no social-science tasks or downstream estimates. - Coqueret et al., "Randomness in large language models: What researchers need to know (and report)," arXiv:2607.24372 (2026)
Shows sampling randomness changes downstream regressions and names silent updates as a source; does not measure drift over time.
Search queries used
LLM model version drift annotation social science replication downstream conclusions over timetemporal stability GPT annotation same prompt months later reproducibility text classification political sciencemodel deprecation OR API snapshot LLM social science measurement replicability downstream regression coefficients changelongitudinal re-annotation LLM model updates same model name changes results computational social science replication 2026
Provenance
RO-LD07-01 v1. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-07 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.