Research opportunity · AI in the research apparatus
Does quantization or the serving engine change social-science findings?
Earlier version (v1)
This is RO-LD07-02 as it was published at v1, kept unchanged. The current version is v2; its changes are listed there.
Question
With weights, prompts and hardware held fixed, do fp16, 8-bit and 4-bit builds of open-weight models, and different inference engines, change substantive estimates on published annotation and synthetic-respondent tasks more than prompt paraphrasing does?
Instrument stabilityMaster's thesisLow costNo study found
Why it matters
Open-weight models are the main route to reproducible, low-cost LLM research, and most researchers run quantized builds on whatever engine is available. If these invisible choices move conclusions, methods sections need to report them and reviewers need to ask.
The debate lists this among what we still do not know: For open-weight models, do deployment choices such as quantization materially change social-science reliability metrics?
Evidence that motivates it
- In a large 2025 preprint spanning 37 annotation tasks from 21 published studies, plausible choices of LLM and annotation configuration produced incorrect inferential conclusions in about one third of tested hypotheses for state-of-the-art models, with higher rates for smaller models and near conventional significance thresholds.
Analytic flexibility in model configuration already changes inferential conclusions; deployment settings are an unmeasured extension of that configuration space. - Model identity is time-indexed: proprietary services can change materially across deployed snapshots, so exact version and date are part of the measurement protocol.
The instrument is defined by more than its name; deployment is part of its identity.
A possible design
Take 10–15 annotation and silicon-sample tasks with published estimates. Run two or three open-weight model families at several precisions (fp16, 8-bit, 4-bit in common formats) and on two or three serving engines, holding prompts and seeds fixed. Compare the spread of substantive estimates across deployment settings with the spread across prompt paraphrases, using the paraphrase spread as the benchmark for "material".
Data needed
Public replication datasets; open-weight model checkpoints and their quantized builds; one or two consumer or rented GPUs.
Feasibility
Master's thesis. Very low: open-weight models only, a few days of GPU time. A strong first project for a computational methods student.
Has this been done?
The search found no study that does the core design (first-pass screen of 12 October 2026; not yet checked in depth).
Work exists on inference-backend effects on ML benchmarks and on decoding and sampling randomness in research use, but none found varies precision and serving engine and measures changes in social-science conclusions.
How strong this evidence is
- Who searched and checked: an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: none of the 3 nearest works below is recorded as judged from an abstract only. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Pape, Evertz & Schönherr, "The Silent Hyperparameter," arXiv:2605.19537 (2026)
Inference backend alone shifts ML benchmark scores by up to 16.6 points; no social-science tasks, no quantization. - Singh & Sajjad, "Interpreting the Effects of Quantization on LLMs," IJCNLP-AACL 2025, arXiv:2508.16785
Finds minor effects of 4- and 8-bit quantization on calibration and internals; no task-level downstream conclusions. - Cummins, "The threat of analytic flexibility in using LLMs to simulate human data," arXiv:2509.13397 (2025; v4 2026)
Silicon-sample fidelity varies widely across 252 model, sampling and prompt configurations; does not cover quantization or engines.
Search queries used
quantization effects LLM annotation social science 4-bit 8-bit text classification validityinference engine vLLM llama.cpp nondeterminism hardware LLM outputs differ survey simulation or annotation reproducibility social sciencequantized open-weight models silicon sampling public opinion survey responses quantization biasLLM hacking researcher degrees of freedom model configuration temperature annotation statistical conclusions
Provenance
RO-LD07-02 v1. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-07 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.