Research opportunity · AI in the research apparatus
Does quantization or the serving engine change social-science findings?
Question
With weights, prompts and hardware held fixed, do fp16, 8-bit and 4-bit builds of open-weight models, and different inference engines, change substantive estimates on published annotation and synthetic-respondent tasks more than prompt paraphrasing does?
Instrument stabilityMaster's thesisLow costPartly addressed
Why it matters
Open-weight models are the main route to reproducible, low-cost LLM research, and most researchers run quantized builds on whatever engine is available. If these invisible choices move conclusions, methods sections need to report them and reviewers need to ask.
The debate lists this among what we still do not know: For open-weight models, do deployment choices such as quantization materially change social-science reliability metrics?
Evidence that motivates it
- In a large 2025 preprint spanning 37 annotation tasks from 21 published studies, plausible choices of LLM and annotation configuration produced incorrect inferential conclusions in about one third of tested hypotheses for state-of-the-art models, with higher rates for smaller models and near conventional significance thresholds.
Analytic flexibility in model configuration already changes inferential conclusions; deployment settings are an unmeasured extension of that configuration space. - Model identity is time-indexed: proprietary services can change materially across deployed snapshots, so exact version and date are part of the measurement protocol.
The instrument is defined by more than its name; deployment is part of its identity.
A possible design
Take 10–15 annotation and silicon-sample tasks with published estimates. Run two or three open-weight model families at several precisions (fp16, 8-bit, 4-bit in common formats) and on two or three serving engines, holding prompts and seeds fixed. Compare the spread of substantive estimates across deployment settings with the spread across prompt paraphrases, using the paraphrase spread as the benchmark for "material".
Data needed
Public replication datasets; open-weight model checkpoints and their quantized builds; one or two consumer or rented GPUs.
Feasibility
Master's thesis. Very low: open-weight models only, a few days of GPU time. A strong first project for a computational methods student.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026, checked again on 13 October 2026).
Work on general benchmarks now shows that quantization flips item-level bias responses (Hua, Lotfi and Chen 2026; Rath and Maliakkal 2026), that inference backends and serving context shift scores and classifier probabilities (Pape et al. 2026; Masoudian et al. 2026; Kasa et al. 2026), and one study tests quantized Llama-3.1 on qualitative thematic coding (Adeseye et al. 2026). None varies precision and serving engine jointly on published social-science annotation or silicon-sample tasks, measures changes in substantive estimates, or benchmarks the spread against prompt paraphrasing.
How strong this evidence is
- Who searched and checked: Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert; an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: none of the 10 nearest works below is recorded as judged from an abstract only. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist. Literature check of 2026-10-13: Only abstracts were read for the 2026 preprints; the full texts may include tasks not described in the abstracts (for example, unnamed benchmarks in Masoudian et al. and Kasa et al.). The submission date shown for Adeseye et al. (4 April 2026) conflicts with its May 2026 arXiv identifier and was not resolved. Scholarly APIs were not used. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Pape, Evertz & Schönherr, "The Silent Hyperparameter," arXiv:2605.19537 (2026)
Inference backend alone shifts ML benchmark scores by up to 16.6 points; no social-science tasks, no quantization. - Singh & Sajjad, "Interpreting the Effects of Quantization on LLMs," IJCNLP-AACL 2025, arXiv:2508.16785
Finds minor effects of 4- and 8-bit quantization on calibration and internals; no task-level downstream conclusions. - Cummins, "The threat of analytic flexibility in using LLMs to simulate human data," arXiv:2509.13397 (2025; v4 2026)
Silicon-sample fidelity varies widely across 252 model, sampling and prompt configurations; does not cover quantization or engines. - Hua, Lotfi & Chen, "Investigating Social Bias Changes in Quantized Language Models," arXiv:2602.06181 (2026)
Evaluates 50 quantized models on 13 bias datasets; responses flip between biased and unbiased in up to 21% of cases, 4-bit causes 4-6 times more changes than 8-bit, and group-level bias worsens for some groups and improves for others. Bias benchmarks only; no annotation or survey-experiment estimates, no engine comparison. - Rath & Maliakkal, "Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels," arXiv:2605.15208 (2026)
Qwen2.5-7B, Mistral-7B and Phi-3.5-mini from BF16 to 3-bit on BBQ: 2.2-5.6% of items develop new bias at 4-bit (the abstract says 2.5%) and 6-21% at 3-bit. The authors describe perplexity as barely moving, but their own table shows +2.8% to +10.5% at 4-bit and +10% to +83% at 3-bit. Shows item-level shifts hidden by aggregate metrics; BBQ only, no social-science tasks. - Marcuzzi, Ning, Schwartz & Gurevych, "How Quantization Shapes Bias in Large Language Models," EACL 2026 (Long Papers), pp. 363-404
Weight and activation quantization across 13 bias benchmarks: little effect on sentiment, lower toxicity, slight increases in stereotypes and unfairness under aggressive compression. Model-evaluation framing; no annotation validity or downstream estimates. - Adeseye, Isoaho & Adeseye, "Improving Quantized Model Performance in Qualitative Analysis with Multi-Pass Prompt Verification," arXiv:2605.20193 (2026)
Closest to a social-science use: compares 8-, 4-, 3- and 2-bit Llama-3.1-8B against BF16 on thematic extraction and frequency analysis in qualitative coding; 8-bit stays closest to the gold standard, lower bits degrade, and quantization type matters at a given bit level. One model, one engine family, gold standard partly built from BF16 output, no inferential estimates and no paraphrase benchmark. - Masoudian, Shafaei, Swain & Schedl, "What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend," arXiv:2608.04714 (2026)
Five inference frameworks (including HuggingFace, vLLM and Ollama) on six benchmarks; backend effects are larger on factual than on social-bias benchmarks. No quantization axis and no social-science estimates. - Kasa, Kasa & Negi, "Same Text, Different Prediction: Serving-Context Nondeterminism in Text Classifiers," arXiv:2610.09111 (2026)
With checkpoint and text fixed, changing batch shape under bf16 shifts up to 56.7 points of predicted probability mass (no label changes under fp32); generative classifiers change more labels. Relevant mechanism for annotation, but trained classifiers on unnamed tasks, no social-science conclusions. - Yuan, Li, Ding, Xie, Li, Zhao, Wan, Shi, Hu & Liu, "Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference" (v1 titled "Give Me FP32 or Give Me Death?"), arXiv:2506.09501 (2025)
Batch size, GPU count and GPU type change outputs under greedy decoding, up to 9% accuracy for a reasoning model in bf16. Reasoning benchmarks only.
Literature check
Question stands; related work found. No study was found that varies precision and serving engine and measures the change in substantive social-science estimates against a prompt-paraphrase benchmark. However, related work is no longer thin: Hua, Lotfi and Chen (2026) and Rath and Maliakkal (2026) show that 4-bit and 3-bit quantization flips item-level bias responses, Masoudian et al. (2026) and Pape et al. (2026) show inference-backend effects, Kasa et al. (2026) show bf16 serving context shifts classifier probabilities, and Adeseye et al. (2026) test quantized Llama-3.1 on qualitative thematic coding. The verdict should therefore move from OPEN to PARTLY_ADDRESSED while the opportunity stands.
Searched: web (extended); web (standard); arxiv.org; osf.io; ssrn.com; aclanthology.org; dl.acm.org; openreview.net. Checked by Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert.
The screen before this check (verdict changed)
The search found no study that does the core design (first-pass screen of 12 October 2026).
Work exists on inference-backend effects on ML benchmarks and on decoding and sampling randomness in research use, but none found varies precision and serving engine and measures changes in social-science conclusions.
Search queries used
quantization effects LLM annotation social science 4-bit 8-bit text classification validityinference engine vLLM llama.cpp nondeterminism hardware LLM outputs differ survey simulation or annotation reproducibility social sciencequantized open-weight models silicon sampling public opinion survey responses quantization biasLLM hacking researcher degrees of freedom model configuration temperature annotation statistical conclusions
Literature check, 13 October 2026:
quantized LLM survey simulation persona responses GPTQ AWQ GGUF bias opinion distribution effectquantization effect on LLM text annotation classification stance sentiment computational social science 4-bitquantization bias fairness social bias stereotypes compressed language models evaluationnumerical precision nondeterminism LLM inference reproducibility bf16 fp32 GPU batch size results differ evaluation"The Silent Hyperparameter" inference backend vLLM benchmark scoresOllama llama.cpp quantized model researchers social science reproducibility silicon sampling local models results differ from full precisionquantization changes opinion alignment survey responses LLM 4-bit 8-bit OpinionQA World Values Survey
Provenance
RO-LD07-02 v2. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-07 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.
Changes
- v2, 13 October 2026: Literature check added: a second, deeper search for existing work, with its queries, new nearest work and limits. The verdict changed from "No study found doing the core design" to "Related work exists" because related work turned up; the question stands. See v1 as published.