Does quantization or the serving engine change social-science findings?

Research question

With weights, prompts and hardware held fixed, do fp16, 8-bit and 4-bit builds of open-weight models, and different inference engines, change substantive estimates on published annotation and synthetic-respondent tasks more than prompt paraphrasing does?

Why it matters

Open-weight models are the main route to reproducible, low-cost LLM research, and most researchers run quantized builds on whatever engine is available. If these invisible choices move conclusions, methods sections need to report them and reviewers need to ask.

The debate lists this among what we still do not know: For open-weight models, do deployment choices such as quantization materially change social-science reliability metrics?

Evidence that motivates it

1. In a large 2025 preprint spanning 37 annotation tasks from 21 published studies, plausible choices of LLM and annotation configuration produced incorrect inferential conclusions in about one third of tested hypotheses for state-of-the-art models, with higher rates for smaller models and near conventional significance thresholds. (LD07-HQ-C02 v1.0)
   Analytic flexibility in model configuration already changes inferential conclusions; deployment settings are an unmeasured extension of that configuration space.
   Cites: Baumann et al., “Large Language Model Hacking”
2. Model identity is time-indexed: proprietary services can change materially across deployed snapshots, so exact version and date are part of the measurement protocol. (LD07-HQ-C12 v1.0)
   The instrument is defined by more than its name; deployment is part of its identity.
   Cites: Chen et al. (2024), “How is ChatGPT’s behavior changing…”

Claims are shown as they read on the day this pack was built; claim pages keep every earlier version.

A possible design

Take 10–15 annotation and silicon-sample tasks with published estimates. Run two or three open-weight model families at several precisions (fp16, 8-bit, 4-bit in common formats) and on two or three serving engines, holding prompts and seeds fixed. Compare the spread of substantive estimates across deployment settings with the spread across prompt paraphrases, using the paraphrase spread as the benchmark for "material".

Data needed

Public replication datasets; open-weight model checkpoints and their quantized builds; one or two consumer or rented GPUs.

Feasibility

Master's thesis. Very low: open-weight models only, a few days of GPU time. A strong first project for a computational methods student.

Has this been done?

Partly done already: related work covers some of this question (first-pass screen of 12 October 2026, checked again on 13 October 2026).

Work on general benchmarks now shows that quantization flips item-level bias responses (Hua, Lotfi and Chen 2026; Rath and Maliakkal 2026), that inference backends and serving context shift scores and classifier probabilities (Pape et al. 2026; Masoudian et al. 2026; Kasa et al. 2026), and one study tests quantized Llama-3.1 on qualitative thematic coding (Adeseye et al. 2026). None varies precision and serving engine jointly on published social-science annotation or silicon-sample tasks, measures changes in substantive estimates, or benchmarks the spread against prompt paraphrasing.

Literature check. No study was found that varies precision and serving engine and measures the change in substantive social-science estimates against a prompt-paraphrase benchmark. However, related work is no longer thin: Hua, Lotfi and Chen (2026) and Rath and Maliakkal (2026) show that 4-bit and 3-bit quantization flips item-level bias responses, Masoudian et al. (2026) and Pape et al. (2026) show inference-backend effects, Kasa et al. (2026) show bf16 serving context shifts classifier probabilities, and Adeseye et al. (2026) test quantized Llama-3.1 on qualitative thematic coding. The verdict should therefore move from OPEN to PARTLY_ADDRESSED while the opportunity stands. Checked by Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert.

Where the search could not reach. Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist. Literature check of 2026-10-13: Only abstracts were read for the 2026 preprints; the full texts may include tasks not described in the abstracts (for example, unnamed benchmarks in Masoudian et al. and Kasa et al.). The submission date shown for Adeseye et al. (4 April 2026) conflicts with its May 2026 arXiv identifier and was not resolved. Scholarly APIs were not used. Absence of a hit is not proof that the work does not exist.

Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.

Nearest work found

• Pape, Evertz & Schönherr, "The Silent Hyperparameter," arXiv:2605.19537 (2026)
  Inference backend alone shifts ML benchmark scores by up to 16.6 points; no social-science tasks, no quantization.
• Singh & Sajjad, "Interpreting the Effects of Quantization on LLMs," IJCNLP-AACL 2025, arXiv:2508.16785
  Finds minor effects of 4- and 8-bit quantization on calibration and internals; no task-level downstream conclusions.
• Cummins, "The threat of analytic flexibility in using LLMs to simulate human data," arXiv:2509.13397 (2025; v4 2026)
  Silicon-sample fidelity varies widely across 252 model, sampling and prompt configurations; does not cover quantization or engines.
• Hua, Lotfi & Chen, "Investigating Social Bias Changes in Quantized Language Models," arXiv:2602.06181 (2026)
  Evaluates 50 quantized models on 13 bias datasets; responses flip between biased and unbiased in up to 21% of cases, 4-bit causes 4-6 times more changes than 8-bit, and group-level bias worsens for some groups and improves for others. Bias benchmarks only; no annotation or survey-experiment estimates, no engine comparison.
• Rath & Maliakkal, "Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels," arXiv:2605.15208 (2026)
  Qwen2.5-7B, Mistral-7B and Phi-3.5-mini from BF16 to 3-bit on BBQ: 2.2-5.6% of items develop new bias at 4-bit (the abstract says 2.5%) and 6-21% at 3-bit. The authors describe perplexity as barely moving, but their own table shows +2.8% to +10.5% at 4-bit and +10% to +83% at 3-bit. Shows item-level shifts hidden by aggregate metrics; BBQ only, no social-science tasks.
• Marcuzzi, Ning, Schwartz & Gurevych, "How Quantization Shapes Bias in Large Language Models," EACL 2026 (Long Papers), pp. 363-404
  Weight and activation quantization across 13 bias benchmarks: little effect on sentiment, lower toxicity, slight increases in stereotypes and unfairness under aggressive compression. Model-evaluation framing; no annotation validity or downstream estimates.
• Adeseye, Isoaho & Adeseye, "Improving Quantized Model Performance in Qualitative Analysis with Multi-Pass Prompt Verification," arXiv:2605.20193 (2026)
  Closest to a social-science use: compares 8-, 4-, 3- and 2-bit Llama-3.1-8B against BF16 on thematic extraction and frequency analysis in qualitative coding; 8-bit stays closest to the gold standard, lower bits degrade, and quantization type matters at a given bit level. One model, one engine family, gold standard partly built from BF16 output, no inferential estimates and no paraphrase benchmark.
• Masoudian, Shafaei, Swain & Schedl, "What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend," arXiv:2608.04714 (2026)
  Five inference frameworks (including HuggingFace, vLLM and Ollama) on six benchmarks; backend effects are larger on factual than on social-bias benchmarks. No quantization axis and no social-science estimates.
• Kasa, Kasa & Negi, "Same Text, Different Prediction: Serving-Context Nondeterminism in Text Classifiers," arXiv:2610.09111 (2026)
  With checkpoint and text fixed, changing batch shape under bf16 shifts up to 56.7 points of predicted probability mass (no label changes under fp32); generative classifiers change more labels. Relevant mechanism for annotation, but trained classifiers on unnamed tasks, no social-science conclusions.
• Yuan, Li, Ding, Xie, Li, Zhao, Wan, Shi, Hu & Liu, "Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference" (v1 titled "Give Me FP32 or Give Me Death?"), arXiv:2506.09501 (2025)
  Batch size, GPU count and GPU type change outputs under greedy decoding, up to 9% accuracy for a reasoning model in bf16. Reasoning benchmarks only.

References

Sources behind the motivating claims:

• Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber & Hovy, “Large Language Model Hacking,” arXiv:2509.08825v2 (SRC-LD07-BAUMANN-LLM-HACKING-2025, original)
• Chen, Zaharia & Zou, “How is ChatGPT’s behavior changing over time?,” Harvard Data Science Review 6(2) (2024) (SRC-LD07-CHEN-ZAHARIA-ZOU-DRIFT-2024, original)

BibTeX

The first entry cites this opportunity; then its nearest work; then the sources above. Nearest work and sources are recorded as reference strings, so each goes in whole as the title; add authors and year from the original before you cite it.

@misc{sociolens_ro_ld07_02_v2,
  author = {{SocioLens project}},
  title = {Does quantization or the serving engine change social-science findings?},
  year = {2026},
  date = {2026-10-13},
  howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld07-02/v2/}},
  note = {Research opportunity RO-LD07-02, version v2}
}

@misc{ro_ld07_02_nearest_1,
  title = {Pape, Evertz \& Schönherr, "The Silent Hyperparameter," arXiv:2605.19537 (2026)},
  url = {https://arxiv.org/abs/2605.19537},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_02_nearest_2,
  title = {Singh \& Sajjad, "Interpreting the Effects of Quantization on LLMs," IJCNLP-AACL 2025, arXiv:2508.16785},
  url = {https://arxiv.org/abs/2508.16785},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_02_nearest_3,
  title = {Cummins, "The threat of analytic flexibility in using LLMs to simulate human data," arXiv:2509.13397 (2025; v4 2026)},
  url = {https://arxiv.org/abs/2509.13397},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_02_nearest_4,
  title = {Hua, Lotfi \& Chen, "Investigating Social Bias Changes in Quantized Language Models," arXiv:2602.06181 (2026)},
  url = {https://arxiv.org/abs/2602.06181},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_02_nearest_5,
  title = {Rath \& Maliakkal, "Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels," arXiv:2605.15208 (2026)},
  url = {https://arxiv.org/abs/2605.15208},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_02_nearest_6,
  title = {Marcuzzi, Ning, Schwartz \& Gurevych, "How Quantization Shapes Bias in Large Language Models," EACL 2026 (Long Papers), pp. 363-404},
  url = {https://aclanthology.org/2026.eacl-long.17/},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_02_nearest_7,
  title = {Adeseye, Isoaho \& Adeseye, "Improving Quantized Model Performance in Qualitative Analysis with Multi-Pass Prompt Verification," arXiv:2605.20193 (2026)},
  url = {https://arxiv.org/abs/2605.20193},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_02_nearest_8,
  title = {Masoudian, Shafaei, Swain \& Schedl, "What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend," arXiv:2608.04714 (2026)},
  url = {https://arxiv.org/abs/2608.04714},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_02_nearest_9,
  title = {Kasa, Kasa \& Negi, "Same Text, Different Prediction: Serving-Context Nondeterminism in Text Classifiers," arXiv:2610.09111 (2026)},
  url = {https://arxiv.org/abs/2610.09111},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld07_02_nearest_10,
  title = {Yuan, Li, Ding, Xie, Li, Zhao, Wan, Shi, Hu \& Liu, "Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference" (v1 titled "Give Me FP32 or Give Me Death?"), arXiv:2506.09501 (2025)},
  url = {https://arxiv.org/abs/2506.09501},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_baumann_llm_hacking_2025,
  title = {Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber \& Hovy, “Large Language Model Hacking,” arXiv:2509.08825v2},
  url = {https://arxiv.org/abs/2509.08825v2},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld07_chen_zaharia_zou_drift_2024,
  title = {Chen, Zaharia \& Zou, “How is ChatGPT’s behavior changing over time?,” Harvard Data Science Review 6(2) (2024)},
  url = {https://doi.org/10.1162/99608f92.5317da47},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}