Research opportunity · AI in the research apparatus
Does post-training change what an LLM measures when it codes text?
Earlier version (v1)
This is RO-LD07-05 as it was published at v1, kept unchanged. The current version is v2; its changes are listed there.
Question
Holding base weights fixed, how much do instruction tuning, preference tuning and region-specific fine-tunes shift coding validity and downstream estimates on stance and attitude annotation tasks, and do the shifts track the fine-tune's target culture?
Causal mechanism not isolatedOne PhD chapter or paperLow costPartly addressed
Why it matters
Observed differences between providers are routinely attributed to alignment or regulation, but existing comparisons cannot separate those from differences in base models and training data. Within-family checkpoint contrasts can isolate the effect of post-training on measurement itself.
The debate lists this among what we still do not know: How much do jurisdictional alignment regimes causally change social-science measurement, rather than merely refusal behavior?
Evidence that motivates it
- Provider and jurisdictional context can be associated with materially different refusal and accuracy patterns on politically sensitive prompts; existing evidence does not isolate regulation as the causal mechanism.
Provider and jurisdiction are associated with different response patterns, but the causal mechanism is not isolated. - Chinese-language silicon-sample studies benchmarked against CGSS data find partial macro-level reproduction alongside representational bias, micro/macro/meso instability and cross-model differences; current systems do not warrant replacement of human respondents.
Chinese-language silicon-sample studies show cross-model differences whose source is unknown.
A possible design
Use model families with public checkpoints at each post-training stage (base, supervised fine-tuning, preference tuning) and community or regional fine-tunes of the same base. Run established stance and attitude coding tasks with expert gold labels in two or more languages. Estimate the stage effect on accuracy, bias by target group and downstream coefficients, holding prompts and decoding fixed.
Data needed
Open checkpoints with documented training stages; annotated political-text corpora in at least two languages; GPU time.
Feasibility
One PhD chapter or paper. Low: open-weight checkpoints and existing labelled corpora.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026; not yet checked in depth).
For survey simulation, Ziaei (22 September 2026) traces base-to-tuned checkpoints and finds supervised tuning roughly halves response dispersion; Santurkar et al. contrast base and RLHF opinions. No study found applies the same checkpoint design to annotation and coding validity or to regional fine-tunes of a shared base.
How strong this evidence is
- Who searched and checked: an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: none of the 3 nearest works below is recorded as judged from an abstract only. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Ziaei, "The Limits of Simulated Societies: How Post-Training and Survey Fine-Tuning Erase Cross-Cultural Variance," arXiv:2609.25760 (2026)
Base-to-SFT-to-DPO checkpoint trace on World Values Survey simulation; survey simulation only, not coding tasks. - Santurkar et al., "Whose Opinions Do Language Models Reflect?" ICML 2023
Base versus RLHF opinion shift on OpinionsQA; opinions, not measurement validity. - Kotte, "Two Wrongs, No Right: Auditing Social-Desirability Bias in LLM Annotators for Computational Social Science," arXiv:2606.12426 (2026)
Audits social-desirability bias in three 7B instruction-tuned annotators; no base-model contrast.
Search queries used
base model versus instruction-tuned public opinion survey responses alignment effect social science measurement same model familyeffect of RLHF instruction tuning on LLM text annotation stance detection coding accuracy base vs chat model comparisonpost-training stage checkpoints OLMo Tulu SFT DPO effect on annotation codingregion-specific fine-tune same base model political attitudes values survey
Provenance
RO-LD07-05 v1. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-07 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.