Research opportunity · AI in the research apparatus
Does post-training change what an LLM measures when it codes text?
Question
Holding base weights fixed, how much do instruction tuning, preference tuning and region-specific fine-tunes shift coding validity and downstream estimates on stance and attitude annotation tasks, and do the shifts track the fine-tune's target culture?
Causal mechanism not isolatedOne PhD chapter or paperLow costPartly addressed
Why it matters
Observed differences between providers are routinely attributed to alignment or regulation, but existing comparisons cannot separate those from differences in base models and training data. Within-family checkpoint contrasts can isolate the effect of post-training on measurement itself.
The debate lists this among what we still do not know: How much do jurisdictional alignment regimes causally change social-science measurement, rather than merely refusal behavior?
Evidence that motivates it
- Provider and jurisdictional context can be associated with materially different refusal and accuracy patterns on politically sensitive prompts; existing evidence does not isolate regulation as the causal mechanism.
Provider and jurisdiction are associated with different response patterns, but the causal mechanism is not isolated. - Chinese-language silicon-sample studies benchmarked against CGSS data find partial macro-level reproduction alongside representational bias, micro/macro/meso instability and cross-model differences; current systems do not warrant replacement of human respondents.
Chinese-language silicon-sample studies show cross-model differences whose source is unknown.
A possible design
Use model families with public checkpoints at each post-training stage (base, supervised fine-tuning, preference tuning) and community or regional fine-tunes of the same base. Run established stance and attitude coding tasks with expert gold labels in two or more languages. Estimate the stage effect on accuracy, bias by target group and downstream coefficients, holding prompts and decoding fixed.
Data needed
Open checkpoints with documented training stages; annotated political-text corpora in at least two languages; GPU time.
Feasibility
One PhD chapter or paper. Low: open-weight checkpoints and existing labelled corpora.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026, checked again on 13 October 2026).
For survey simulation, Ziaei (22 September 2026) traces base-to-tuned checkpoints and finds supervised tuning roughly halves response dispersion; Santurkar et al. contrast base and RLHF opinions; a MENA study compares alignment stages for cultural-value survey responses, and a September 2026 preprint finds task faithfulness eroding most at the DPO stage. No study found applies the checkpoint design to annotation and coding validity against expert gold labels, or to regional fine-tunes of a shared base.
How strong this evidence is
- Who searched and checked: Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert; an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: 3 of the 6 nearest works below were judged from an abstract, summary or search listing only, and 1 has details that could not be confirmed; each is marked. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist. Literature check of 2026-10-13: Abstracts only for the new preprints; full texts may contain annotation-style tasks not visible in the abstracts. The Chinese-language query returned general explainers and one alphaXiv page on factual sycophancy, not annotation studies. Santurkar et al. was not re-checked. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Ziaei, "The Limits of Simulated Societies: How Post-Training and Survey Fine-Tuning Erase Cross-Cultural Variance," arXiv:2609.25760 (2026)
Checkpoint trace from base model through SFT, DPO and GRPO stages on World Values Survey simulation; survey simulation only, not coding tasks. - Santurkar et al., "Whose Opinions Do Language Models Reflect?" ICML 2023
Base versus RLHF opinion shift on OpinionsQA; opinions, not measurement validity. - Kotte, "Two Wrongs, No Right: Auditing Social-Desirability Bias in LLM Annotators for Computational Social Science," arXiv:2606.12426 (2026)
Audits social-desirability bias in three 7B instruction-tuned annotators; no base-model contrast. - Zahraei, Tur, Hakkani-Tür & Asgari, "The Alignment Veto: How Safety Training Suppresses Cultural Knowledge in LLMs," arXiv:2510.13154 (v1 15 October 2025; v2 18 June 2026) Judged from an abstract, summary or search listing only
Abstract read: 26 models compared with 1.53M survey responses in 16 MENA countries; internal distributions match human data better than free-text answers ('suppression'), with comparisons across alignment stages and a DPO-stage feature in Tulu-3-8B. Survey-response alignment, not annotation validity against gold labels; does not test regional fine-tunes of a shared base, per the abstract. - Zahraei, Singh, Tur & Hakkani-Tür, "Emergent Unfaithfulness: How Alignment Training Causes Language Models to Silently Override Task Faithfulness," arXiv:2610.00568 (30 September 2026) Judged from an abstract, summary or search listing only
Abstract read: intermediate post-training checkpoints show models increasingly override task instructions, with the gap growing most and becoming least visible at the DPO stage, on a FaithConflict dataset. Shows a stage effect on task faithfulness that a coding-validity study would expect; the abstract does not mention annotation, classification or stance coding. - "Evaluating Decision Models for Text Annotation in Computational Social Science," arXiv:2609.24574 (2026; authors not confirmed) Judged from an abstract, summary or search listing onlySome details not confirmed
Abstract read: compares a commercial decision model and open-weight counterparts against 19 LLMs on zero-shot annotation; no base versus tuned checkpoints of the same model are mentioned.
Literature check
Stands as first published. Six new queries found no study that holds base weights fixed and measures how SFT, preference tuning and regional fine-tunes shift coding validity and downstream estimates on stance or attitude annotation. Checkpoint traces now exist for survey simulation (Ziaei 2026), for cultural-value survey responses across alignment stages in MENA countries (Zahraei et al., arXiv:2510.13154) and for task faithfulness, which they find is amplified during DPO (Zahraei et al., arXiv:2610.00568). None of these tests annotation validity against expert gold labels or regional fine-tunes of a shared base.
Searched: web (extended and standard); arxiv.org; osf.io; socarxiv.org; aclanthology.org; dl.acm.org; cambridge.org; journals.sagepub.com; ojs.aaai.org; Chinese-language web; direct fetches: arxiv.org abstracts 2609.24574, 2510.13154, 2610.00568; existing nearest-work pages. Checked by Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert.
The screen before this check
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026).
For survey simulation, Ziaei (22 September 2026) traces base-to-tuned checkpoints and finds supervised tuning roughly halves response dispersion; Santurkar et al. contrast base and RLHF opinions. No study found applies the same checkpoint design to annotation and coding validity or to regional fine-tunes of a shared base.
Search queries used
base model versus instruction-tuned public opinion survey responses alignment effect social science measurement same model familyeffect of RLHF instruction tuning on LLM text annotation stance detection coding accuracy base vs chat model comparisonpost-training stage checkpoints OLMo Tulu SFT DPO effect on annotation codingregion-specific fine-tune same base model political attitudes values survey
Literature check, 13 October 2026:
base model versus instruction-tuned LLM text annotation accuracy stance detection comparison same model family [allowed_domains: arxiv.org, osf.io, socarxiv.org]effect of alignment RLHF DPO on LLM annotation zero-shot classification computational social science checkpoints [allowed_domains: aclanthology.org, dl.acm.org, cambridge.org, journals.sagepub.com, ojs.aaai.org]regional fine-tune same base model annotation bias stance political Chinese Arabic fine-tuned Llama Qwen comparison cultural alignment downstream coding指令微调 基座模型 对比 文本标注 立场检测 准确率 偏好对齐 影响 社会科学"The Limits of Simulated Societies" Ziaei post-training annotationOLMo Tulu intermediate checkpoints SFT DPO effect on classification annotation label distribution bias social science measurement [allowed_domains: arxiv.org, aclanthology.org, osf.io]
Provenance
RO-LD07-05 v2. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-07 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.
Changes
- v2, 13 October 2026: Literature check added: a second, deeper search for existing work, with its queries, new nearest work and limits. The question and verdict stand. See v1 as published.