Research opportunity · AI in the research apparatus
Can synthetic respondents recover strong treatment-effect heterogeneity?
Question
In probability-sample survey experiments fielded after model training cutoffs and selected for substantial, well-powered moderation, do synthetic samples recover the sign and ranking of conditional treatment effects better than a model that assumes no heterogeneity?
Wrong or missing validation targetOne PhD chapter or paperLow costPartly addressed
Why it matters
The most-cited positive evidence shows LLMs forecast average effects well. Policy and theory usually turn on who responds differently, and current tests of interactions have little power because few archived experiments show strong moderation.
The debate lists this among what we still do not know: Can synthetic populations ever recover subgroup treatment effects and causal mechanisms out of distribution?
Evidence that motivates it
- LLM simulations can usefully forecast the relative pattern of treatment effects in some US survey-experiment archives, while systematically overstating effect magnitudes.
Forecasting the pattern of average effects works in some US archives but overstates magnitudes. - Synthetic respondents can match selected survey averages while failing to reproduce real-world variance, regression relationships, individual structure and subgroup heterogeneity.
Synthetic samples can match selected averages while failing to reproduce variance and subgroup heterogeneity. - Statistical realism of synthetic responses is not a reliable proxy for treatment-effect accuracy.
Response-level realism does not predict treatment-effect accuracy, so CATE recovery must be tested directly.
A possible design
Assemble post-cutoff experiments with pre-registered or published strong moderation (by partisanship, education, age, intersectional cells), including some outside the US. Generate persona-conditioned synthetic samples with several models. Compare estimated conditional effects with human estimates, using a no-heterogeneity benchmark and an honest power analysis, and report where synthetic samples beat, match or fall below that benchmark.
Data needed
Archives of recent survey experiments with microdata (for example TESS and journal replication files); post-2025 studies to avoid training contamination; API access to two or three models.
Feasibility
One PhD chapter or paper. Low: existing archives and modest API spend. The hard part is assembling enough post-cutoff experiments with strong moderation.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026, checked again on 13 October 2026).
Ashokkumar et al. already test subgroup average effects (high accuracy) and pooled interactions (weak), but their ground truth contains few strongly moderated effects. Pew Data Labs (30 September 2026, after the evidence cut) found biased subgroup estimates and large cross-model divergence for descriptive polling. A 2026 preprint notes that moderation patterns vary across models in a single survey experiment, without a formal test. Recovery of strong, well-powered heterogeneity in post-cutoff experiments against a no-heterogeneity benchmark remains untested.
How strong this evidence is
- Who searched and checked: Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert; an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: 3 of the 7 nearest works below were judged from an abstract, summary or search listing only, and 1 has details that could not be confirmed; each is marked. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist. Literature check of 2026-10-13: Full texts of the new preprints were not read (abstracts only). Forward citations of the Nature paper could not be listed with web search; only search-visible follow-ups were checked. The Pew Data Labs and Peng et al. entries were not re-checked. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Ashokkumar, Hewitt, Ghezae & Willer, "Large language models can predict the results of social science experiments," Nature 656:115–122 (2026)
Subgroup average effects predicted well; interaction effects predicted weakly, with only 6–15% of original effects significantly moderated. - Chapekis, Lau, Bestvater, Shah, Mercer & Smith, "How synthetic polling results change based on the AI model," Pew Research Center Data Labs (30 Sep 2026)
Digital twins of about 6,700 panelists err by 11–13 points on 2026 survey waves, with biased subgroup estimates and large divergence between models; descriptive items, not experiments. Published after LD-07's evidence cut. - Li & Ji, "Statistical realism is not evidence that LLMs can estimate treatment effects in social science experiments," arXiv:2604.02458 (2026)
Realism weakly related to treatment-effect accuracy across 62 countries; conditional effects are not the focus. - Peng, Gui, Brucks, … Toubia, "Digital Twins as Funhouse Mirrors: Five Key Distortions," arXiv:2509.19088 (2025; v5 2026)
Across 19 pre-registered studies, individual digital twins built from 500+ answers beat a generic LLM only slightly and show stereotyping, representation and ideological distortions; individual prediction, not conditional effects in experiments. - Hoq & Weninger, "Evaluating LLMs as Human Surrogates in Controlled Experiments," arXiv:2604.15329 (v1, 8 March 2026) Judged from an abstract, summary or search listing only
Abstract read: compares LLM and human responses in a survey experiment and reports that effect magnitudes and moderation patterns vary across models. One experiment; the abstract describes no formal heterogeneity test, no no-heterogeneity benchmark and no post-cutoff selection. - "Passing Coarse Marginal Checks Can Be Cheap: Persona Mixtures and Imprecise Treatment-Response Estimates in an LLM Persona Panel," arXiv:2608.00979 (2026; authors not confirmed) Judged from an abstract, summary or search listing onlySome details not confirmed
Abstract read: one fixed model-prompt persona panel passes registered marginal criteria without precisely estimating the treatment-response object; no human comparison of subgroup or moderator effects. Supports the need for a stricter heterogeneity test. - Sen, Ahnert, von der Heyde, Lasser, Weiß & Strohmaier, "Total Simulated Survey Error: Designing and Diagnosing Survey Responses from Large Language Models," arXiv:2609.10280 (9 September 2026) Judged from an abstract, summary or search listing only
Abstract read: error framework for LLM survey simulation with case studies; the abstract does not mention experiments or treatment heterogeneity. Methodological context only.
Literature check
Stands as first published. Six new queries (preprint, journal, forward and plain web) found no study that selects post-cutoff survey experiments with strong, well-powered moderation and tests synthetic samples against a no-heterogeneity benchmark. Ashokkumar et al. (Nature 2026) and Li & Ji (arXiv 2026) remain the closest. New preprints touch moderation only in passing: Hoq & Weninger (arXiv:2604.15329) note that moderation patterns vary across models in one survey experiment without a formal heterogeneity test, and a persona-panel study (arXiv:2608.00979) reports only marginal contrasts.
Searched: web (extended and standard); arxiv.org; osf.io; socarxiv.org; ssrn.com; cambridge.org; journals.sagepub.com; academic.oup.com; tandfonline.com; nature.com; pnas.org; onlinelibrary.wiley.com; direct fetches: arxiv.org abstracts 2602.20383, 2609.10280, 2608.00979, 2604.15329; existing nearest-work pages. Checked by Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert.
The screen before this check
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026).
Ashokkumar et al. already test subgroup average effects (high accuracy) and pooled interactions (weak), but their ground truth contains few strongly moderated effects. Pew Data Labs (30 September 2026, after the evidence cut) found biased subgroup estimates and large cross-model divergence for descriptive polling. Recovery of strong, well-powered heterogeneity in experiments remains untested.
Search queries used
LLM simulate survey experiments heterogeneous treatment effects subgroups conditional average treatment effects human ground truthsynthetic respondents recover subgroup treatment effects TESS post training cutoff out-of-distribution experimentsheterogeneous treatment effects large language models simulated participants survey experiment replication arXiv 2026LLM digital twins predict individual-level treatment effects
Literature check, 13 October 2026:
LLM simulated survey experiments heterogeneous treatment effects moderators partisanship silicon subjects interaction effects [allowed_domains: arxiv.org, osf.io, ssrn.com, papers.ssrn.com, socarxiv.org]large language models predict conditional average treatment effects survey experiments subgroup heterogeneity accuracy 2026synthetic respondents LLM conjoint experiment subgroup effects replication heterogeneity political science [allowed_domains: cambridge.org, journals.sagepub.com, academic.oup.com, tandfonline.com, nature.com, pnas.org, onlinelibrary.wiley.com]"Large language models can predict the results of social science experiments" interaction effects moderation subgroups follow-up 2026 preprintLLM simulation recover treatment effect heterogeneity by party education age survey experiment ground truth benchmark homogeneous effect [allowed_domains: arxiv.org, osf.io, socarxiv.org, ssrn.com, papers.ssrn.com]"Statistical realism is not evidence" LLMs treatment effects Li Ji
Provenance
RO-LD07-04 v2. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-07 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.
Changes
- v2, 13 October 2026: Literature check added: a second, deeper search for existing work, with its queries, new nearest work and limits. The question and verdict stand. See v1 as published.