Research opportunity · AI in the research apparatus
Can synthetic respondents recover strong treatment-effect heterogeneity?
Earlier version (v1)
This is RO-LD07-04 as it was published at v1, kept unchanged. The current version is v2; its changes are listed there.
Question
In probability-sample survey experiments fielded after model training cutoffs and selected for substantial, well-powered moderation, do synthetic samples recover the sign and ranking of conditional treatment effects better than a model that assumes no heterogeneity?
Wrong or missing validation targetOne PhD chapter or paperLow costPartly addressed
Why it matters
The most-cited positive evidence shows LLMs forecast average effects well. Policy and theory usually turn on who responds differently, and current tests of interactions have little power because few archived experiments show strong moderation.
The debate lists this among what we still do not know: Can synthetic populations ever recover subgroup treatment effects and causal mechanisms out of distribution?
Evidence that motivates it
- LLM simulations can usefully forecast the relative pattern of treatment effects in some US survey-experiment archives, while systematically overstating effect magnitudes.
Forecasting the pattern of average effects works in some US archives but overstates magnitudes. - Synthetic respondents can match selected survey averages while failing to reproduce real-world variance, regression relationships, individual structure and subgroup heterogeneity.
Synthetic samples can match selected averages while failing to reproduce variance and subgroup heterogeneity. - Statistical realism of synthetic responses is not a reliable proxy for treatment-effect accuracy.
Response-level realism does not predict treatment-effect accuracy, so CATE recovery must be tested directly.
A possible design
Assemble post-cutoff experiments with pre-registered or published strong moderation (by partisanship, education, age, intersectional cells), including some outside the US. Generate persona-conditioned synthetic samples with several models. Compare estimated conditional effects with human estimates, using a no-heterogeneity benchmark and an honest power analysis, and report where synthetic samples beat, match or fall below that benchmark.
Data needed
Archives of recent survey experiments with microdata (for example TESS and journal replication files); post-2025 studies to avoid training contamination; API access to two or three models.
Feasibility
One PhD chapter or paper. Low: existing archives and modest API spend. The hard part is assembling enough post-cutoff experiments with strong moderation.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026; not yet checked in depth).
Ashokkumar et al. already test subgroup average effects (high accuracy) and pooled interactions (weak), but their ground truth contains few strongly moderated effects. Pew Data Labs (30 September 2026, after the evidence cut) found biased subgroup estimates and large cross-model divergence for descriptive polling. Recovery of strong, well-powered heterogeneity in experiments remains untested.
How strong this evidence is
- Who searched and checked: an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: none of the 4 nearest works below is recorded as judged from an abstract only. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Ashokkumar, Hewitt, Ghezae & Willer, "Large language models can predict the results of social science experiments," Nature 656:115–122 (2026)
Subgroup average effects predicted well; interaction effects predicted weakly, with only 6–15% of original effects significantly moderated. - Chapekis, Lau, Bestvater, Shah, Mercer & Smith, "How synthetic polling results change based on the AI model," Pew Research Center Data Labs (30 Sep 2026)
Digital twins of about 6,700 panelists err by 11–13 points on 2026 survey waves, with biased subgroup estimates and large divergence between models; descriptive items, not experiments. Published after LD-07's evidence cut. - Li & Ji, "Statistical realism is not evidence that LLMs can estimate treatment effects in social science experiments," arXiv:2604.02458 (2026)
Realism weakly related to treatment-effect accuracy across 62 countries; conditional effects are not the focus. - Peng, Gui, Brucks, … Toubia, "Digital Twins as Funhouse Mirrors: Five Key Distortions," arXiv:2509.19088 (2025; v5 2026)
Across 19 pre-registered studies, individual digital twins built from 500+ answers beat a generic LLM only slightly and show stereotyping, representation and ideological distortions; individual prediction, not conditional effects in experiments.
Search queries used
LLM simulate survey experiments heterogeneous treatment effects subgroups conditional average treatment effects human ground truthsynthetic respondents recover subgroup treatment effects TESS post training cutoff out-of-distribution experimentsheterogeneous treatment effects large language models simulated participants survey experiment replication arXiv 2026LLM digital twins predict individual-level treatment effects
Provenance
RO-LD07-04 v1. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-07 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.