Research opportunity · AI in the research apparatus
Do silicon samples work outside the North Atlantic, and in which language?
Earlier version (v1)
This is RO-LD07-08 as it was published at v1, kept unchanged. The current version is v2; its changes are listed there.
Question
Using Afrobarometer, Arab Barometer and Latinobarómetro items fielded after model training cutoffs, does synthetic-sample accuracy at country, subgroup and individual level depend on interview language (for example Swahili or Hausa versus English, Arabic versus English), and do LLM codes of political constructs in low-resource languages pass a replacement test against trained in-country coders?
Geographic or language coverageA PhD projectLow to medium costPartly addressed
Why it matters
The validation studies in this debate come mainly from the US, Europe and China, and this screen found few from Africa, the Arab region or Latin America. Researchers working in and on those regions, where survey budgets are often tight and cheap substitutes most tempting, need evidence about their own contexts.
Evidence that motivates it
- LLM simulations can usefully forecast the relative pattern of treatment effects in some US survey-experiment archives, while systematically overstating effect magnitudes.
Forecasting success comes from US archives; the claim would change with failure on non-US archives. - Chinese-language silicon-sample studies benchmarked against CGSS data find partial macro-level reproduction alongside representational bias, micro/macro/meso instability and cross-model differences; current systems do not warrant replacement of human respondents.
Locally grounded validation (here, Chinese CGSS) finds representational bias and instability. - Synthetic respondents can match selected survey averages while failing to reproduce real-world variance, regression relationships, individual structure and subgroup heterogeneity.
Subgroup and variance failures are documented in the contexts tested so far.
A possible design
Select recent waves of two or three regional barometers. Generate synthetic samples conditioned on respondent profiles, prompting in English and in the interview language. Compare marginal distributions, subgroup estimates and key associations with the human data. Separately, have trained in-country coders and LLMs code a political-text task in one or two low-resource languages and apply a formal replacement test.
Data needed
Public barometer microdata; API and open-weight models with multilingual support; a small budget for in-country coders.
Feasibility
A PhD project. Low to medium: survey data are free; coder time is the main cost. Strongest with a co-author or partner institution in the region.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026; not yet checked in depth).
Studies use the World Values Survey for a range of countries, including one comparing English with native-language prompting across ten countries, plus Chilean and Egypt/US benchmarks, and NLP benchmarks cover low-resource languages. No study found benchmarks silicon samples against Afrobarometer, Arab Barometer or Latinobarómetro, or validates low-resource-language coding of social-science constructs against trained in-country coders.
How strong this evidence is
- Who searched and checked: an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: none of the 5 nearest works below is recorded as judged from an abstract only. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Qu & Wang, "Performance and biases of Large Language Models in public opinion simulation," Humanities & Social Sciences Communications 11:1095 (2024)
World Values Survey in six countries including South Africa and Brazil, with native-language prompts for some countries; accuracy worse outside Western, English-speaking settings; regional barometers not used. - González-Bustamante, Verelst & Cisternas, "Emulating Public Opinion: A Proof-of-Concept of AI-Generated Synthetic Survey Responses for the Chilean Case," arXiv:2509.09871 (2025)
Chilean probability survey benchmark; item-level accuracy varies widely. - AlKhamissi et al., "Investigating Cultural Alignment of Large Language Models," arXiv:2402.13231 (2024)
Egypt/US survey simulation; alignment better when prompting in Arabic; one country pair. - Kastrati et al., "Do language families matter?" Frontiers in Artificial Intelligence 9 (2026)
LLM sentiment classification in 36 low-resource languages against prior models, not human coders or social-science constructs. - Liu, Kaneko & Chu, "On the Alignment of Large Language Models with Global Human Opinion," arXiv:2509.01418 (2025)
World Values Survey framework; compares English with native-language prompting for 8 languages and 10 countries and finds language steering effective; no regional barometers, no validation against local coders.
Search queries used
silicon samples Afrobarometer LLM simulate survey responses African countriesArab Barometer OR Latinobarómetro large language model synthetic respondents alignment public opinion arXivWorld Values Survey LLM cultural alignment country simulation Global South native language promptingLLM annotation validity African languages Swahili Hausa Yoruba hate speech sentiment GPT zero-shot vs human annotators social science
Provenance
RO-LD07-08 v1. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-07 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.