Research opportunity · AI in the research apparatus
Do silicon samples work outside the North Atlantic, and in which language?
Question
Using Afrobarometer, Arab Barometer and Latinobarómetro items fielded after model training cutoffs, does synthetic-sample accuracy at country, subgroup and individual level depend on interview language (for example Swahili or Hausa versus English, Arabic versus English), and do LLM codes of political constructs in low-resource languages pass a replacement test against trained in-country coders?
Geographic or language coverageA PhD projectLow to medium costPartly addressed
Why it matters
The validation studies in this debate come mainly from the US, Europe and China, and this screen found few from Africa, the Arab region or Latin America. Researchers working in and on those regions, where survey budgets are often tight and cheap substitutes most tempting, need evidence about their own contexts.
Evidence that motivates it
- LLM simulations can usefully forecast the relative pattern of treatment effects in some US survey-experiment archives, while systematically overstating effect magnitudes.
Forecasting success comes from US archives; the claim would change with failure on non-US archives. - Chinese-language silicon-sample studies benchmarked against CGSS data find partial macro-level reproduction alongside representational bias, micro/macro/meso instability and cross-model differences; current systems do not warrant replacement of human respondents.
Locally grounded validation (here, Chinese CGSS) finds representational bias and instability. - Synthetic respondents can match selected survey averages while failing to reproduce real-world variance, regression relationships, individual structure and subgroup heterogeneity.
Subgroup and variance failures are documented in the contexts tested so far.
A possible design
Select recent waves of two or three regional barometers. Generate synthetic samples conditioned on respondent profiles, prompting in English and in the interview language. Compare marginal distributions, subgroup estimates and key associations with the human data. Separately, have trained in-country coders and LLMs code a political-text task in one or two low-resource languages and apply a formal replacement test.
Data needed
Public barometer microdata; API and open-weight models with multilingual support; a small budget for in-country coders.
Feasibility
A PhD project. Low to medium: survey data are free; coder time is the main cost. Strongest with a co-author or partner institution in the region.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026, checked again on 13 October 2026).
Studies use the World Values Survey and other global cultural surveys for a range of countries, including one comparing English with native-language prompting across ten countries (Liu et al., AAAI 2026) and fine-tuned models tested on unseen countries (Cao et al., NAACL 2025), plus Chilean and Egypt/US benchmarks, and NLP benchmarks such as AfroBench cover African languages. No study found benchmarks silicon samples against Afrobarometer, Arab Barometer or Latinobarometro by interview language, or validates low-resource-language coding of social-science constructs against trained in-country coders.
How strong this evidence is
- Who searched and checked: Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert; an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: 1 of the 8 nearest works below was judged from an abstract, summary or search listing only, and 1 has details that could not be confirmed; each is marked. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist. Literature check of 2026-10-13: Full texts of Cao et al. and SimBench were not read, so their dataset lists (and whether a regional barometer appears) are unconfirmed. Google Scholar, SSRN search and regional journals (e.g. African or Latin American political science journals) were not searched directly; the Spanish and Arabic queries returned only news and survey-organisation pages. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Qu & Wang, "Performance and biases of Large Language Models in public opinion simulation," Humanities & Social Sciences Communications 11:1095 (2024)
World Values Survey in six countries including South Africa and Brazil, with native-language prompts for some countries; accuracy worse outside Western, English-speaking settings; regional barometers not used. - González-Bustamante, Verelst & Cisternas, "Emulating Public Opinion: A Proof-of-Concept of AI-Generated Synthetic Survey Responses for the Chilean Case," arXiv:2509.09871 (2025)
Chilean probability survey benchmark; item-level accuracy varies widely. - AlKhamissi et al., "Investigating Cultural Alignment of Large Language Models," arXiv:2402.13231 (2024)
Egypt/US survey simulation; alignment better when prompting in Arabic; one country pair. - Kastrati et al., "Do language families matter?" Frontiers in Artificial Intelligence 9 (2026)
LLM sentiment classification in 36 low-resource languages against prior models, not human coders or social-science constructs. - Liu, Kaneko & Chu, "On the Alignment of Large Language Models with Global Human Opinion," arXiv:2509.01418 (2025); published in Proceedings of AAAI 2026
World Values Survey framework; compares English with native-language prompting for 8 languages and 10 countries and finds language steering effective; no regional barometers, no validation against local coders. - Cao, Liu, Arora, Augenstein, Röttger & Hershcovich, "Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations," NAACL 2025 (long), pp. 3141-3154 Some details not confirmed
Fine-tunes LLMs on country-level response distributions from two global cultural surveys and tests on unseen questions, countries and one unseen survey; models still struggle on unseen questions. The abstract page does not name the surveys, the countries or the prompt language; it does not mention regional barometers, interview-language comparisons or coder validation (not confirmed from full text). - Hu, Baumann, Lupo, Collier, Hovy & Röttger, "SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors," arXiv:2510.17516 (v4, Apr 2026)
Benchmark of 20 datasets of human behaviour and opinion across a global participant pool; models struggle most with specific demographic groups. The abstract does not list the datasets, so whether any regional barometer is included was not confirmed; it is not a language-of-prompt or in-country-coder study. - AfroBench, "AfroBench: How Good are Large Language Models on African Languages?" Findings of ACL 2025; arXiv:2311.07978 Judged from an abstract, summary or search listing only
Seen in search results only (title, venue, identifier). An NLP benchmark of LLM performance on African-language tasks; not survey simulation and not validation of political-construct coding against trained in-country coders. Author list not checked.
Literature check
Stands as first published. Seven new queries (English, Spanish and Arabic; preprint- and journal-restricted; forward search on Liu, Kaneko & Chu) found no study that benchmarks silicon samples against Afrobarometer, Arab Barometer or Latinobarometro, or that compares interview-language with English prompting on those data. Cross-national simulation work stays on global cultural surveys (Cao et al., NAACL 2025; Liu et al., AAAI 2026) or general benchmarks (SimBench), and African-language LLM work is NLP benchmarking (AfroBench) rather than validation against trained in-country coders of social-science constructs.
Searched: web (extended); arxiv.org; osf.io; ssrn.com; socarxiv.org; aclanthology.org; cambridge.org; journals.sagepub.com; academic.oup.com; tandfonline.com; sciencedirect.com; Spanish- and Arabic-language web. Checked by Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert.
The screen before this check
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026).
Studies use the World Values Survey for a range of countries, including one comparing English with native-language prompting across ten countries, plus Chilean and Egypt/US benchmarks, and NLP benchmarks cover low-resource languages. No study found benchmarks silicon samples against Afrobarometer, Arab Barometer or Latinobarómetro, or validates low-resource-language coding of social-science constructs against trained in-country coders.
Search queries used
silicon samples Afrobarometer LLM simulate survey responses African countriesArab Barometer OR Latinobarómetro large language model synthetic respondents alignment public opinion arXivWorld Values Survey LLM cultural alignment country simulation Global South native language promptingLLM annotation validity African languages Swahili Hausa Yoruba hate speech sentiment GPT zero-shot vs human annotators social science
Literature check, 13 October 2026:
Afrobarometer large language model synthetic respondents survey simulation African languagesLLM silicon sampling Arab Barometer Latinobarometro public opinion prediction language of prompt [allowed_domains: arxiv.org, osf.io, ssrn.com, papers.ssrn.com, socarxiv.org]large language models synthetic survey responses Global South country subgroup accuracy multilingual prompting political attitudes [allowed_domains: aclanthology.org, cambridge.org, journals.sagepub.com, academic.oup.com, tandfonline.com, sciencedirect.com]LLM annotation political text African languages compared with native-speaker expert coders validity social science Swahili Hausa Amharicmodelos de lenguaje encuestas sintéticas Latinobarómetro opinión pública América Latina simulación"On the Alignment of Large Language Models with Global Human Opinion" OR Afrobarometer OR "Arab Barometer" LLM survey simulation 2026نماذج اللغة الكبيرة محاكاة استطلاعات الرأي الباروميتر العربي
Provenance
RO-LD07-08 v2. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-07 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.
Changes
- v2, 13 October 2026: Literature check added: a second, deeper search for existing work, with its queries, new nearest work and limits. The question and verdict stand. See v1 as published.