Research opportunity · Persuasion, Intermediation & the Information Order
Are AI search answers less faithful to their cited sources outside US English?
Question
For matched public-interest queries (news, health, elections) issued in several languages and country markets, how often are claims in Google AI Overviews and AI Mode unsupported by the pages they cite, judged claim by claim with native-speaker validation, and how do these rates differ from a US English baseline run through the same pipeline?
Geographic or language coverageMaster's thesisLow to medium costPartly addressed
Why it matters
Citation fidelity is one of the few directly measurable qualities of AI intermediation, and it matters because citations raise trust even when they are wrong. The only large claim-level audit of AI Overviews is US-localized, and multilingual audits of chatbots check sourcing by journalist judgement rather than claim by claim. If fidelity is worse in smaller languages, the users with the fewest alternative sources may get the least reliable intermediation.
The debate lists this among what we still do not know: stronger non-English empirical coverage
Evidence that motivates it
- A 2026 Google AI Overviews audit found 13.7% overall activation, 64.7% on question-form queries, nearly 30% of cited domains absent from the co-displayed first results page, and 11.0% of atomic claims unsupported by cited pages.
Reports 11.0% of atomic claims unsupported by cited pages in a 2026 AI Overviews audit; a cross-market replication is named as what would change it, and it does not license claims about user beliefs. - A 2025 EBU/BBC-led audit across 18 countries and 14 languages found significant issues in 45% of AI-assistant news answers, serious sourcing problems in 31% and major accuracy issues in 20%.
A multilingual audit of AI assistants (18 countries, 14 languages) found serious sourcing problems in 31% of news answers, but covered chatbots rather than answers built into search and does not give a platform-wide error rate. - In a preregistered U.S. experiment, citations increased trust in AI-search answers even when incorrect or hallucinated, while uncertainty highlighting reduced trust and sharing.
Citations increased trust in AI-search answers even when incorrect, which is why unsupported citations matter; the claim does not establish real-world reliance levels.
A possible design
Build a matched query set (about 500 queries per language) from trending searches and fixed public-interest topics, translated and localized by native speakers, in four to six languages spanning high- and low-resource tiers (for example English, German, Spanish, Polish, Hindi, Swahili). Collect AI Overviews and AI Mode answers with their cited pages from in-country vantage points (residential proxies or volunteers) over four to eight weeks. Extract atomic claims and judge support against cited pages with an LLM pipeline validated by native-speaker annotators on a stratified sample in every language, reporting per-language agreement. The primary outcome is the unsupported-claim rate. Activation rates, the share of local-language and local-country sources (compared with the recent health audit by Zha and Chang rather than presented as new) and overlap with the co-displayed results are secondary. Compare them using the same pipeline on a US English arm as the benchmark.
Data needed
In-country collection points; native-speaker annotators (paid) for translation and validation; page archiving for cited sources; an LLM or natural-language-inference model for claim checking.
Feasibility
Master's thesis. Low to medium: proxies and model calls a few hundred to a few thousand US dollars; annotator pay is the main cost (roughly US$2,000 to 6,000 for two to three languages). A two-language version suits a master's thesis. Automated collection may conflict with platform terms; check the legal position and prefer volunteer or research-access routes. Claim-checking models are weaker in low-resource languages, so human validation per language is essential, not optional.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026, checked again on 13 October 2026).
Claim-level checks of AI answers in search against their cited pages exist only for English queries from US locations (Xu et al.; Huang et al.; a journalistic Oumi analysis for the New York Times), and only Xu et al. report any human validation. Cross-country and cross-language audits now document where AI Overviews draw their sources (Zha and Chang for health in 12 country settings and 4 languages; Li et al. for Baidu and Google in English and Chinese; Zhen et al. for Chinese platforms; Weinbrand for Australian political search) but none checks whether claims are supported by the cited pages, and none covers AI Mode. A human-validated, claim-level fidelity comparison of AI answers in search across languages and markets was not found.
How strong this evidence is
- Who searched and checked: Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert; an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: 3 of the 11 nearest works below were judged from an abstract, summary or search listing only, and 1 has details that could not be confirmed; each is marked. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search in English on the date shown; searches in other languages were not run, so audits published only in those languages (for example by national regulators or newsrooms) may have been missed. Google Scholar and paywalled journals were not searched. Absence of a hit is not proof that the work does not exist. Literature check of 2026-10-13: ScienceDirect, SSRN and the New York Times could not be opened; the Oumi analysis was read through a trade-press summary. German and Chinese searches were run, but audits published only in other languages (for example by national regulators) may have been missed. The web-search budget for this session ran out during later checks. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Xu, Iqbal & Montgomery, "Measuring Google AI Overviews: Activation, Source Quality, Claim Fidelity, and Publisher Impact," arXiv:2605.14021 (13 May 2026)
55,393 trending queries from US-localized Google Trends over 40 days (March to April 2026), collected from a US server; 98,020 atomic claims checked by an LLM verifier, validated by two annotators on a sample of 100 verdicts; 11.0% unsupported. Single market; does not test other languages or countries. - Aral, Li & Zuo, "The Rise of AI Search: Implications for Information Markets and Human Judgement at Scale," arXiv:2602.13415 (13 Feb 2026)
Collects AI and traditional results in 243 countries (location set through a search API) for queries with non-English ones excluded; compares response variety, source popularity, credibility and lean. No claim-level check against cited pages and no non-English queries. - Nguyen, Noorily, Flathers, Notsu, Ospina-Pinillos, Nguyen, Clark, Keane, Thompson & Torous, "Sources of Truth: A Multi-Platform, Multilingual Audit of Citations in AI Mental Health Information Queries," arXiv:2609.00319 (31 Aug 2026)
15,942 citations from ChatGPT, Perplexity and Google AI Overview on 20 English mental-health questions, a subset of three translated into six further languages; non-English queries got fewer citations and were less often directed to language-appropriate resources. Classifies citation types; the abstract does not describe checking whether claims are supported by cited pages. - European Broadcasting Union & BBC, "News Integrity in AI Assistants: An International PSM Study" (EBU, 21 Oct 2025) Judged from an abstract, summary or search listing only
Journalists from 22 public-service media organisations in 18 countries rated over 3,000 answers from ChatGPT, Copilot, Gemini and Perplexity in 14 languages. Covers chatbots, not AI answers in search, and judges sourcing overall rather than claim by claim. Read through the EBU report page and a summary, not the full report PDF. - Zha & Chang, "Who Anchors AI Overviews in Health? Baidu, Google, and the Geography of Authority," arXiv:2609.06798 (6 Sep 2026)
120 health queries from Google Trends, run across country and language settings to give 1,920 results, in 12 country settings (Australia, Canada, China, India, Japan, Malaysia, Nigeria, Norway, Singapore, South Korea, UK, US) and English plus Chinese, Korean and Japanese, collected through SerpApi location parameters; Google AI Overviews, with China queried through Baidu; no AI Mode. Smaller, less localized countries get fewer domestic sources, and native-language queries raise the local-source share 3.5 to 13.5 times. Citation analysis is at domain level; the paper states it did not check whether claims are supported by cited pages and names that as future work. Covers the localization half of this opportunity for health. - Li, Guan, Cai, Liu & Pierri, "Auditing Source Exposure in Baidu and Google AI Search," arXiv:2609.24407 (21 Sep 2026)
Cross-lingual audit of Baidu and Google AI overviews using MS MARCO queries in English and Chinese translation; overview availability and visible source domains differ sharply across platform-language settings while answers are semantically similar. The abstract describes no check of claims against cited pages. - Zhen, Liu, Zhang & Niu, "What Do Chinese-Language Generative Search Engines Cite and Surface? A Large-Scale Empirical Study," arXiv:2607.15771 (17 Jul 2026)
614 queries run three times on four Chinese generative search platforms (web and app), 160,860 citation records; analyses citation behaviour and entity exposure. Reports that about 71% of contact-information exposures could not be matched to cited page text, but the abstract describes no general claim-support evaluation. Chinese platforms, not Google. - Huang, Goyal, Saha & Chandrasekharan, "Answer Bubbles: Information Exposure in AI-Mediated Search," arXiv:2603.16138 (v2 28 Aug 2026)
Measures how well AI Overviews and other generative search summaries are covered by their cited sources, using atomic content units and an NLI entailment model, on 11,000 English Natural Questions queries; AI Overviews queried from Urbana, Illinois. No human validation of the entailment step and no other languages or markets. - Oumi analysis commissioned by The New York Times, reported in "Google AI Overviews: etwa 91 Prozent der Antworten sind richtig," SEO Südwest (9 Apr 2026), citing a New York Times article of 7 Apr 2026 Judged from an abstract, summary or search listing only
Journalistic, not peer-reviewed: 4,326 SimpleQA questions run as Google searches; 91% of AI Overview answers correct under Gemini 3, but 56% of correct answers were not fully supported by the linked pages. Read only through the German trade-press summary; the NYT article (https://www.nytimes.com/2026/04/07/technology/google-ai-overviews-accuracy.html) was not opened. Apparently English queries; no cross-language comparison. - Suzgun, Shen, Bianchi, Spangher, Icard, Ho, Jurafsky & Zou, "Evaluating Commercial AI Chatbots as News Intermediaries," arXiv:2605.22785 (21 May 2026)
2,100 factual questions from six BBC regional services (including Arabic, Hindi, Russian, Turkish) put to six chatbots over 14 days in February 2026; reports Anglophone retrieval bias (Hindi queries citing English Wikipedia more than any Hindi outlet). Chatbots, not AI answers in search; the abstract does not say claims were checked against cited pages. - Weinbrand, "A search changer: auditing Google's AI overviews interface in political and news search," ScienceDirect pii S0736585326000535 (2026) Judged from an abstract, summary or search listing onlySome details not confirmed
Australian audit of more than 17,000 political and news searches (desktop and mobile, April to June 2025) mapping AI Overview interface elements and activation; flags weak inline sourcing but, per the ADM+S release, does not check claims against cited pages. Journal page blocked; details read from the release. The pii prefix matches the ISSN of Telematics and Informatics, so that is the likely journal (inferred, not confirmed).
Literature check
Narrowed to what the literature leaves open. The local-language and local-country sourcing part of the question is now partly answered: Zha and Chang audit Google and Baidu AI Overviews for health queries in 12 country settings and 4 languages and find that native-language queries raise locally sourced citations 3.5 to 13.5 times, Li et al. compare Baidu and Google overview source exposure across English and Chinese, and Zhen et al. map citations in Chinese generative search. None of these checks whether claims are supported by the cited pages (Zha and Chang name claim-level checking as future work), and the only grounding checks found (Xu et al.; Huang et al.; the Oumi analysis for the New York Times) are English and US-located. The opportunity should be narrowed to claim-level, human-validated fidelity across languages and markets, with source localization as a secondary measure.
Searched: web (extended); arxiv.org; osf.io; ssrn.com; aclanthology.org; dl.acm.org; German-language web; Chinese-language web. Checked by Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert.
The screen before this check
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026).
The claim-level fidelity audit (Xu et al.) uses trending queries from the US-localized Google Trends dashboard, collected from a US cloud server. A 243-country audit (Aral et al.) measures activation and source mix for English queries only, without claim-level support checks. A multilingual mental-health audit (Nguyen et al.) classifies citations across platforms and found fewer and less language-appropriate citations for non-English queries, but only three questions were translated and it does not check claims against cited pages. The EBU/BBC study covers chatbots, not AI answers in search. A human-validated, claim-level fidelity comparison of AI answers in search across languages and markets was not found.
Search queries used
audit Google AI Overviews citations unsupported claims atomic claims 2026 activation ratemultilingual audit AI search answers citation accuracy non-English languages Perplexity AI Overviews sources faithfulnessEBU BBC News Integrity in AI Assistants report 18 countries 14 languages 45%AI Overviews audit German OR Spanish OR Hindi OR Portuguese language cross-country comparison sources accuracy study
Literature check, 13 October 2026:
"A search changer" auditing Google's AI overviews interface political news search Telecommunications Policycross-lingual audit AI Overviews factual accuracy citation support non-English queries [allowed_domains: arxiv.org, osf.io, ssrn.com]multilingual generative search engine verifiability citation recall precision languages evaluation [allowed_domains: aclanthology.org, dl.acm.org]Google AI Overviews Fehler Quellen Prüfung Studie deutschsprachig Behauptungen belegtAI Overviews hallucination grounding audit Spanish OR Portuguese OR Hindi OR Indonesian queries citations unsupported 2026"Measuring Google AI Overviews" Xu Iqbal Montgomery claim fidelity follow-up cross-marketAI搜索 生成式回答 引用来源 准确性 审计 百度 谷歌 AI概览 研究
Provenance
RO-LD10-03 v2. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-10 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.
Changes
- v2, 13 October 2026: Narrowed after the literature check. The local-language and local-country sourcing part of the question is now partly answered: Zha and Chang audit Google and Baidu AI Overviews for health queries in 12 country settings and 4 languages and find that native-language queries raise locally sourced citations 3.5 to 13.5 times, Li et al. compare Baidu and Google overview source exposure across English and Chinese, and Zhen et al. map citations in Chinese generative search. None of these checks whether claims are supported by the cited pages (Zha and Chang name claim-level checking as future work), and the only grounding checks found (Xu et al.; Huang et al.; the Oumi analysis for the New York Times) are English and US-located. The opportunity should be narrowed to claim-level, human-validated fidelity across languages and markets, with source localization as a secondary measure. See v1 as published.
Earlier title: Do AI search answers stay faithful to their cited sources in other languages and markets?
Earlier question: For matched public-interest queries (news, health, elections) issued in several languages and country markets, how often do claims in Google AI Overviews and AI Mode lack support in the pages they cite, how often do they cite sources in the user's own language and country, and do these rates differ from the US English baseline?