Research opportunity · AI in the research apparatus
Do AI literature-review tools change the conclusions of social-science reviews?
Question
Across a sample of published social-science meta-analyses, do the studies retrieved by AI review tools, compared with those the original review included, change the pooled effect, its heterogeneity or the stated conclusion, and is the missed evidence systematically different (grey literature, non-English studies, null results)?
Not yet measuredOne PhD chapter or paperLow costPartly addressed
Why it matters
Students and researchers now start literature reviews with AI search and deep-research tools. If those tools systematically miss certain kinds of evidence, they can shift what a field believes without anyone noticing, which matters most for the early-career researchers who rely on them.
The debate lists this among what we still do not know: How often does retrieval-augmented synthesis change which social-science evidence enters a review and therefore changes its conclusion?
Evidence that motivates it
- High annotation accuracy does not by itself guarantee valid downstream inference; treating imperfect predicted labels as observed variables can bias estimates and invalidate confidence intervals, while design-based validation with a probability-sampled human-labelled subset can restore valid inference under stated assumptions.
The debate's core rule: validate the scientific conclusion, not just the intermediate output. For review tools the intermediate output is recall; the conclusion is the pooled finding.
A possible design
Sample 20–30 recent social-science meta-analyses with open data (education, psychology, political science, sociology). Re-run each search question through several AI review tools. Recompute the pooled estimate using only the retrieved studies, and code what was missed by publication type, language and result direction.
Data needed
Meta-analyses with open study-level data; access to AI review tools (several have free tiers); a coding protocol for missed studies.
Feasibility
One PhD chapter or paper. Low: free or entry-level tool tiers and existing meta-analytic data. Tool behaviour changes quickly, so record versions and dates.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026, checked again on 13 October 2026).
Medical and health studies measure AI-tool recall (Elicit sensitivity around 38% across four health reviews), a deep-research audit shows search skews evidence direction on a systematic-review benchmark, and one 2026 agentic pipeline re-ran a single psychology/communication meta-analysis and turned a null pooled effect significant. No study found re-runs a sample of published social-science meta-analyses through several AI review tools, tests how often the pooled conclusion changes, and codes whether the missed evidence is systematically grey, non-English or null.
How strong this evidence is
- Who searched and checked: Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert; an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: 1 of the 8 nearest works below was judged from an abstract, summary or search listing only; it is marked. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist. Literature check of 2026-10-13: Wiley (Cochrane ESM), PMC, ScienceDirect and Medscape pages were blocked or failed to load; a 2026 ScienceDirect paper titled 'Evaluating the Utility of Artificial Intelligence in Conducting Systematic Reviews' (S2352344126000841) and a 2026 Medscape report 'AI Falls Short in Systematic Review Searches' could not be opened and should be checked by hand. Scholarly APIs and Google Scholar not used.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Xiao, Wang, Chen et al., "Search Shapes Conclusions: Auditing Evidence Selection Bias in Deep Research Agents," arXiv:2609.39026 (2026)
Adaptive search by deep-research agents skews evidence direction and shifts estimates; biomedical benchmark, not published social-science reviews. - Bernard et al., "Using artificial intelligence for systematic review: the example of elicit," BMC Medical Research Methodology 25:75 (2025)
Elicit retrieved only 3 of the umbrella review's included studies (missing 17) plus 3 new ones, with results varying across repeated runs; recall only. - Hilkenmeier et al., "Evaluating the AI Tool 'Elicit' as a Semi-Automated Second Reviewer for Data Extraction," Social Science Computer Review (2025)
Tests data extraction accuracy, not search or conclusions. - Taherinezhad, Maier, Vitagliano, Pierri & Feuerriegel, "AutoSynthesis: An agentic system for automated meta-analysis," arXiv:2607.15247 (2026)
Opened. An LLM search agent (arXiv, Semantic Scholar, Crossref, OSF, PubMed) re-ran one published meta-analysis on the persuasiveness of LLM-generated messages: it recovered 5 of 7 originally included studies (6 of 7 after manual correction) and added others, and its pooled Hedges' g was 0.143 (significant) versus the original 0.020 (not significant). This is a single-case demonstration that AI retrieval can change a social-science pooled conclusion; it does not sample many meta-analyses, compare commercial review tools, or code missed studies by publication type, language or result direction. - Lau et al., "Comparison of Elicit AI and Traditional Literature Searching in Evidence Syntheses Using Four Case Studies," Cochrane Evidence Synthesis and Methods (2025), doi:10.1002/cesm.70050; preprint medRxiv 10.1101/2025.06.17.25329772
Opened the medRxiv preprint (the journal page and PMC copy were not reachable; first author taken from the Wiley search listing). Elicit sensitivity against four health reviews (vaping harms, physical-activity economics, fertility, breast reconstruction) averaged 37.9%. It counts unique extra studies but does not test whether conclusions or pooled estimates change; health topics only. - Xie, Su, Zhou, Liu, Zhang & Ai, "MetaSyn: A Benchmark for LLM Agents on Meta-Analysis Articles from Nature Portfolio," arXiv:2606.17041 (2026) Judged from an abstract, summary or search listing only
Opened abstract page. 422 meta-analyses from Nature Portfolio with included studies and a PubMed-anchored corpus with distractors; evaluates agents stage by stage. The abstract does not describe live search with real tools, recomputing pooled effects, or a social-science subset, so it is a benchmark resource rather than a test of conclusion change. - Zabaleta & Lin, "SciLitBench: Benchmark and Design Principles for LLM-Powered Systematic Literature Reviews," arXiv:2609.05505 (2026)
Opened. Evaluates 22 open-weight LLMs on screening and extraction for one review about literature-review automation; no search re-runs of published meta-analyses and no pooled estimates. - Lagisz, Mizuno, Morrison, Pollo, Ricolfi, Yang & Nakagawa, "Using Elicit AI research assistant for data extraction in systematic reviews: a feasibility study across environmental and life sciences," Research Synthesis Methods (2025)
Opened the EcoEvoRxiv preprint. Data extraction only, environmental and life sciences; explicitly did not extract effect sizes or test search.
Literature check
Stands as first published. The closest new work is Taherinezhad et al. (AutoSynthesis, arXiv:2607.15247), which re-runs one published psychology/communication meta-analysis with an agentic pipeline and finds the pooled effect turns significant, and Lau et al. (Cochrane ESM 2025), which measures Elicit recall against four health reviews without testing conclusions. Neither samples a set of social-science meta-analyses, compares several review tools, or codes the missed studies by type, language or result direction, so the core design is still open.
Searched: web (extended); web (standard); arxiv.org; osf.io; ssrn.com; medrxiv.org; biorxiv.org; ecoevorxiv.org; cambridge.org; onlinelibrary.wiley.com; sciencedirect.com; tandfonline.com; journals.sagepub.com; link.springer.com. Checked by Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert.
The screen before this check
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026).
Medical and health studies measure AI-tool recall, and a new paper shows deep-research agents' search skews evidence direction on a biomedical benchmark. No study found re-runs published social-science meta-analyses and tests whether the pooled conclusion changes.
Search queries used
Elicit Consensus AI tools reproduce social science systematic review recall included studiesdeep research agent replicate meta-analysis conclusion change psychology economics arXiv 2026Research Synthesis Methods AI search tools Undermind Elicit recall benchmark published reviews education sociologyOSF preprint AI-assisted evidence synthesis reproduce Campbell Collaboration systematic review LLM search screening
Literature check, 13 October 2026:
AI research assistant retrieval replicate published meta-analysis pooled effect size changes social scienceLLM literature search recall systematic review education psychology Elicit Undermind SciSpace evaluation 2025 2026 [allowed: arxiv.org, osf.io, ssrn.com, papers.ssrn.com]Research Synthesis Methods generative AI search sensitivity compared with Boolean search included studies meta-analysis reproduction [allowed: cambridge.org, onlinelibrary.wiley.com, sciencedirect.com, tandfonline.com, journals.sagepub.com]"Comparison of Elicit AI and Traditional Literature Searching in Evidence Syntheses Using Four Case Studies""On the search sensitivity of a meta-analysis" AI-generated MCQs Medical Teacher 2026"Search Shapes Conclusions" deep research agents evidence selection biasAI-assisted search missed studies grey literature non-English null results bias evidence synthesis Campbell systematic review LLM tooldeep research agent reproduce meta-analysis conclusion benchmark "meta-analysis" agent recompute effect size 2026 [allowed: arxiv.org, osf.io, medrxiv.org, biorxiv.org]"Search Shapes Conclusions" OR "evidence selection bias" deep research agents meta-analysis social science follow-up 2026KI-Recherchetools systematische Übersichtsarbeit Sozialwissenschaften Vollständigkeit Elicit Consensus Vergleich"Using artificial intelligence for systematic review: the example of elicit" cited replication social science reviews AI search recallChatGPT deep research OR Undermind recall of studies included in published education meta-analyses evaluation [allowed: tandfonline.com, journals.sagepub.com, sciencedirect.com, link.springer.com, edarxiv.org, osf.io, cambridge.org]
Provenance
RO-LD07-06 v2. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-07 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.
Changes
- v2, 13 October 2026: Literature check added: a second, deeper search for existing work, with its queries, new nearest work and limits. The question and verdict stand. See v1 as published.