Research opportunity · AI in the research apparatus
Do AI literature-review tools change the conclusions of social-science reviews?
Earlier version (v1)
This is RO-LD07-06 as it was published at v1, kept unchanged. The current version is v2; its changes are listed there.
Question
Across a sample of published social-science meta-analyses, do the studies retrieved by AI review tools, compared with those the original review included, change the pooled effect, its heterogeneity or the stated conclusion, and is the missed evidence systematically different (grey literature, non-English studies, null results)?
Not yet measuredOne PhD chapter or paperLow costPartly addressed
Why it matters
Students and researchers now start literature reviews with AI search and deep-research tools. If those tools systematically miss certain kinds of evidence, they can shift what a field believes without anyone noticing, which matters most for the early-career researchers who rely on them.
The debate lists this among what we still do not know: How often does retrieval-augmented synthesis change which social-science evidence enters a review and therefore changes its conclusion?
Evidence that motivates it
- High annotation accuracy does not by itself guarantee valid downstream inference; treating imperfect predicted labels as observed variables can bias estimates and invalidate confidence intervals, while design-based validation with a probability-sampled human-labelled subset can restore valid inference under stated assumptions.
The debate's core rule: validate the scientific conclusion, not just the intermediate output. For review tools the intermediate output is recall; the conclusion is the pooled finding.
A possible design
Sample 20–30 recent social-science meta-analyses with open data (education, psychology, political science, sociology). Re-run each search question through several AI review tools. Recompute the pooled estimate using only the retrieved studies, and code what was missed by publication type, language and result direction.
Data needed
Meta-analyses with open study-level data; access to AI review tools (several have free tiers); a coding protocol for missed studies.
Feasibility
One PhD chapter or paper. Low: free or entry-level tool tiers and existing meta-analytic data. Tool behaviour changes quickly, so record versions and dates.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026; not yet checked in depth).
Medical and health studies measure AI-tool recall, and a new paper shows deep-research agents' search skews evidence direction on a biomedical benchmark. No study found re-runs published social-science meta-analyses and tests whether the pooled conclusion changes.
How strong this evidence is
- Who searched and checked: an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: none of the 3 nearest works below is recorded as judged from an abstract only. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Xiao, Wang, Chen et al., "Search Shapes Conclusions: Auditing Evidence Selection Bias in Deep Research Agents," arXiv:2609.39026 (2026)
Adaptive search by deep-research agents skews evidence direction and shifts estimates; biomedical benchmark, not published social-science reviews. - Bernard et al., "Using artificial intelligence for systematic review: the example of elicit," BMC Medical Research Methodology 25:75 (2025)
Elicit retrieved only 3 of the umbrella review's included studies (missing 17) plus 3 new ones, with results varying across repeated runs; recall only. - Hilkenmeier et al., "Evaluating the AI Tool 'Elicit' as a Semi-Automated Second Reviewer for Data Extraction," Social Science Computer Review (2025)
Tests data extraction accuracy, not search or conclusions.
Search queries used
Elicit Consensus AI tools reproduce social science systematic review recall included studiesdeep research agent replicate meta-analysis conclusion change psychology economics arXiv 2026Research Synthesis Methods AI search tools Undermind Elicit recall benchmark published reviews education sociologyOSF preprint AI-assisted evidence synthesis reproduce Campbell Collaboration systematic review LLM search screening
Provenance
RO-LD07-06 v1. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-07 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.