Research opportunity · Persuasion, Intermediation & the Information Order
An updated meta-analysis of AI persuasion that separates attitudes from behaviour
Earlier version (v1)
This is RO-LD10-06 as it was published at v1, kept unchanged. The current version is v2; its changes are listed there.
Question
Pooling randomized experiments published from 2023 to 2026, including those testing 2025–26 frontier models, how large is the persuasive effect of AI messages and conversations, relative to no message and to human persuaders, and how does it vary with model generation, conversation format, message length, topic, country and outcome type (attitude, vote intention, behaviour, persistence at follow-up)?
Not yet measuredMaster's thesisLow costNo study found
Why it matters
The only meta-analysis pooled seven studies up to May 2025 and found no average difference between AI and human persuaders, with large unexplained variation. Since then, several large preregistered experiments report frontier models beating expert humans, outperforming campaign ads and changing real behaviour. Readers currently have to weigh a dated null pooled estimate against individual newer studies. An updated synthesis that keeps attitude, behaviour and persistence outcomes apart would tell researchers and policymakers what the body of evidence now supports, and where it is still thin.
The debate lists this among what we still do not know: a newer meta-analysis of frontier-model persuasion
Evidence that motivates it
- A 2025 meta-analysis of seven earlier studies found no average persuasiveness difference between LLMs and humans, with substantial heterogeneity.
A 2025 meta-analysis of seven studies found no average AI–human difference with substantial heterogeneity; an updated meta-analysis including 2025–26 frontier systems is named as what would change it. - In four preregistered 2025–26 UK experiments, frontier AI out-persuaded strong human text persuaders; its advantage over coached elite debaters disappeared when AI was constrained to the debaters’ message length and writing speed.
Newer UK experiments found frontier AI out-persuaded strong human text persuaders, with the advantage disappearing under human-like length and speed limits; message length and speed are therefore moderators a new synthesis should code. - In two preregistered UK experiments, brief AI conversations increased real petition signing by 12.8 and 19.7 percentage points, while attitude effects did not predict behavioural effects.
Brief AI conversations increased real petition signing while attitude effects did not predict behavioural effects, which is why attitude and behaviour outcomes must be pooled separately.
A possible design
Preregister a systematic review and meta-analysis (PROSPERO-style protocol on OSF). Search Web of Science, Scopus, ACM, arXiv, OSF, PsyArXiv, SocArXiv and SSRN for randomized experiments from 2023 to the search date in which AI-generated messages or conversations aim to change attitudes, intentions or behaviour. Code two contrasts separately (AI versus no-message control; AI versus human persuader), and outcome type (attitude, vote intention, behaviour, follow-up persistence). Code moderators: model and release date, interactive versus one-shot, message length, personalization, information-dense prompting, topic, country and language, sample source and preregistration. Fit multilevel random-effects models with robust variance estimation, test for small-study effects, and run sensitivity analyses that exclude studies under correction or expression of concern. Release the coded dataset as a living database that others can update.
Data needed
Published papers and preprints with effect sizes or raw data; replication files where available; two independent coders for screening and coding.
Feasibility
Master's thesis. Low: mainly researcher time; no data collection. A second coder is needed for reliability. Many recent studies are preprints and some have been corrected; record version and status for each included estimate. Outcomes are measured on different scales, so standardization choices must be preregistered.
Has this been done?
The search found no study that does the core design (first-pass screen of 12 October 2026; not yet checked in depth).
The only meta-analysis found (Hölbling et al.) closed its search on 22 May 2025, included seven studies and pooled only the AI-versus-human contrast. Large later experiments (Hackenburg et al., June 2026; Chen, Kalla & Le, March 2026) report primary results but no synthesis. A study of nine automated persuasion benchmarks (Dementaviciute et al., October 2026) shows that evaluation methods disagree, but it compares methods, not pooled human-subject effects. No meta-analysis covering 2025–26 frontier-model experiments or separating attitude, behaviour and persistence outcomes was found.
How strong this evidence is
- Who searched and checked: an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: none of the 4 nearest works below is recorded as judged from an abstract only. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown. PROSPERO, OSF registrations, Google Scholar and SSRN were not searched directly, so a registered but unpublished review could exist. A September 2025 ResearchGate entry titled "The Persuasive Power of LLMs: A Systematic Literature Review and Meta-Analysis" could not be opened; it may be an earlier version of Hölbling et al. or a separate review. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Hölbling, Maier & Feuerriegel, "A meta-analysis of the persuasive power of large language models," Scientific Reports 15, 43818 (12 Dec 2025)
Seven studies, 17,422 participants, 12 effect sizes, search closed 22 May 2025; pooled Hedges' g = 0.02 for AI versus humans with I² of about 76%. Models up to GPT-4o and Claude 3.5 and 3.7 Sonnet. As described, it pools the AI-versus-human contrast only and does not report behavioural or persistence outcomes separately. - Hackenburg, Wagner, Hewitt, Tappin, Saunders, Kirk, Margetts & Summerfield, "AI systems out-persuade expert humans," arXiv:2606.16475 (15 Jun 2026)
Four preregistered experiments (18,978 conversations, 6,923 people) comparing AI with laypeople, tournament winners, canvassers and champion debaters. A primary study a new meta-analysis would include, not a synthesis. - Chen, Kalla & Le, "Benchmarking Political Persuasion Risks Across Frontier Large Language Models," arXiv:2603.09884 (10 Mar 2026)
Two survey experiments (19,145 Prolific participants, US policy issues) testing multi-turn conversations with seven frontier models against standard campaign ads; reports the models outperform the ads, with persuasiveness varying by model. Primary study, not a synthesis. - Dementaviciute, Vaitonyte & De Bie, "LLM Persuasion Is in the Eye of the Evaluation," arXiv:2610.10232 (7 Oct 2026)
Runs nine published automated persuasion-evaluation methods on fifteen models and finds weak agreement between them. Compares automated benchmarks, not human-subject experiments, so it is not a meta-analysis of persuasion effects.
Search queries used
meta-analysis LLM persuasion versus humans 2025 seven studies heterogeneitymeta-analysis persuasive effects of large language models 2026 attitude change AI conversations systematic reviewliving systematic review AI persuasion experiments frontier models effect sizes moderators preregistered"meta-analysis" "AI persuasion" OR "LLM persuasion" 2026 arXiv OR PsyArXiv OR OSF updated frontier GPT-4o Claude studies
Provenance
RO-LD10-06 v1. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-10 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.