Research opportunity · Persuasion, Intermediation & the Information Order
An updated meta-analysis of AI persuasion that separates attitudes from behaviour
Question
Pooling randomized experiments published from 2023 to 2026, including those testing 2025–26 frontier models, how large is the persuasive effect of AI messages and conversations, relative to no message and to human persuaders, and how does it vary with model generation, conversation format, message length, topic, country and outcome type (attitude, vote intention, behaviour, persistence at follow-up)?
Not yet measuredMaster's thesisLow costPartly addressed
Why it matters
The only meta-analysis pooled seven studies up to May 2025 and found no average difference between AI and human persuaders, with large unexplained variation. Since then, several large preregistered experiments report frontier models beating expert humans, outperforming campaign ads and changing real behaviour. Readers currently have to weigh a dated null pooled estimate against individual newer studies. An updated synthesis that keeps attitude, behaviour and persistence outcomes apart would tell researchers and policymakers what the body of evidence now supports, and where it is still thin.
The debate lists this among what we still do not know: a newer meta-analysis of frontier-model persuasion
Evidence that motivates it
- A 2025 meta-analysis of seven earlier studies found no average persuasiveness difference between LLMs and humans, with substantial heterogeneity.
A 2025 meta-analysis of seven studies found no average AI–human difference with substantial heterogeneity; an updated meta-analysis including 2025–26 frontier systems is named as what would change it. - In four preregistered 2025–26 UK experiments, frontier AI out-persuaded strong human text persuaders; its advantage over coached elite debaters disappeared when AI was constrained to the debaters’ message length and writing speed.
Newer UK experiments found frontier AI out-persuaded strong human text persuaders, with the advantage disappearing under human-like length and speed limits; message length and speed are therefore moderators a new synthesis should code. - In two preregistered UK experiments, brief AI conversations increased real petition signing by 12.8 and 19.7 percentage points, while attitude effects did not predict behavioural effects.
Brief AI conversations increased real petition signing while attitude effects did not predict behavioural effects, which is why attitude and behaviour outcomes must be pooled separately.
A possible design
Preregister a systematic review and meta-analysis (PROSPERO-style protocol on OSF). Search Web of Science, Scopus, ACM, arXiv, OSF, PsyArXiv, SocArXiv and SSRN for randomized experiments from 2023 to the search date in which AI-generated messages or conversations aim to change attitudes, intentions or behaviour. Code two contrasts separately (AI versus no-message control; AI versus human persuader), and outcome type (attitude, vote intention, behaviour, follow-up persistence). Code moderators: model and release date, interactive versus one-shot, message length, personalization, information-dense prompting, topic, country and language, sample source and preregistration. Fit multilevel random-effects models with robust variance estimation, test for small-study effects, and run sensitivity analyses that exclude studies under correction or expression of concern. Release the coded dataset as a living database that others can update.
Data needed
Published papers and preprints with effect sizes or raw data; replication files where available; two independent coders for screening and coding.
Feasibility
Master's thesis. Low: mainly researcher time; no data collection. A second coder is needed for reliability. Many recent studies are preprints and some have been corrected; record version and status for each included estimate. Outcomes are measured on different scales, so standardization choices must be preregistered.
Worked example
Research proposal outline: An updated meta-analysis of AI persuasion that separates attitudes from behaviour: a draft research proposal outline built from this opportunity, showing how it could become a project.
Drafted by AI (v0.2, 14 October 2026). Not peer reviewed and not reviewed by a person or domain expert.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 12 October 2026, checked again on 13 October 2026).
An earlier meta-analysis (Huang & Wang, 2023) pooled 121 experiments comparing AI and human communicators and separated perceptions, attitudes, intentions and behaviour, but mostly predates LLM-era studies. The only LLM-specific meta-analysis found (Hölbling et al.) closed its search on 22 May 2025, included seven studies and pooled only the AI-versus-human contrast. Large later experiments (Hackenburg et al., June 2026; Chen, Kalla & Le, March 2026) report primary results but no synthesis, and a study of nine automated persuasion benchmarks (Dementaviciute et al., October 2026) compares methods, not pooled human-subject effects. No meta-analysis covering 2025-26 frontier-model experiments, separating AI-versus-control from AI-versus-human contrasts, or separating attitude, behaviour and persistence outcomes for LLM persuasion was found.
How strong this evidence is
- Who searched and checked: Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert; an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: 2 of the 7 nearest works below were judged from an abstract, summary or search listing only; each is marked. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown. PROSPERO, OSF registrations, Google Scholar and SSRN were not searched directly, so a registered but unpublished review could exist. A September 2025 ResearchGate entry titled "The Persuasive Power of LLMs: A Systematic Literature Review and Meta-Analysis" could not be opened; it may be an earlier version of Hölbling et al. or a separate review. Absence of a hit is not proof that the work does not exist. Literature check of 2026-10-13: PROSPERO was not searched directly and an OSF/PsyArXiv record (osf.io/preprints/psyarxiv/ehg7n) surfaced by the protocol query could not be read, so a registered but unpublished review may exist. The Current Opinion in Psychology review 'Using conversational AI to reduce science skepticism' (ScienceDirect S2352250X25002295) could not be opened; it appears to be a short review rather than a meta-analysis. Google Scholar was not searched. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Hölbling, Maier & Feuerriegel, "A meta-analysis of the persuasive power of large language models," Scientific Reports 15, 43818 (12 Dec 2025)
Seven studies, 17,422 participants, 12 effect sizes, search closed 22 May 2025; pooled Hedges' g = 0.02 for AI versus humans with I² of about 76%. Models up to GPT-4o and Claude 3.5 and 3.7 Sonnet. As described, it pools the AI-versus-human contrast only and does not report behavioural or persistence outcomes separately. - Hackenburg, Wagner, Hewitt, Tappin, Saunders, Kirk, Margetts & Summerfield, "AI systems out-persuade expert humans," arXiv:2606.16475 (15 Jun 2026)
Four preregistered experiments (18,978 conversations, 6,923 people) comparing AI with laypeople, tournament winners, canvassers and champion debaters. A primary study a new meta-analysis would include, not a synthesis. - Chen, Kalla & Le, "Benchmarking Political Persuasion Risks Across Frontier Large Language Models," arXiv:2603.09884 (10 Mar 2026)
Two survey experiments (19,145 Prolific participants, US policy issues) testing multi-turn conversations with seven frontier models against standard campaign ads; reports the models outperform the ads, with persuasiveness varying by model. Primary study, not a synthesis. - Dementaviciute, Vaitonyte & De Bie, "LLM Persuasion Is in the Eye of the Evaluation," arXiv:2610.10232 (7 Oct 2026)
Runs nine published automated persuasion-evaluation methods on fifteen models and finds weak agreement between them. Compares automated benchmarks, not human-subject experiments, so it is not a meta-analysis of persuasion effects. - Huang & Wang, "Is artificial intelligence more persuasive than humans? A meta-analysis," Journal of Communication 73(6), 552-562 (2023) Judged from an abstract, summary or search listing only
Meta-analysis of 121 randomized experiments (N = 53,977) comparing AI and human agents: overall equally persuasive; AI less effective on behavioural intentions, no significant difference on perceptions, attitudes or actual behaviours; moderators include AI role, communication direction, setting and demographics. Separates outcome types, but pools AI-versus-human contrasts only and mostly predates LLM-era experiments (search dates not stated on the page). Read from the HKBU repository record. - Hölbling, Maier & Feuerriegel, "A Meta-Analysis of the Persuasive Power of Large Language Models," arXiv:2512.01431 (Dec 2025)
Preprint version of the Scientific Reports article already listed. No later update or extension citing it was found in this round. - Hackenburg, Hewitt, Wagner, Tappin & Summerfield, "Artificial intelligence can persuade people to take political actions," arXiv:2604.09200 (2026) Judged from an abstract, summary or search listing only
Authors, title and date (10 April 2026) checked on the arXiv page; the paper itself was not read. From its title, a primary study with a behavioural (political action) outcome that a new meta-analysis would include; not a synthesis.
Literature check
Question stands; related work found. No meta-analysis pooling 2023-2026 LLM persuasion experiments, with separate AI-versus-control and AI-versus-human contrasts and separate attitude, behaviour and persistence outcomes, was found. Two syntheses bound the space: Hölbling, Maier & Feuerriegel (Scientific Reports 2025; arXiv:2512.01431) pool seven LLM-versus-human studies with a search closed in May 2025, and Huang & Wang (Journal of Communication 2023) pool 121 earlier experiments comparing AI and human agents and do separate perceptions, attitudes, intentions and behaviours, but predate most LLM-era studies and only cover the AI-versus-human contrast. The verdict moves from OPEN to PARTLY_ADDRESSED because outcome-type separation has been done for pre-LLM AI communicators.
Searched: web (extended and standard); arxiv.org; osf.io; psyarxiv; ssrn.com; nature.com; pnas.org; academic.oup.com; journals.sagepub.com; sciencedirect.com; science.org; tandfonline.com; scholars.hkbu.edu.hk; German-language web. Checked by Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert.
The screen before this check (verdict changed)
The search found no study that does the core design (first-pass screen of 12 October 2026).
The only meta-analysis found (Hölbling et al.) closed its search on 22 May 2025, included seven studies and pooled only the AI-versus-human contrast. Large later experiments (Hackenburg et al., June 2026; Chen, Kalla & Le, March 2026) report primary results but no synthesis. A study of nine automated persuasion benchmarks (Dementaviciute et al., October 2026) shows that evaluation methods disagree, but it compares methods, not pooled human-subject effects. No meta-analysis covering 2025–26 frontier-model experiments or separating attitude, behaviour and persistence outcomes was found.
Search queries used
meta-analysis LLM persuasion versus humans 2025 seven studies heterogeneitymeta-analysis persuasive effects of large language models 2026 attitude change AI conversations systematic reviewliving systematic review AI persuasion experiments frontier models effect sizes moderators preregistered"meta-analysis" "AI persuasion" OR "LLM persuasion" 2026 arXiv OR PsyArXiv OR OSF updated frontier GPT-4o Claude studies
Literature check, 13 October 2026:
meta-analysis persuasion large language models AI-generated messages attitude change randomized experiments 2026 [allowed_domains: osf.io, psyarxiv.com, arxiv.org, ssrn.com, papers.ssrn.com, osf.io/preprints]systematic review meta-analysis AI chatbot persuasion conversations belief change conspiracy vaccine behavior follow-up effect size [allowed_domains: nature.com, pnas.org, academic.oup.com, journals.sagepub.com, sciencedirect.com, apa.org, tandfonline.com, science.org]"A meta-analysis of the persuasive power of large language models" Hölbling cited OR update OR extendsPROSPERO OR OSF registered protocol systematic review meta-analysis generative AI persuasion human versus AI messages effect sizesMetaanalyse Überzeugungswirkung KI-generierter Botschaften Sprachmodelle Einstellungsänderung (standard)pooled analysis AI persuasion experiments vote intention behavior durability decay frontier models GPT-5 2026 synthesis of studies
Provenance
RO-LD10-06 v2. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-10 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.
Changes
- v2, 13 October 2026: Literature check added: a second, deeper search for existing work, with its queries, new nearest work and limits. The verdict changed from "No study found doing the core design" to "Related work exists" because related work turned up; the question stands. See v1 as published.