Research proposal outline: An updated meta-analysis of AI persuasion that separates attitudes from behaviour

Built from: SocioLens research opportunity RO-LD10-06 (v2, 13 Oct 2026), debate LD-10 “Persuasion, Intermediation & the Information Order” (v0.1, evidence cut 25 Sep 2026), claims LD10-HQ-C01 to C11 and C25. Version: v0.2, 14 October 2026. Supersedes v0.1 (RO-LD10-06-proposal-outline.md, kept unchanged). What changed and why: see “Changes from v0.1” at the end. Drafted: 14 October 2026 by Claude (SocioLens Project Lead, AI). Revised the same day after an AI stand-in review and a prior-work check (RO-LD10-06-review-and-prior-work-check.md). Not reviewed by a person or a domain expert. Status: Outline for a researcher to adopt, adapt or reject. It is not a registered protocol.

How to read this document

Every statement below carries one of four labels, so the reader can see what kind of statement it is.

Statements about existing work come from the RO-LD10-06 novelty screen (12 Oct 2026), its literature check (13 Oct 2026) and a prior-work check on 14 Oct 2026. All three were done by AI agents through web search and arXiv, with stated limits (Section 9).

1. Research question

[From RO-LD10-06] Pooling randomized experiments published from 2023 to the search date, including those testing 2025–26 frontier models, how large is the persuasive effect of AI messages and conversations, relative to no message and to human persuaders, and how does it vary with model generation, conversation format, message length, topic, country and outcome type (attitude, vote intention, behaviour, persistence at follow-up)?

Sub-questions:

• RQ1. What is the effect of AI persuasion against a no-message or neutral control, separately for each outcome type? Pooled within domain (political and policy attitudes as the primary domain), and reported as a distribution across domains rather than one cross-domain number.
• RQ2. What is the pooled AI-versus-human difference, and does it change with model release date?
• RQ3. How much of any AI advantage is explained by throughput (message length, amount of factual material, speed)?
• RQ4. In studies that measure both, how strongly does individual attitude change predict behaviour? Within-study relationships are pooled where reported. A study-level correlation of attitude and behaviour effects is reported only as descriptive, because it is ecological.
• RQ5. What share of the immediate effect remains at follow-up?

2. Why this matters now

1. [Evidence, C07] The only LLM-specific meta-analysis (Hölbling, Maier & Feuerriegel, Scientific Reports 2025) pooled 7 studies, 17,422 participants and 12 effects, with a search closed on 22 May 2025. It found no average LLM–human difference (g = 0.02, p = .530) with high heterogeneity (I² ≈ 76%). The claim record states that it does not license “a timeless claim that frontier AI is no more persuasive than humans”.
2. [Evidence, C08, preprint] Four later preregistered UK experiments (Hackenburg et al., arXiv:2606.16475; 18,978 conversations, 6,923 people) found frontier AI ahead of random laypeople (+8.2 pp), selected laypeople (+5.6 pp) and elite debaters (+4.6 pp). Against coached elite debaters, the advantage was 0.0 pp [−1.7, +1.6] when AI was held to the debaters’ message length and writing speed.
3. [Evidence, C09, preprint] Two preregistered UK experiments (Hackenburg et al., arXiv:2604.09200; 17,950 responses) found brief AI conversations raised real petition signing by 12.8 and 19.7 pp, while attitude–behaviour correlations were r = .05 and r = .18, both non-significant.
4. [SocioLens synthesis, LD-10] “Model vintage is part of the estimand.” Older and newer systems should not be averaged as if “LLM persuasion” were a single, stable treatment.
5. [SocioLens synthesis, C01] Persuasion capability is not realized social influence. Population effects also need evidence on exposure, persistence and behavioural transmission. A meta-analysis can sharpen the per-contact and persistence parts of that chain; it cannot settle population influence.

Readers now weigh a dated, null pooled estimate against individual newer studies with different designs. The proposed synthesis would say what the body of evidence supports as of its search date, and where it is still thin.

3. Contribution relative to existing work

[From RO-LD10-06 novelty screen; verdict PARTLY_ADDRESSED]

Novelty is a claim, not a label. Neither the screen nor the 14 Oct check found a meta-analysis doing all three things (frontier models, two separate contrasts, separate outcome types). The two records flagged on 13 Oct were both existing papers (Huang & Wang’s preprint; an earlier version of Hölbling et al., inferred). PROSPERO and OSF Registries have not been searched directly, so a registered but unpublished review cannot be ruled out (Section 9). Step 0 of the work plan finishes this check.

4. Hypotheses

All four are [Hypothesis]: expectations drawn from single studies or from the contrast between studies, to be tested, not assumed.

• H1 (vintage). Persuasive effect increases with model release date. Motivated by the contrast between C07 (older models, g ≈ 0) and C08 (frontier models, positive gaps). Two data points from different designs; between studies the pattern may reflect design rather than vintage. Primary test: within studies that compare several models under one design (e.g. Hackenburg et al., Science 2025; Chen, Kalla & Le 2026), pooling the within-study slope on release date or capability. Between-study meta-regression is secondary.
• H2 (throughput). Studies that hold AI to human-like length or speed show smaller AI advantages. Motivated by C08 (advantage vanished under constraint against coached debaters) and C02 (information-dense prompting raised persuasiveness, with lower factual accuracy).
• H3 (attitude–behaviour decoupling). Within studies, individual attitude change weakly predicts behaviour. Motivated by C09 (two experiments). Few studies may report this, so H3 may end descriptive.
• H4 (partial persistence). A minority to roughly half of immediate attitude effects persist at follow-up. Motivated by C03 (≈36–42% retained at one month, one study, GPT-4o, UK). One estimate is not a prior for the field; H4 is stated loosely on purpose.

A null on any of these is a reportable result.

5. Design

5.1 Protocol and registration

[Design choice] Preregister on OSF (PROSPERO accepts health-related outcomes only, so it may not fit; check). Follow PRISMA 2020. Register before full-text screening.

5.2 Eligibility

• Include: randomized experiments, 2023 to search date, in which an AI system generates messages or conducts conversations intended to change attitudes, intentions or behaviour; human participants; an AI arm compared with a no-message/neutral control and/or a human-persuader arm.
• Exclude: automated or simulated-participant evaluations (e.g. LLM-as-judge persuasion benchmarks); studies of AI-labelled human text without an AI-generated arm (relevant to C25, but a different question); observational studies.
• [Design choice] Edge-case rules, fixed before screening: human-edited AI text and human–AI co-written messages are eligible but coded as a separate “hybrid” arm, not pooled with pure AI arms; when an AI arm’s AI source is disclosed in some conditions and not others, disclosure is coded as a moderator; multi-arm studies sharing one control are handled by the multilevel model (or by splitting the shared control), never by counting the control twice.
• [Design choice] Peer-reviewed papers and preprints are both eligible, because much of the frontier evidence is preprint-only (C08, C09). Publication status is a coded moderator and a sensitivity tier.

5.3 Search

[From RO-LD10-06] Web of Science, Scopus, ACM Digital Library, arXiv, OSF, PsyArXiv, SocArXiv, SSRN. [Design choice] Add forward citation tracking from Hölbling et al. 2025 and Huang & Wang 2023, the reference list of Noels et al. 2026, and direct requests to the main labs running this work.

Seed set (not a search; from the SocioLens record):

Whether the two Chen papers report overlapping samples is unknown from the SocioLens record and must be checked before both are coded.

5.4 Two contrasts, four outcome families

[Design choice] Never pool across these cells:

Cells with fewer than about five independent studies are reported descriptively, with no pooled estimate. The AI-versus-control row is pooled within domain only (political and policy attitudes primary); other domains are reported as a distribution of study effects. Expected study counts per cell come from the Step 0 scoping count and are stated in the protocol.

5.5 Moderators to code

• Model and release date (continuous; the main test of H1). [SocioLens synthesis] Vintage is treated as part of the estimand, so pooled estimates are reported by vintage band as well as overall.
• Throughput: words per message, number of factual claims where reported, response speed, whether AI was length- or speed-constrained (H2).
• Format: one-shot message vs multi-turn conversation.
• Personalization, coded with the C06 caveat: the corrected Salvi et al. direct comparison is not significant (P = .07).
• Information-dense prompting / post-training for persuasion (C02).
• Human comparator type: random layperson, selected layperson, expert (debater, canvasser), campaign ad (C08, C10).
• Topic (policy issue, candidate, conspiracy, health, science), country and language, sample source (opt-in panel vs other), preregistration, publication status.

5.6 Effect sizes

[Design choice] Hedges’ g for continuous outcomes; log odds ratio for binary behaviour, converted to g only for cross-family figures, never for pooling. Preregister how 0–100 preference scales (C04, C05) and percentage-point differences (C08, C09) are standardized. Persistence is coded as the ratio of follow-up to immediate effect, with attrition recorded.

5.7 Models

[From RO-LD10-06] Multilevel random-effects models with robust variance estimation (effects nested in studies nested in research groups, since several studies share teams and panels). Meta-regression for moderators. Small-study tests (funnel asymmetry, PET-PEESE, a selection model), interpreted cautiously because many studies are large and preregistered. [Design choice] Because one research group and one panel provider supply a large share of participants, report each team’s share of the total weight and run a leave-one-team-out analysis.

5.8 Sensitivity tiers

1. All eligible studies.
2. Excluding studies under expression of concern (C11) or with corrections that affect the coded estimate (C06 for the personalization contrast).
3. Peer-reviewed only.
4. Preregistered primary outcomes only.
5. Excluding studies rated high risk of bias (5.10).

Each included estimate records the version read (preprint version, published version, correction date).

5.9 Coding reliability

Two independent coders for screening and extraction; agreement reported; disagreements resolved by discussion with the record kept.

5.10 Risk of bias

[Design choice] Each study is rated with an adapted RoB 2 covering: randomization; attention checks and exclusions; attrition at follow-up (persistence outcomes); demand effects on opt-in panels; and whether the coded outcome was the preregistered primary outcome. Ratings feed a sensitivity tier.

6. What the results could and could not support

• Could: a current per-contact pooled estimate by contrast, outcome and vintage; whether the AI advantage tracks vintage or throughput; whether attitude effects are a usable proxy for behaviour; how much effect persists.
• Could not [SocioLens synthesis, C01]: population-scale influence, election-outcome effects or face-to-face superiority. Nearly all seed studies use paid, opt-in online panels in the UK and US (C02–C09 scopes), so external validity outside that setting stays unknown and must be reported as a limit, not corrected for.

7. Outputs

1. Registered protocol (OSF).
2. Article for a journal such as Journal of Communication, Nature Human Behaviour or Psychological Bulletin (fit to be judged by the author).
3. [From RO-LD10-06] The coded dataset released as a living database others can update, with version and correction status per estimate.
4. For SocioLens: if published, the result would be the update named in C07’s “what would change this claim”, and would enter LD-10 as a new source with its own claim.

8. Feasibility and resources

[From RO-LD10-06] Scale: master’s thesis or a focused paper. Cost: low; researcher time only, no data collection. Needs a second coder.

Work plan (phases, not durations):

0. Go/no-go and scoping. (a) Search PROSPERO and OSF Registries for a registered or in-progress review (“persuasion” with “language model”, “chatbot” or “generative AI”); stop or pivot if one covers the same cells. Published reviews and the two records flagged on 13 Oct were checked on 14 Oct (none overlaps). (b) Count likely eligible studies per cell from the seed set and Noels et al. 2026; if the behaviour and persistence cells are clearly too thin, narrow the protocol to attitude outcomes plus a descriptive behaviour section.
1. Draft and register protocol.
2. Search, deduplicate, screen.
3. Extract and double-code; contact authors for missing statistics.
4. Analyse; run sensitivity tiers.
5. Write up; release the dataset.

9. Risks and open unknowns

• Possible existing review [Unknown, reduced]. The two records the 13 Oct check could not read turned out to be existing papers: osf.io/preprints/psyarxiv/ehg7n is the Huang & Wang 2023 preprint (read 14 Oct from a copy the Owner supplied; match to the ID inferred), and the ResearchGate entry is probably an earlier version of Hölbling et al. (inferred from search snippets). PROSPERO, OSF Registries and Google Scholar have still not been searched directly.
• Scoop risk. Hölbling et al. end by calling for the question to be revisited as models advance; an update from them is plausible. Preprinting the protocol early and releasing the living dataset are the best protection.
• Poolability. A 2026 health-domain systematic review judged pooling infeasible. Within-domain pooling (Section 5.4) is the response; if heterogeneity is still extreme, the result is a structured map rather than a pooled estimate.
• Moving evidence base. Many key studies are preprints and two have formal corrections or concerns. Results may shift as they are revised.
• Thin cells. Behaviour and persistence outcomes may have too few studies to pool (H3, H4 may end descriptive).
• Team and panel dependence. A large share of the frontier evidence comes from one research group and one panel provider. The multilevel structure handles statistical dependence; it does not create independent replication.
• Vintage confounding. Newer studies differ from older ones in design as well as model, so H1 can show association, not that model capability causes the difference.
• Evidence in this outline was not re-read. Figures come from SocioLens claim records, whose verification state is “content verified” with verifier recorded as unknown. A researcher adopting this outline should read each source.

10. Provenance

Changes from v0.1

• Nearest work: added Noels et al. 2026 v2 and the 2026 health-materials systematic review; stated that the Huang & Wang estimand differs (they include source-label designs; this proposal does not); recorded that both records flagged on 13 Oct are existing papers.
• RQ1: AI-versus-control pooled within domain, not across domains, because a cross-domain average would not be meaningful.
• H1 and H3 / RQ4: primary tests moved to within-study comparisons; between-study analyses are secondary or descriptive, because between-study versions are confounded or ecological.
• Design: added eligibility edge-case rules (5.2), risk-of-bias assessment (5.10), team-weight reporting and leave-one-team-out analysis (5.7).
• Work plan: Step 0 now includes a scoping count of studies per cell; the remaining registry search is named exactly.
• Risks: updated the existing-review risk; added scoop risk and poolability.
• Unchanged: the question’s core, the seed set, effect-size rules, sensitivity tiers 1–4 and outputs.