Research opportunity · AI in the research apparatus
How much AI-assisted or agent responding reaches probability panels?
Question
In probability-based panels, what share of completed interviews show AI assistance or autonomous-agent completion when measured with embedded multi-method detection validated against consented chatbot-history linkage, and how does that compare with opt-in platforms measured the same way?
Not yet measuredOne PhD chapter or paperLow to medium costNo study found
Why it matters
Probability panels are the reference standard that much of social science relies on, and providers say they are less exposed. That is currently an assumption rather than a measurement. A measured rate, even a low one, would give the dispute about AI contamination of survey research its first evidence from this tier of panels.
The debate lists this among what we still do not know: How common are fully autonomous agents in probability-based or high-assurance survey panels?
Evidence that motivates it
- Autonomous LLM agents can complete online surveys while passing conventional attention and logic checks at very high rates under tested conditions.
Agents can pass conventional quality checks, so standard panel QA cannot be assumed to catch them. - AI-mediated responding is measurable in current online research, but self-reported assistance, overt outsourcing, AI-test failure and autonomous-agent detection are different estimands and cannot be pooled into one prevalence rate.
Existing prevalence estimates measure different pathways (assistance, outsourcing, AI-test failure, agents) and cannot be pooled into one rate; none of them comes from a probability panel.
A possible design
Partner with a probability panel that accepts researcher-proposed modules. Embed a short detection module: paste and keystroke paradata, honeypot items, and AI-capability test items, with a consented sub-sample linking chatbot histories (following Sehgal et al.) to validate the detectors. Field the identical module on two opt-in platforms. Report pathway-specific rates (assistance, outsourcing, agent completion) rather than one contamination rate.
Data needed
A module in a probability panel (several panels accept academic proposals at low or no cost); parallel opt-in samples; consent and ethics approval for history linkage.
Feasibility
One PhD chapter or paper. Low to medium: panel access is often free for academic proposals; opt-in samples cost a few hundred to a few thousand US dollars. Depends on a panel partner; the ethics approval for history linkage is the main hurdle.
Has this been done?
The search found no study that does the core design (first-pass screen of 12 October 2026, checked again on 13 October 2026).
No measurement in a probability panel was found. Sehgal et al. (7 October 2026) linked donated chatbot histories to over 700,000 submissions on a curated opt-in platform and found about 1% AI-assisted, highly concentrated among a few workers, with half of assisted submissions outside what the platform detector covers; that method could validate detectors in a probability panel. Opt-in studies now estimate AI use and agent presence across many platforms (Gordon et al.; Chen, Urminsky, Zhang et al.; Xu and Malkin 2026) and validated agent-detection items exist (Affonso 2026), and NORC has built an AmeriSpeak-trained detector, but none reports a prevalence rate in a probability panel measured with validated, pathway-specific detection.
How strong this evidence is
- Who searched and checked: Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert; an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Every check so far was done by AI; no person or domain expert has checked it yet.
- What was read: none of the 10 nearest works below is recorded as judged from an abstract only. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search on the date shown; Google Scholar, SSRN and paywalled journal pages were only partly reachable. Absence of a hit is not proof that the work does not exist. Literature check of 2026-10-13: Probability-panel provider reports (LISS, GESIS Panel, Understanding America Study, Ipsos KnowledgePanel, Pew ATP methodology notes) were not systematically searched beyond the queries listed; a provider-internal rate could exist in grey literature or conference talks (for example AAPOR 2026 sessions) not indexed by web search. Scholarly APIs were not used. Absence of a hit is not proof that the work does not exist. A NORC talk at AAPOR (Lerner, 15 May 2026) on detecting AI-generated survey responses was not read; which panel it uses and whether it reports a prevalence rate is unknown.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- Sehgal, Tonneau, Folk, Ungar & Guntuku, "AI-Assisted Submissions in Online Research Are Rare and Highly Concentrated but Routinely Approved," arXiv:2610.09279 (7 Oct 2026)
About 1% AI-assisted submissions on a curated opt-in platform via chatbot-history linkage, concentrated among few workers; half of the assisted submissions were bounded responses that the platform detector does not cover. ChatGPT only, no agents, no probability panel. Published after LD-07's evidence cut. - Gordon, Rothschild, Affonso, Sulik, Hauser, Pepin & Jones, "AI Agent Prevalence and Data Quality Across Multiple Online Sample Providers," PsyArXiv (Apr 2026)
Pre-registered study of 5,200 respondents across 10 opt-in platforms: agent detections concentrated on MTurk (11–16%), at or below 1% elsewhere; already cited in LD-07; no probability panel and no validation against known AI use. - Chen, Urminsky, Zhang et al., "Estimating the threat of AI-agent responding across online survey platforms," PsyArXiv (Mar 2026)
AI-test failure rates of 6–41% across seven platforms; the abstract does not indicate any probability panel. - NORC, "Fraudulent respondents and bots in nonprobability surveys: A literature review" (Apr 2026)
States that probability panels are less susceptible but reports no measured AI-responding rate. - Sepulvado, "Detecting AI Responses in Survey Data: NORC's Next Leap for Data Quality," NORC (Dec 2025)
Describes a classifier for AI-written open-ended answers trained on AmeriSpeak panelists versus LLM output; reports detector accuracy, not a prevalence rate in the panel. - Xu & Malkin, "A Penny for Your Prompts: Experiments Detecting and Mitigating LLM Usage by Survey Respondents," arXiv:2607.00403 (2026)
N = 250 across surveys on Prolific and MTurk; estimated LLM use ranges from under 10% (Prolific) to over 80% (MTurk), and no participant used browser agents at the time. Opt-in crowdsourcing only; the abstract does not describe validation against known AI use. - Affonso, "Brief Commentary: A Framework for Detecting AI Agents in Online Research," Journal of Consumer Research 53(3):619-633 (2026), doi:10.1093/jcr/ucag006
Cognitive-trap items flag 97.1% of 526 deployed agents and 4.1% of 1,007 humans on Prolific, versus 2.3% of agents caught by attention checks; replicated on MTurk and CloudResearch Connect. Provides validated agent-detection items usable in the proposed module; no probability panel and no prevalence estimate in one. - Zhang, Xu & Alvero, "Generative AI Meets Open-Ended Survey Responses: Research Participant Use of AI and Homogenization," Sociological Methods & Research 54(3) (2025), doi:10.1177/00491241251327130
34% of participants on an online research platform self-report using LLMs for open-ended answers; LLM answers are more homogeneous and positive. Self-report on an opt-in platform, not a probability panel and not validated detection. - Wang, Mamaev & Leckie, "Towards Detecting AI-Assisted Responses in Online Surveys," arXiv:2609.17317 (2026)
Builds detectors for AI-assisted responses (full generation, revision, persona-grounded agentic completion) using three real surveys paired with controlled LLM responses. Detection method only; no prevalence estimate and sample type not stated in the abstract. - Huang, Sepulvado & Lerner, "Tool Development and Bias Mitigation: Detecting AI-Generated Survey Responses," AAPOR presentation, NORC (14 May 2025)
Detector trained on AmeriSpeak Omnibus open-ended answers versus GPT-3.5, GPT-4, Llama 3.1 and Claude 3.5 Sonnet output; reports about 2% misclassification of humans, doubled for postgraduates. Probability-panel data used for training, but no prevalence of AI use among panelists is reported.
Literature check
Stands as first published. No measurement of AI-assisted or agent responding in a probability-based panel was found. New work is all on opt-in platforms (Xu and Malkin 2026: under 10% on Prolific to over 80% on MTurk; Affonso 2026: cognitive-trap detection of agents on Prolific, MTurk and CloudResearch; Zhang, Xu and Alvero 2025: 34% self-reported use on an online platform) or on detector building (Wang et al. 2026; NORC's AmeriSpeak detector reports misclassification, not prevalence). The design gap therefore stands; verdict stays OPEN because nothing measures the core quantity in the probability-panel tier, though the opt-in comparison arm and detector components are now well developed.
Searched: web (extended); arxiv.org; osf.io; ssrn.com; socarxiv.org; academic.oup.com; ojs.ub.uni-konstanz.de (Survey Research Methods); journals.sagepub.com; tandfonline.com; cambridge.org; norc.org; pewresearch.org. Checked by Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert.
The screen before this check
The search found no study that does the core design (first-pass screen of 12 October 2026).
No measurement in a probability panel was found. Since the debate's evidence cut, Sehgal et al. (7 October 2026) linked donated chatbot histories to over 700,000 submissions on a curated opt-in platform and found about 1% AI-assisted, highly concentrated among a few workers, with half of assisted submissions falling outside what the platform detector covers. That method could validate detectors in a probability panel. Opt-in platform studies (Gordon et al.) and NORC's AmeriSpeak detector work exist, but neither reports a prevalence rate in a probability panel measured with validated detection.
Search queries used
LLM-generated responses probability-based panel LISS OR AmeriSpeak OR Understanding America Study OR KnowledgePanel detection AI use open-ended answersSocArXiv OR OSF prevalence LLM-assisted survey responding high-quality panel paste detection keystroke 2026AI agent bot responses probability-based panel prevalence LISS Understanding America Study GESIS detectionZhang Urminsky AI agents survey responses prevalence online panels Chicago Booth
Literature check, 13 October 2026:
probability panel respondents use ChatGPT to answer open-ended questions prevalence paste detection LISS GESIS PanelAI-generated open-ended survey responses detection prevalence probability-based panel 2026large language model use by survey respondents data quality probability sample web survey Survey Research Methods Journal of Survey Statistics and MethodologyPew American Trends Panel OR KnowledgePanel OR Understanding America Study respondents AI chatbot answers open-ended measured shareNORC AmeriSpeak AI-generated responses rate detected open-ended 2026 AAPORKI-generierte Antworten Umfrage Panel Befragte ChatGPT offene Fragen Erkennung Anteil GESIS LISS"AI Agent Prevalence and Data Quality Across Multiple Online Sample Providers" OR Gordon Rothschild agents survey probability panel follow-up
Provenance
RO-LD07-03 v2. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-07 debate on 2026-10-12. Checked by an independent AI reviewer with fresh context (not a person), 12 October 2026: every nearest-work citation checked against its source page; corrections applied. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.
Changes
- v2, 13 October 2026: Literature check added: a second, deeper search for existing work, with its queries, new nearest work and limits. The question and verdict stand. See v1 as published.