Benchmarking Political Persuasion Risks Across Frontier Large Language Models

Abstract: Concerns persist regarding the capacity of LLMs to sway political views. Although prior research has claimed that LLMs are not more persuasive than standard political campaign practices, the recent rise of frontier models warrants further study. In two survey experiments (N=19,145) across bipartisan issues and stances, we evaluate seven state-of-the-art LLMs developed by Anthropic, OpenAI, Google, and xAI. We find that LLMs outperform standard campaign advertisements, with heterogeneity in performance across models. Specifically, Claude models exhibit the highest persuasiveness, while Grok exhibits the lowest. The results are robust across issues and stances. Moreover, in contrast to the findings in Hackenburg et al. (2025b) and Lin et al. (2025) that information-based prompts boost persuasiveness, we find that the effectiveness of information-based prompts is model-dependent: they increase the persuasiveness of Claude and Grok while substantially reducing that of GPT. We introduce a data-driven and strategy-agnostic LLM-assisted conversation analysis approach to identify and assess underlying persuasive strategies. Our work benchmarks the persuasive risks of frontier models and provides a framework for cross-model comparative risk assessment.

• The paper demonstrates that frontier LLMs significantly outperform human campaign ads in shifting political opinions, with statistically significant effect sizes.
• The study employs large-scale, randomized survey experiments across multiple models and prompt types to reveal model-specific persuasion dynamics.
• Prompt-model interactions show that information prompts can both heighten and reduce persuasiveness, underscoring the need for tailored regulatory measures.

Benchmarking Political Persuasion Risks Across Frontier LLMs

Introduction

This paper introduces a robust framework for benchmarking political persuasion risks posed by frontier LLMs, with a special focus on comparative evaluation across leading models released by Anthropic, OpenAI, Google, and xAI. The authors address crucial deficiencies in prior work: lack of direct benchmarking against real-world campaign advertisements, outdated model evaluations, and insufficient analysis of underlying AI persuasion strategies. Employing two large-scale, randomized survey experiments (total $N=19,145$) across bipartisan political issues and stances, the study delivers both head-to-head comparison among seven frontier LLMs and systematic quantification of model-induced persuasion relative to human-driven campaign media.

Experimental Design and Methodology

Two sequential survey experiments constitute the core methodology. Study 1 benchmarks four LLMs (Claude Sonnet 4, Gemini 2.5 Flash, GPT-4.1, Grok 4) against human campaign video treatments and placebo controls on U.S. political issues of immigration and minimum wage. Study 2 extends to four subsequent frontier releases (Claude Sonnet 4.5, Gemini 3, GPT-5, Grok 4), further crossing prompt type (plain vs. information) and persuasion direction (support vs. oppose), allowing analysis of both inter-model and intra-model heterogeneity. Outcomes are measured as immediate post-exposure changes in issue support, binarized for interpretability, and effect sizes are estimated with linear regressions and meta-analytic pooling.

Comparative Persuasion Efficacy: LLMs Outperform Human Baselines

A central empirical result is that, across both studies and policy contexts, all tested LLMs meaningfully exceed the persuasive effect of standard campaign ads.

Figure 1: 95% CIs for LLM average treatment effects on policy shift vs. placebo, compared to (generalized) campaign ad effect, for both Immigration and Minimum Wage issues.

This robustly contradicts previous findings limited to earlier LLM generations, in which AI-driven persuasion was not separable from human nor placebo effects using matched settings (2603.09884). Claude models, particularly Sonnet 4 and 4.5, attain the highest effect sizes (e.g., $0.224$ scale points for pro-immigration arguments in Study 1), while Grok consistently yields the lowest, yet still exceeds human and placebo treatments. GPT-4.1, GPT-5, and Gemini series populate the middle tier, but occasionally swap relative rankings depending on issue and stance. The effect sizes are statistically significant with $p<0.001$ across all LLM-human contrasts.

Prompt and Model Interaction: Heterogeneous and Contradictory Effects

Unlike earlier work reporting a uniform persuasive benefit for information-based prompts (e.g., \cite{hackenburg2025levers}; \cite{lin2025persuading}), this study provides strong evidence for highly model-dependent prompt effectiveness.

Figure 2: Pooled persuasion effects stratified by model and prompt, showing full crossing for Study 1.

Information prompts increase persuasiveness for Claude and Grok models but decrease performance for GPT-4.1 and GPT-5. The same pattern persists for Gemini but is less pronounced. This interaction is important for both practical deployment and regulatory assessment: prompt engineering is not a universally amplifying tactic—in some architectures, added information actually attenuates persuasive efficacy.

Figure 3: Model-by-prompt persuasion effect pooling (Study 2), reinforcing interaction heterogeneity among top-tier LLMs.

Issue and Stance Robustness

Persuasion magnitudes generalize across both policy issues and directional stances. There is consistent asymmetry: it is easier for LLMs to move attitudes in favor of the more liberal/Democratic position (policy support) than away from it (policy opposition), but this may be confounded by population-level baseline susceptibilities due to non-random stance assignment.

Figure 4: Average treatment effects by LLM, issue (Immigration/Min. Wage), and persuasion direction (Support/Oppose).

Model rankings remain substantively invariant across issues and stances, reinforcing the generality of the observed LLM stratification. Notably, GPT-5 matches Claude’s effect size in the "oppose" condition but trails when moving participants toward support.

LLM-Assisted Strategy Discovery and Content Analysis

To identify the causal pathways driving these persuasive effects, the study introduces a strategy-agnostic, LLM-assisted conversation analysis pipeline that uses LLMs both for emergent strategy discovery (via GPT-5 mini) and large-scale rating (via GPT-5.2) of over 4,700 conversations.

Figure 5: Schematic of LLM-assisted, two-phase conversational strategy analysis pipeline.

Ten distinct persuasive strategies are discovered (e.g., Call-to-Action Messaging, Appeals to Morals/Values, Explicit Sources/Evidence, Argumentative Framing, etc.), with strong inter-model and prompt-driven differences in their deployment intensity. Plain prompts elicit more moral appeals, while information prompts increase explicit evidence and source citation.

Figure 6: Mean strategy ratings by model and prompt, illuminating large differences in the frequency of call-to-action and source-citing techniques.

The strongest positive association with attitude shift is Call-to-Action Messaging ($\beta\approx0.38$), whereas explicit sources/evidence—central to information prompts—has negligible effect. Argumentative Framing and Hedging/Tradeoffs yield significant negative associations with persuasion.

Figure 7: OLS coefficient estimates for the effect of each persuasion strategy dimension on individual attitude change, under various model/prompt fixed-effect controls.

Implications, Limitations, and Future Directions

The demonstrated ability of current LLMs to outperform human campaigners at persuading participants in text-based conversations has profound implications for information security, election integrity, and regulatory governance. These findings signal an urgent need for standardized risk benchmarks and ongoing monitoring of LLMs used in political or mass communication settings. The heterogeneity in prompt-model interactions further suggests that naive prompt engineering may backfire and highlights the necessity of nuanced, model-specific guardrails.

There are several key limitations: the experimental paradigm studies direct chatbot-voter persuasion rather than indirect effects (e.g., mass social media content propagation), effects may vary under different demographic or cultural contexts, and observed strategy-persuasion associations are correlational—not causally identified—due to endogenous strategy selection by LLMs.

Prospective research should pursue:

• Randomized assignment of conversational strategies (possibly using constrained generation or chain-of-thought prompting) to causally separate mechanisms.
• Expansion to polarization and information quality benchmarks.
• Longitudinal monitoring of persuasion risk as LLM capabilities advance.
• End-to-end evaluation of indirect persuasion mediated through multi-agent or networked contexts.

Conclusion

This paper establishes that state-of-the-art LLMs currently exceed human campaigners in political persuasion efficacy in matched experiments. It also shows that the persuasive effectiveness of prompting strategies is highly model-specific and at times contradicts prior findings (e.g., information prompts can reduce persuasion for GPT models). The LLM-assisted analysis reveals that the most potent strategies involve actionable messaging, not explicit evidence provision. Given these results, the regulatory, ethical, and technical communities must consider instituting continuous and comparative benchmarks for persuasion risks across deployed LLMs, and accelerate the development of targeted mitigations for large-scale influence threats.

(2603.09884)