Evaluating AI Models' Capability to Automate Voice Phishing Attacks

Abstract: Voice phishing (vishing) attacks have traditionally been limited by the need for human operators. The rapid emergence of high-quality AI voice synthesis and LLMs reduces this bottleneck and enables scalable, automated scams. In this paper, we conduct a large-scale survey experiment (N=4100) and qualitative interviews (N=12) to assess U.S. adults' susceptibility to AI-powered voice phishing attacks. Participants were exposed to audio recordings or transcripts of scam scenarios generated using leading voice models such as Llama Full Duplex (Llama FD), Sesame, Gemini, OAI AVM, Play$.$AI, and ElevenLabs and the corresponding human baselines. The results show high compliance rates. Up to 36% of participants would or might comply with phishing requests in the "relative-in-distress" category. Overall compliance rate across all five scam categories was 16.5%, a striking figure given the low cost and high scalability of AI-automated voice phishing. Caller persuasiveness was the strongest predictor of compliance and certain models (most notably Sesame) achieved ratings comparable to human voices, or sometimes even slightly surpassing them. Our economic analysis suggests that while human-operated vishing is unprofitable at US wages, AI-powered vishing appears to be economically viable for several models. The primary risk of present-day AI-enabled vishing thus lies in the economics of automation rather than novel or "superhuman" persuasive techniques, though these cannot be ruled out for future systems. This raises significant concerns for the design of AI systems, consumer protection, and model release policies.

• The paper demonstrates that AI-generated voices in emotional scam contexts can drive compliance rates up to 36%, rivaling human operators.
• The paper employs a large-scale U.S. study with 4,100 adults and ablation analyses to dissect the impact of voice quality, content, and emotional cues on scam outcomes.
• The paper highlights that persuasive delivery, more than mere human-likeness, predicts scam success, urging the need for regulatory and educational interventions.

Large-Scale Assessment of AI-Driven Vishing Threats

Study Design and Methodology

The authors provide an extensive population-level evaluation of susceptibility to AI-powered voice phishing (vishing), using a representative sample of 4,100 U.S. adults. Participants were exposed to scam audio—generated via six state-of-the-art AI voice synthesis systems (Llama Full Duplex, OpenAI AVM, Google Gemini, Sesame, Play.AI, ElevenLabs)—and human-spoken baselines. Five vishing scenarios were included, spanning from impersonation of institutional support (MasterCard, Gmail) to emotionally charged personal appeals (charity donation solicitations, the classic “grandma” scam, and the “relative-in-distress” voice clone attack).

The sample was demographically representative in terms of age, gender, race/ethnicity, and education, as verified by post-stratification against U.S. 2023 Current Population Survey benchmarks.

Figure 1: Demographic balancing of the experimental sample demonstrates rigorous stratification and representativeness relative to the U.S. adult population.

In each scenario, participants rated sentiment, persuasiveness, trustworthiness, human-likeness of the caller, and stated willingness to comply with scam requests. These variables enabled ablation-style analyses: the study decomposed the contributions of voice quality, conversational content, modality (voice vs text), and interpersonal context (stranger vs cloned/familiar voice) to overall scam effectiveness.

Twelve qualitative interviews supplemented survey data, probing participant reasoning and perception. The study also implemented manipulation checks and robust measures of model detectability, including participant AI-familiarity, to gauge defense through AI literacy.

Effects of Scam Context and Compliance Patterns

The transition from neutral (non-scam) to scam scenarios sharply diminished the perception of AI-generated callers across sentiment, persuasiveness, trustworthiness, and human-likeness. The negative impact of scam framing was substantial (Cohen’s $d$ up to 1.2 for persuasiveness).

Figure 2: Scam context consistently and substantially depresses model perception scores relative to neutral interactions, highlighting strong contextual suspicion effects.

Self-reported willingness to comply with scam requests was 16.5% overall, with substantial scenario-level heterogeneity. Scams leveraging personal or emotional appeals outperformed generic account-support vishing by a factor of 3–5:

• Relative-in-distress (voice clone, ElevenLabs model): compliance up to 36%
• Donation requests, police-grandma scam: compliance odds ratio 3.0–3.1 vs. generic scams (MasterCard, Gmail)
• Human-voiced controls in emotional scams: compliance up to 32.6%

Figure 3: Odds ratios show marked elevation in compliance for personal appeal scams, especially those using cloned or familial voices.

These results indicate that AI-powered vishing, when combined with emotionally resonant scripts and, particularly, personalization, constitutes a potent attack vector exceeding the effectiveness of many classic credential-phishing attempts.

Model Benchmarks: Human Parity and the Role of Persuasiveness

Multiple AI voices (Sesame, ElevenLabs, Play.AI) approached or matched human performance in scam contexts on sentiment, trustworthiness, human-likeness, and persuasiveness. Particularly noteworthy, the Sesame model attained statistical parity with human baselines across all measured dimensions; in some emotional scam scenarios (e.g., Play.AI/ElevenLabs for donation and relative-in-distress), AI voices marginally exceeded human ratings for persuasiveness and trust.

Figure 4: In high-emotion family emergency scenarios, advanced AI voice models rival or surpass human performance metrics.

Figure 5: AI voice models approach or achieve human-level performance across multiple vishing scenarios, normalized to the human baseline.

Despite improvements in human-likeness, regression analyses reveal that persuasiveness is the strongest independent predictor of scam compliance—not fidelity to the human voice. Human-likeness did not significantly predict compliance when controlling for persuasiveness, sentiment, and trustworthiness. The implication is that the psychological narrative and delivery, rather than spectrally perfect voice synthesis, dominates victim susceptibility.

Detection and Defensibility: AI Familiarity Confers Little Protection

Contrary to expectations, participants’ general AI-familiarity and prior use of voice assistants did not significantly improve their capacity to detect synthetic voices or reduce compliance rates. For most scenarios and AI models, misclassification rates of AI voices as human remained high, especially for advanced cloning systems (e.g., Sesame and ElevenLabs).

Figure 6: User AI familiarity does not translate into improved ability to detect AI-generated voices; misclassification rates are roughly invariant across AI experience.

Qualitative analyses corroborate these results, with users citing conversational artifacts (e.g., utterance repetition, odd pacing) as cues, yet often attributing them to human script-reading or caller nervousness—further blurred by continuous advances in prosody and emotional affect modeling.

Economic Analysis and Automation Incentives

Economic modeling in the study demonstrates a marked difference in profitability between human-operated and AI-driven vishing. While vishing run by live human operators is not economically viable at U.S. wage rates (estimated loss >$27/hour), multiple AI systems (notably Gemini, Sesame, ElevenLabs) achieve positive expected profits ($1–$3/hour) at 2025 model inference costs. These numbers, coupled with the ability to easily scale out concurrent attacks, indicate that the primary risk of AI-powered vishing lies in the “automation dividend” of scalable, low-marginal-cost deployment rather than strictly superhuman deception capabilities.

Implications, Limitations, and Future Outlook

This work establishes that current AI systems enable scalable, persuasive voice phishing at effectiveness levels comparable to, and sometimes exceeding, those of human adversaries—particularly in emotionally charged attacks employing cloned voices. The main theoretical advance is quantifying the dissociation between human-likeness and actual scam compliance, shifting defense prioritization away from perfect synthetic voice detection toward educational and policy interventions targeting manipulative conversational tactics.

From a practical perspective, these findings have direct implications:

• Model governance: Current safeguards in commercial voice AI can be trivially bypassed; real mitigation requires robust provenance, monitoring, and deployment-level controls.
• Education: Users should be primed to recognize manipulative conversational strategies rather than focusing on superficial human/AI cues.
• Regulatory adaptation: The economic feasibility and scale of AI-driven vishing demands regulatory frameworks considering automation, not simply technical detection.

Technically, as voice synthesis continues to improve and costs decline, the “break-even” line will move further in favor of attackers, threatening to render traditional fraud prevention and detection obsolete for large classes of victims and attack surfaces.

Conclusion

This systematic analysis confirms that the threat landscape for voice phishing is fundamentally altered by generative AI. Although the models studied do not (yet) exhibit universally superhuman deception, their ability to cheaply scale high-quality, emotionally resonant vishing is empirically validated, with up to 36% of U.S. online adults self-reporting willingness to comply in optimal scenarios. Success is powered not by voice realism per se but by persuasive delivery and psychological manipulation, indicating that threat mitigation should address the economics and psychology of vishing, not merely the technical signature of AI synthesis. The study’s findings underscore the urgency for targeted policy interventions, continued empirical measurement, and next-generation anti-fraud technologies attuned to the realities of AI-mediated social engineering (2607.09970).