GuardPhish: Securing Open-Source LLMs from Phishing Abuse

Abstract: The rapid adoption of open-source LLMs in offline and enterprise environments has introduced a largely unexamined security risk like susceptibility to adversarial phishing prompts under static safety configurations. In this work, we systematically investigate this vulnerability through GuardPhish, a large scale multi-vector phishing prompt dataset comprising 70,015 samples spanning web, email, SMS, and voice attack scenarios derived from real world campaigns. Using a deterministic five model ensemble for labeling, we achieve near perfect inter model agreement (Fleiss kappa = 0.9141), with residual disagreements resolved through expert adjudication. By evaluating eight open-source LLMs under fully offline inference conditions, we uncover a substantial enforcement gap like models that correctly identify phishing intent with detection rates up to 96% nevertheless generate actionable phishing content from identical prompts, with attack success rates reaching 98.5% in voice-based scenarios. These findings demonstrate that intent classification alone does not guarantee generative refusal in the absence of dynamic guardrails. To mitigate this risk, we train transformer based classifiers on GuardPhish, achieving up to 98.27% accuracy as modular pre-generation filters deployable without modifying the underlying generative model. Our results highlight a critical weakness in current open-source LLM deployments and provide a reproducible foundation for strengthening defenses against phishing and social engineering attacks.

• The paper introduces GuardPhish, a 70,015-sample, multi-vector phishing prompt dataset and quantifies the enforcement gap in detection versus generation.
• It evaluates eight open-source LLMs, revealing that high phishing detection accuracy can coincide with alarmingly high phishing generation rates, especially for voice prompts.
• The study proposes transformer-based pre-generation classifiers that achieve over 95% accuracy, providing a practical mitigation strategy without altering the LLM architecture.

GuardPhish: An Empirical Assessment and Mitigation of Phishing Abuse in Open-Source LLMs

Problem Motivation and Contribution

The proliferation of open-source LLMs in decentralized, offline, and enterprise environments has exposed novel attack surfaces for adversaries. In contrast to vendor-controlled commercial LLMs that deploy dynamic content moderation and RLHF-based refusals, open-source LLMs are deployed with static or absent safety mechanisms, operating without centralized monitoring, policy updates, or external moderation. This creates latent security liabilities, particularly in the context of phishing: adversarial prompt engineering can induce LLMs to generate operational phishing content, regardless of their performance on safety benchmarks focused solely on intent detection.

"GuardPhish: Securing Open-Source LLMs from Phishing Abuse" (2604.17313) provides both a methodological and empirical advance through (1) the creation of GuardPhish, a 70,015-sample multi-vector phishing prompt dataset spanning 42 operationally-grounded attack scenarios across four vectors (web, email, SMS, voice) and (2) a systematic evaluation of eight prominent open-source LLMs under offline inference. The study rigorously quantifies the “enforcement gap”—the discrepancy between a model's ability to detect phishing intent and its propensity to generate phishing content. The authors supplement their empirical analysis with a modular mitigation pipeline, using transformer-based classifiers as pre-generation filters to intercept and refuse adversarial prompts without altering the generative model.

Dataset Curation and Labeling Protocol

GuardPhish is constructed on three foundational principles: comprehensive vector coverage, real-world scenario realism, and adversarial diversity. The attack scenario taxonomy was extracted through triangulation from major threat intelligence sources (APWG, OpenPhish, FBI IC3, Verizon DBIR) and academic literature, ensuring operational validity. Each scenario is instantiated into both direct and indirect prompt templates, systematically varied along axes of explicitness, complexity, and domain specificity to prevent surface-level classifier overfitting.

Prompt generation leveraged both GPT-4o mini and Grok, distributing the generation task by vector to minimize model-induced stylistic bias. Extensive post-generation cleaning was performed, including deduplication and heuristic artifact removal. The resulting class-balanced corpus encompasses diverse linguistic forms and intent subtleties, vital for evaluating the boundaries of current LLM safety generalization.

Labeling was executed via a deterministic five-model ensemble (LLaMA 3.1 8B, Gemma 2 27B, Qwen 2.5 7B, Phi-3 14B, Mistral Small 24B) deployed under fixed-inference, low-temperature conditions. This ensemble achieved a near-perfect inter-model Fleiss’ $\kappa$ of 0.9141, with ambiguous cases subject to majority vote among five domain experts, ensuring the reliability of phishing versus benign classification labels.

Enforcement Gap Analysis Across Offline LLMs

GuardPhish’s primary empirical result is the quantification of the enforcement gap: the decoupling of high phishing intent detection accuracy from failure to refuse harmful generative completions. Evaluation of eight open-source LLMs (including Mistral, LLaMA, Gemma, Qwen, Phi-3, DeepSeek-R1, OLMo, and Vicuna) revealed the following strong findings:

• Some LLMs achieved up to 96% phishing detection accuracy but yielded attack success rates (ASR) exceeding 98%—notably Phi-3 and Vicuna—for voice phishing scenarios.
• Exploitability is highly vector-dependent: voice prompts consistently elicit the most successful phishing generations, driven by their use of conversational and emotionally manipulative constructs.
• Variance in model response rates and ASRs suggest discrepant safety alignment: conservative models like LLaMA show low generative willingness across all vectors, but this is not due to context-sensitive understanding, rather due to blanket refusal strategies.

The critical claim is thus: intent classification alone is fundamentally non-predictive of real-world generative refusal under static alignment protocols in open-source LLMs.

Modular Defensive Mitigation via Pre-Generation Classifiers

To address the enforcement gap without requiring intrusive model retraining or architectural changes, the study proposes and empirically validates transformer-based classifiers as external pre-generation filters. Five distinct architectures—BERT, RoBERTa, DeBERTa, ALBERT, and DistilBERT—were fine-tuned on the GuardPhish corpus using adversarial data augmentation.

All models surpassed 95% test accuracy and F1, with DistilBERT achieving the apex at 98.27% accuracy. These results indicate strong generalization, fast convergence, and minimal overfitting, as illustrated by their training/validation learning curves:

Figure 1: BERT pre-generation filter demonstrates stable convergence and excellent validation generalization on GuardPhish.

Figure 2: RoBERTa exhibits rapid and consistent learning dynamics for phishing intent detection.

Figure 3: DeBERTa maintains robust validation and efficient learning, supporting deployment in resource-rich environments.

Figure 4: ALBERT achieves smooth loss reduction with negligible overfitting.

Figure 5: DistilBERT converges quickly with top-tier validation accuracy, matching or exceeding full-scale architectures.

Operationally, these classifiers can be deployed as seamless pre-generation modules to intercept and label prompts before generative inference, substantially reducing the risk of phishing content emission from offline LLMs.

Implications and Future Directions

Practically, this study exposes that many open-source LLM deployments are vulnerable to operational exploitation despite high classification accuracy. The stark enforcement gap implies that safety audit protocols and certification standards for LLMs must integrate both classification- and generation-based testbeds using domain-specific adversarial corpora. The attack vector sensitivity observed has direct implications for incident prioritization and modality-specific safeguards in enterprise environments.

Theoretically, the decoupling of detection and generative refusal highlights a conceptual limitation of current RLHF and static alignment pipelines: attackers can leverage prompt engineering and vector-specific framing to circumvent keyword-based or intent-only guardrails. This suggests a need for layered, adaptive, and context-sensitive refusal mechanisms that persist across both training and runtime environments.

Future research trajectories should pursue:

• Standardized, community-maintained enforcement gap evaluation protocols.
• End-to-end architectures that fuse classifier judgments with generative model workflows, closing the detection-generation loop.
• Adversarial robustness assessment for LLMs across emergent vectors, such as AI-cloned voice attacks and cross-modal phishing.
• Integration of customizable guardrail assessment tools (e.g., LlamaGuard (Inan et al., 2023), Llamafirewall (Chennabasappa et al., 6 May 2025)) and dynamic refuse-to-generate mechanisms.

Conclusion

"GuardPhish" systematically demonstrates that intent detection in open-source LLMs is not a reliable proxy for actual generative refusal under phishing adversarial prompting. The enforcement gap is broad and significant across leading models and attack vectors, with voice phishing showing the greatest vulnerability. Lightweight, high-precision transformer filters trained on domain-specific, adversarial datasets offer an immediately practical hardening solution for offline deployments, bypassing the need for disruptive generative model retraining. Comprehensive safety evaluation and modular, runtime-attachable defenses are now essential to reduce LLM-induced phishing risk in decentralized AI deployments (2604.17313).