E‑PhishLLM: Phishing Benchmark & LLM Systems
- E‑PhishLLM is a term that encompasses a multilingual phishing-email benchmark dataset and a family of LLM-based detection systems designed to expose limitations in legacy corpora.
- The benchmark dataset, produced by the E‑PhishGen pipeline, comprises 16,616 emails in English, Italian, and German, challenging older datasets with rigorous classification results.
- LLM-based E‑PhishLLM systems integrate contextual reasoning, retrieval augmentation, structured prompts, and multi-agent analysis to reduce false positives and address evolving phishing tactics.
Searching arXiv for the primary papers and closely related phishing-LLM work to ground the article. arXiv search query: E-PhishLLM phishing detection LLM dataset user-centric RAG phishing detection E‑PhishLLM is a term used in recent phishing-detection literature in two closely related senses. In the narrower sense, it denotes a multilingual phishing-email benchmark generated by the E‑PhishGen framework: a dataset of 16,616 emails, equally split between benign and phishing, and distributed across English, Italian, and German. In the broader sense, it functions as a label for a family of LLM-centered phishing-defense systems that emphasize contextual reasoning, retrieval, structured prompting, external evidence, and increasingly multi-agent or multi-modal analysis rather than static rules or purely lexical models (Pajola et al., 1 Sep 2025, Barwani et al., 29 Jan 2026).
1. Scope and nomenclature
Recent arXiv work makes the name “E‑PhishLLM” semantically bifurcated. One line of work uses it as the name of a benchmark dataset intended to expose the weakness of legacy email-phishing evaluations. Another uses “E‑PhishLLM-style” language to describe system architectures for operational phishing detection built around LLMs, RAG, or agents. This dual usage is not contradictory; rather, it links a new benchmarking regime with a corresponding design space for detection systems (Pajola et al., 1 Sep 2025, Barwani et al., 29 Jan 2026).
| Usage | Core meaning |
|---|---|
| Dataset sense | A 16,616-email benchmark, equally split between benign and phishing, in English, Italian, and German |
| System sense | An LLM-based phishing-detection pipeline using contextual reasoning, retrieval, structured outputs, and external evidence |
A practical implication is that E‑PhishLLM is best understood as both a benchmark artifact and a system archetype. The dataset side addresses the representational problem of outdated corpora; the system side addresses the operational problem that standalone LLM classifiers often remain brittle, conservative, or expensive without personalization, retrieval, or auxiliary tools. This suggests that the term now marks a research program rather than a single model checkpoint.
2. E‑PhishLLM as a benchmark dataset
In the dataset sense, E‑PhishLLM is created by the E‑PhishGen pipeline, a two-module LLM framework. Module 1 generates synthetic company profiles and employee profiles; Module 2 generates benign and phishing email scenarios and then produces full email content. For each country, the number of emails is given by , where employees per company and each employee receives $10$ emails, specifically $5$ benign and $5$ phishing. The released configuration uses 250 companies for the UK+US, 60 for Italy, and 50 for Germany; malformed outputs, especially cases where JSON formatting failed, are discarded, yielding the final 16,616-email corpus. Language distribution is 69.22% English, 16.26% Italian, and 14.14% German (Pajola et al., 1 Sep 2025).
The point of this construction is not merely scale but difficulty. The dataset is explicitly positioned against classic corpora such as SpamAssassin, Enron, Nazario, LingSpam, TREC, and CEAS, which are described as mostly pre‑2010, English-only, and often conflating spam with phishing. On the English subset of E‑PhishLLM, containing 11,502 emails, detectors trained on the union of eight older datasets fail to achieve . By contrast, 12 of 14 zero-shot LLMs achieve , with claude‑3.5‑haiku at approximately $0.95$ and gpt‑3.5‑turbo at approximately $0.7$. The dataset therefore re-opens a benchmark space that older corpora had made artificially easy (Pajola et al., 1 Sep 2025).
The user study attached to the dataset reinforces this point. Thirty participants rated phishing-email quality on a 1–5 Likert scale. E‑PhishLLM phishing emails achieved a mean of $3.41$ with standard deviation 0, compared with 1 for Nazario, 2 for SpamAssassin, and 3 for Enron; the differences were reported as statistically significant with 4. A common misconception is therefore that near-perfect scores on legacy email corpora imply that phishing-email detection is largely solved. E‑PhishLLM was introduced precisely to contest that inference (Pajola et al., 1 Sep 2025).
3. E‑PhishLLM as an email-detection architecture
On the systems side, the most explicit “E‑PhishLLM-style” reconstruction is the user-centric RAG framework in “User-Centric Phishing Detection.” There the task is binary classification over emails 5, with user-specific retrieval over historical legitimate mail and threat-intelligence enrichment via VirusTotal. Emails are embedded with all‑MiniLM‑L6‑v2 into 6 vectors, indexed with FAISS, and top-7 retrieval uses 8. The prompt includes the query email, five similar historical legitimate emails, and a threat-intelligence snippet, and requires strict JSON output with Classification_decision, Phishing_score, Risk, Social_engineering_elements, Recommended_actions, and Brief_reason. On a 500-email dataset balanced at 250 legitimate and 250 phishing, Llama4‑Scout reaches accuracy 9, F1 $10$0, and reduces FPR from $10$1 to $10$2, a 66.7% reduction in false positives, while recall remains $10$3 (Barwani et al., 29 Jan 2026).
A second formulation uses zero-shot prompted classifiers with structured outputs rather than retrieval. In “Enhancing Phishing Email Identification with LLMs,” 6,867 emails are aggregated from multiple corpora, including human and LLM-generated phishing. The models output JSON fields such as Is_Phishing, Risk, Social_Engineering_Elements, Actions, and Reason. Llama‑3.1‑70b achieves accuracy $10$4, precision $10$5, recall $10$6, and $10$7, whereas Gemma2‑9b reaches accuracy $10$8 with recall $10$9 but a reported FPR of about 15%, illustrating the standard precision–recall trade-off in aggressive LLM phishing classification (Lee, 7 Feb 2025).
A third architectural strand is AdaPhish, which turns the “phish bowl” into a privacy-preserving, continuously updated LLM system. It combines GPT‑4o anonymization, text-embedding-3-small embeddings of dimension 1536, a Chroma vector database, a lazy-learning nearest-neighbor analyzer, a GPT-based analyzer, and a confidence-weighted ensemble. On a sampled evaluation from a corpus of 833,320 emails, the final ensemble achieves accuracy $5$0, precision $5$1, and recall $5$2. This suggests that E‑PhishLLM, in system form, is not a monolithic classifier but a composite pipeline in which LLM reasoning, retrieval, anonymization, and memory each play distinct roles (Meguro et al., 5 Feb 2025).
4. Extension beyond email: multi-agent and multi-modal variants
The broader design lineage around E‑PhishLLM extends beyond email classification into campaign analysis, web phishing, and suspicious-URL triage. PhishLumos reframes phishing mitigation at campaign level: starting from one URL, it investigates IP, DNS, TLS certificates, ASN, and related infrastructure through an LLM-driven multi-agent system. On 103 campaigns from JPCERT/CC data, it reports mean campaign coverage $5$3, median coverage $5$4, 77,391 new URLs found, 62,321 of which were later flagged by at least one VirusTotal engine, and median lead time $5$5 hours before expert confirmation. A key premise is that 77.0% of phishing URLs, and 82.5% of campaigns, were content-inaccessible to scanners, so cloaking becomes a signal for infrastructure pivoting rather than an analysis dead end (Chiba et al., 26 Sep 2025).
PhishDebate applies a different agentic decomposition to phishing websites. It uses four specialist agents for URL structure, HTML structure, semantic content, and brand impersonation, coordinated by a Moderator and a Judge. On a balanced 500+500 Mendeley benchmark, the GPT‑4o configuration reaches TPR $5$6, precision $5$7, accuracy $5$8, and $5$9. The framework is explicitly compared against single-agent prompting and CoT prompting, and its gain is attributed to divergent first-round analysis followed by structured debate (Li et al., 18 Jun 2025).
MemoPhishAgent pushes this further with episodic memory and multi-modal tools. It operates on suspicious URLs, screenshots, extracted images, HTML/Markdown, and external search results, and stores past reasoning trajectories as episodic memory. Across public datasets it improves recall by 13.6%, and on a real-world benchmark of URLs crawled from five social platforms it improves recall by 20%. The paper further reports that episodic memory contributes up to 27% recall gain without additional computational overhead, and that the production deployment processes 60K targeted high-risk URLs weekly with 91.44% recall. Although this work is URL-centric rather than email-centric, it clarifies the direction in which E‑PhishLLM-style systems move once they leave the benchmark setting: from single prompts toward tool orchestration, memory, and recurrent reasoning (Chen et al., 24 Feb 2026).
5. Robustness, multilinguality, and emerging attack surfaces
One of the clearest objections to naive E‑PhishLLM narratives is that LLM-based phishing detectors are themselves attack surfaces. LLM‑PEA evaluates GPT‑4o, Claude Sonnet 4, and Grok‑3 under prompt injection, adversarial paraphrasing, and multilingual attacks. On a balanced dataset, structured prompting gives GPT‑4o 95% accuracy, Claude 94%, and Grok‑3 88%. Yet prompt-injection success remains measurable: under the “Instruction Override” template, attack success rate is 4.2% for GPT‑4o, 1.3% for Claude Sonnet 4, and 12.3% for Grok‑3; under six injection templates applied to 1,134 adversarial phishing cases, ASR is 1.1%, 2.9%, and 1.6%, respectively. Multilingual degradation is sharper: Claude Sonnet 4’s FPR rises from 2.4% on the English baseline to 24.1% across Bangla, Chinese, and Hindi, while GPT‑4o rises from 10.0% to 13.7% and Grok‑3 from 24.1% to 43.3% (Hassan et al., 10 Dec 2025).
Emerging phishing vectors reinforce the same point from the human side. In real organizational simulations over more than 71k emails, traditional button phishing $5$0 and quishing $5$1 had essentially equivalent visit/read effectiveness, with $5$2 versus $5$3, and TOST established equivalence within $5$4. The same study reports that an OSINT- and GPT‑3.5‑assisted survey-style phishing email $5$5 produced 31.5% landing-page visits per opened email and 14.9% credential submissions per opened email in one medium-sized company. This suggests that E‑PhishLLM cannot be limited to credential-reset clichés; it must address QR-code delivery, benign-seeming corporate pretexts, and organization-specific social engineering (Weinz et al., 17 May 2025).
Cost-efficient local LLM deployment introduces a different robustness and systems question. In “Phishing Detection in the Gen-AI Era,” quantized Qwen and DeepSeek models are compared with classical ML and compact DL. The best quantized LLMs, Qwen 2.5 32B Q8 and DeepSeek R1 Distill Qwen 14B Q8, reach about 81% and 79% accuracy, respectively, while using approximately 34 GB and 15 GB of VRAM, whereas Logistic Regression, Random Forest, and Bi‑GRU remain in the 97–99% range on the same dataset. The paper also notes that LLM-rephrased phishing degrades classical detectors more strongly than GPT‑4, but the central operational conclusion is that quantized LLMs are not yet replacements for high-throughput backbone filters; they are better viewed as specialized components for hard cases and explanations (Thapa et al., 10 Jul 2025).
6. Evaluation cautions and research trajectory
A persistent evaluative pitfall in the E‑PhishLLM area is overgeneralizing from narrow LLM demonstrations. One study of 15 LLMs on 15 randomly sampled “419 scam” emails reports mean phishing scores of 9.00 for GPT‑3.5 and GPT‑3.5‑Turbo‑Instruct and 9.87 for ChatGPT 4, but the experimental set contains only phishing emails and no legitimate controls, so it measures how strongly models score known scams rather than how well they manage false positives in mixed traffic (Patel et al., 2024).
Another important trajectory is the shift from static reference lists to dynamic references. The GEPAgent webpage framework uses GPT‑3.5 or GPT‑4‑turbo plus Google Search, Google Image Search, GPT‑4V, and a domain checker. On OpenPhish 5k plus Tranco 5k, the GPT‑4 agent reaches accuracy $5$6 and $5$7, compared with DynaPhish at approximately $5$8 accuracy and $5$9 F1. This suggests that E‑PhishLLM’s long-term evolution will likely favor live evidence acquisition, analyst-style tool use, and structured decision pipelines over static galleries or isolated zero-shot prompts (Wang et al., 2024).
The resulting research direction is comparatively clear. E‑PhishLLM, as dataset, is a corrective to saturated and historically skewed benchmarks. E‑PhishLLM, as architecture, points toward privacy-aware, multilingual, retrieval-augmented, and often agentic systems that can explain decisions, adapt to user context, and remain usable under operational false-positive constraints. A second common misconception is therefore that “using an LLM” is itself the solution. The literature instead indicates that high-performing systems emerge when LLMs are embedded inside carefully engineered retrieval, memory, validation, or multi-agent frameworks, and when evaluation explicitly includes multilinguality, prompt injection, dynamic webpages, and contemporary phishing pretexts rather than only legacy corpora (Pajola et al., 1 Sep 2025, Barwani et al., 29 Jan 2026).