---
title: 'E-PhishGEN: LLM-Generated Phishing Datasets'
url: https://www.emergentmind.com/topics/e-phishgen
type: topic
---

# E-PhishGEN: LLM-Generated Phishing Datasets

E-PhishGEN is an LLM-based, privacy-savvy framework for generating novel phishing-email datasets intended to support more challenging and realistic research in phishing email detection. It is introduced together with E-PhishLLM, a multilingual dataset containing 16,616 emails in English, Italian, and German, and is motivated by the claim that phishing email detection remains an open problem despite frequent reports of near-perfect benchmark accuracy. The framework synthesizes fictional company and employee profiles, then generates both benign and phishing emails tailored to organizational context and personal traits, with the explicit aim of overcoming the obsolescence, monolingual bias, and labeling ambiguities of legacy benchmarks [2509.01791].

## 1. Benchmark critique and problem framing

The central premise behind E-PhishGEN is that high reported performance on phishing-email benchmarks does not imply that phishing email detection has been solved. The associated study argues that many widely used datasets are obsolete and unrepresentative, mostly encompass the English language, and often conflate spam with phishing even though spam is not always malicious and phishing is. It also notes that few prior works release code and clear references, which complicates reproducibility and systematic evaluation [2509.01791].

This critique is directed at a recurring pattern in the literature: models often achieve near-perfect performance when trained and tested on the same dataset, yet generalize poorly across datasets. The paper explicitly presents this as a development bottleneck, asking how future research can outperform methods that are already near perfect under legacy benchmark conditions. In the broader email-phishing literature, high benchmark figures remain common; for example, MultiPhishGuard reports high accuracy at 97.89\%, with low false positive and false negative rates of 2.73\% and 0.20\%, respectively, on six major public datasets [2505.23803]. E-PhishGEN is therefore best understood as a benchmark-generation response to this evaluation paradox rather than as a detector architecture.

Three deficiencies are emphasized. First, legacy datasets such as SpamAssassin, Enron, SpamBase, and Nazario are described as mostly English-language corpora from before 2010, and thus poorly aligned with current phishing campaigns, especially campaigns generated with advanced tools like LLMs. Second, language diversity is limited even though real phishing emails can target users in any language. Third, scope and labeling are inconsistent when spam and phishing are merged into a single category. This suggests that dataset design is not a peripheral issue but a primary determinant of apparent detector progress.

## 2. Framework architecture and operating principles

E-PhishGEN is presented as a modular pipeline using LLMs to generate privacy-preserving, realistic, and diverse phishing and benign emails tailored to user-defined scenarios. Its architecture is organized into two modules: company and user profile generation, and email scenario and content generation. The framework is explicitly designed to avoid the use of real user data or inboxes, thereby addressing privacy or ethical concerns associated with sharing real emails [2509.01791].

The first module synthesizes realistic fictional organizations and employees. Company profiles include attributes such as name, industry, and region, while employee profiles include name, role, seniority, skills, hobbies, and language skills. The second module uses those profiles to generate both benign emails, such as meeting requests or routine updates, and phishing emails spanning varied social engineering strategies including credential harvesting and spear phishing. The generated phishing content is customized to the employee’s organizational context and personal traits rather than being emitted as generic attack text.

Scalability is parameterized. The framework specifies parameters $X$, $Y$, and $N$, which can be set to yield thousands of unique emails per language, country, or scenario. In the paper’s formulation, the total number of generated emails per country or language is given by
$$
\text{Total emails} = X \times Y \times N.
$$
A plausible implication is that E-PhishGEN is intended not merely as a single benchmark artifact but as a reusable dataset-construction mechanism.

## 3. Synthetic profile and email generation workflow

The profile-generation stage uses an LLM, with GPT-4o-mini given as the concrete model used in the reported dataset construction, to synthesize fictional companies reflecting realistic distributions. An illustrative company profile in the paper includes a firm based in Italy, “Fabbri Tech Automazione,” with attributes such as industrial automation systems and robotics, headquarters in Modena, and 320 employees. An illustrative employee profile includes “Marco Bianchi,” a “Junior Project Coordinator” with Italian as a native language and English at an intermediate level [2509.01791].

The email-generation stage operates over these synthetic profiles. For each profile, the system generates benign emails and phishing emails. Benign emails are described as routine organizational communication, such as scheduling a call for supply chain adjustments. Phishing emails are designed to cover multiple social engineering strategies and are framed so that they fit the individual and organizational setting; one example is an Italian-language credential-verification message with an urgent subject line and an impersonated IT role. The workflow thereby attempts to produce realistic contextual coupling between sender, recipient, organizational structure, and attack pretext.

Because all content is synthesized, privacy preservation is a design objective rather than a post hoc mitigation. The framework is described as privacy-preserving and privacy-savvy precisely because it does not rely on real inboxes or real user data. This also enables controlled variation across countries, companies, employee personas, and attack scenarios. The paper presents this as a way to construct more challenging benchmarks that reflect current phishing trends while remaining shareable and reproducible.

## 4. E-PhishLLM: dataset produced with E-PhishGEN

E-PhishGEN is instantiated in the E-PhishLLM dataset, which contains 16,616 emails: 8,308 phishing and 8,308 benign. The dataset is multilingual and includes English, Italian, and German. The language distribution is reported as follows [2509.01791]:

| Language | Emails | Percentage |
|---|---:|---:|
| English | 11,502 | 69.2% |
| Italian | 2,702 | 16.3% |
| German | 2,347 | 14.1% |

The paper further states that the English subset is drawn from US and UK contexts, while Italy and Germany provide the non-English settings. Each country is represented by multiple companies, and each company by multiple employee personas and scenarios. The LLM used to create E-PhishLLM is GPT-4o-mini. The balanced phishing/benign split is notable because it separates the question of detection difficulty from class imbalance and keeps the benchmark focused on content realism and generalization.

E-PhishLLM is intended as a modern benchmark rather than as a direct replica of historical corpora. The associated claim is that it is substantially more challenging and realistic than legacy benchmarks. This claim is supported not only by detector performance degradation but also by user-study validation. A plausible implication is that E-PhishLLM serves both as a benchmark and as a stress test for methods that appear saturated on older datasets.

## 5. Empirical benchmarking and detector behavior

The benchmarking study re-implements and re-assesses a variety of machine learning and LLM-based phishing-email detectors. The considered methods include TF-IDF + Logistic Regression, Naive Bayes, SVM, MLP, DistilBERT, gpt-4o-mini, and Gemini. Evaluation is organized into three experiment types: cross-evaluation, in which models are trained on each legacy dataset and tested on all others; all-vs-one, in which models are trained on all but one dataset and tested on the excluded dataset; and zero-shot LLM evaluation using prompt-only detectors [2509.01791].

On legacy datasets, all classic models, including TF-IDF, SVM, RF, MLP, and DistilBERT, achieve F1 scores around 0.95–0.99 when training and testing on the same dataset. However, the paper reports a significant performance drop in cross-dataset and all-vs-one evaluations, with average F1 drops by 0.3–0.7. This is presented as evidence of poor generalization. The paper therefore treats same-dataset near-perfect performance not as evidence of task closure but as an indicator of dataset-specific overfitting and benchmark saturation.

On E-PhishLLM, the degradation is sharper. Classic ML models often achieve F1-scores in the 0.5–0.75 range at best, and the paper states that detectors trained on combinations of the eight considered datasets struggle to detect phishing emails in E-PhishLLM. Prompted LLMs perform better, with F1-scores often at or above 0.8; the best reported case is Claude 3.5 Haiku up to 0.95, while the worst reported case is gpt-3.5-turbo at 0.7. The study therefore characterizes E-PhishLLM as a more stringent and realistic benchmark with substantially larger room for improvement than existing corpora.

## 6. Validation, implications, and position within phishing research

The quality of E-PhishLLM is validated through a user study involving 30 experts with cybersecurity backgrounds. Each participant assessed 20 phishing emails: 5 from E-PhishLLM and 5 each from SpamAssassin, Nazario, and Enron. Emails were rated for phishing quality, defined in terms of realism and persuasiveness, on a 1–5 Likert scale, and each email was seen by 3 users for cross-validation. The reported average ratings are 3.41 for E-PhishLLM with standard deviation 1.2, 2.65 for Nazario, 1.57 for SpamAssassin, and 1.45 for Enron; a t-test gives $p < 0.05$ versus all legacy datasets [2509.01791].

These findings position E-PhishGEN as an infrastructural contribution to phishing-email research. The framework and all code and data are reported as publicly released, and the paper explicitly presents the system as a template for generating custom datasets for particular organizations, languages, or attack types. It also identifies a dual-use dimension: the same generative process that helps defenders produce stronger benchmarks can reveal how adversaries might employ LLMs, thereby aiding proactive detection development.

Within the broader phishing literature, E-PhishGEN occupies a different layer of the research stack from detector-centric systems. Earlier work such as “Know Your Phish” addresses phishing detection for webpages through a 212-feature, language-independent, client-side system with target identification, showing performance across English, French, German, Portuguese, Italian, and Spanish [1510.06501]. By contrast, E-PhishGEN does not propose a new email detector; it proposes a method for constructing benchmarks that better reflect current phishing trends. The associated recommendation is methodological: future research should think deeply before using any given dataset for phishing-email detection and should state clearly whether the objective is to outperform previously proposed methods or to develop a method that can detect phishing emails in the real world.

Source: https://www.emergentmind.com/topics/e-phishgen