---
title: 'DataPhish2025: AI and Human Phishing Benchmark'
url: https://www.emergentmind.com/topics/dataphish2025
type: topic
---

# DataPhish2025: AI and Human Phishing Benchmark

DataPhish2025 denotes, most explicitly in the current arXiv literature, a contemporary phishing benchmark used to evaluate modern phishing detection under mixed human-written and AI-generated threats. In CyberCane, it is described as a **12.3k-email corpus** with a mixed authorship distribution of **roughly 75% LLM-generated and 25% human-written phishing emails**, and is used to test whether a privacy-preserving, retrieval-augmented phishing detector can generalize across both authoring sources [2604.23563]. In adjacent 2025 phishing-dataset discussions, the label also appears in connection with a phishing webpage resource collector and its released sample dataset, indicating that the term has been used in more than one dataset-centric context [2509.09592].

## 1. Definition and benchmark role

In CyberCane, DataPhish2025 is the benchmark for evaluating phishing detection on **modern, AI-generated phishing rather than only on older, template-like attacks**. The paper repeatedly emphasizes that the dataset spans **“template-based to GPT-generated attacks”** and that it contains both **human-written** and **AI-generated phishing emails**. Its function is therefore not merely archival. It is a stress test for generalization under contemporary phishing conditions, especially where semantically coherent, technically clean messages evade lightweight symbolic heuristics [2604.23563].

The benchmark is also structurally important to CyberCane’s architecture. The training split is indexed for retrieval, but **only phishing training rows are used in the retrieval corpus**. This makes DataPhish2025 not only an evaluation set but also the source of retrieval examples for the RAG stage. A plausible implication is that the benchmark is designed to support both discriminative evaluation and phishing-only semantic grounding.

## 2. Corpus composition and split structure

The paper states that DataPhish2025 has **12,300 emails total**, and Table 3 breaks this into train, validation, and test partitions as follows [2604.23563]:

| Split | Benign | Phishing |
|---|---:|---:|
| Training | 4,932 | 3,063 |
| Validation | 269 | 131 |
| Test | 711 | 1,589 |

The same source states that this yields a test set of **2,300 phishing-related emails** in the robustness analysis, with **37 messages lacking creator attribution excluded from creator-source breakdowns but still included in the overall evaluation**. Separately, the paper reports that across the **2,337 test emails used in one evaluation view**, **50.7% contained at least one redacted PII field**. The article’s own reporting therefore preserves both numbers as stated [2604.23563].

The appendix provides a more granular characterization of the test portion. In the **creator-source breakdown**, phishing emails include **human-written emails (559)** and LLM-generated emails from multiple model families, including **GPT-4o/OpenAI (596), DeepSeek-Chat (602), Mistral Medium 3.1 (54), Gemini 1.5 Pro (9), GPT-5 Mini (9)**, and some grouped frontier model buckets. In the **emotional cohort analysis**, the test portion is stratified into **80 altruism, 976 curiosity, 432 greed, 888 neutral, 908 authority, 655 fear, and 1,177 urgency**. These stratifications make DataPhish2025 useful for robustness studies that go beyond aggregate accuracy [2604.23563].

## 3. Privacy-preserving preprocessing and retrieval configuration

All DataPhish2025 evaluations in CyberCane use the same privacy pipeline: **PHI/PII redaction is applied before any external API calls**. The redaction patterns cover **email addresses, phone numbers, SSNs, and credit cards**; the appendix gives explicit regex-based replacements such as masking emails to forms like `a****b@domain.com`, phones to `***-***-1234`, SSNs to `***-**-6789`, and credit cards to masked groups [2604.23563].

The paper reports that, across the **2,337 test emails used in one evaluation view**, **50.7% contained at least one redacted PII field**, with a mean of **1.63 items per email**. It further states that **LLM-generated emails averaged 1.73 PII items/email versus 1.30 for human-written emails**. These statistics matter because CyberCane’s central claim is that privacy-preserving preprocessing does not meaningfully degrade detection performance on DataPhish2025 [2604.23563].

The retrieval subsystem is calibrated on the validation split. The threshold search ranges over **Phase 1 thresholds \(\{1,2,3,4,5\}\)** and **RAG similarity thresholds \(\{0.45, 0.50, 0.55, 0.60, 0.65, 0.70\}\)**, and the selected operating point is **Phase 1 threshold 2 and RAG threshold 0.70**, chosen to **minimize FPR while maintaining recall above 15%**. The retrieval corpus is built from **2,297 labeled phishing emails** from the training split, embedded with `text-embedding-3-small` and indexed using **HNSW in PostgreSQL/pgvector**. For each email, Phase 2 uses **top-\(k=8\)** nearest neighbors. These details make DataPhish2025 a benchmark with a specified privacy-and-retrieval operating regime rather than a raw corpus alone [2604.23563].

## 4. Benchmark results in CyberCane

On DataPhish2025, CyberCane’s main comparison is between its **Phase 1 symbolic-only baseline** and the **Phase 2 RAG stage**. Table 3 reports that Phase 1 gets **precision 93.4%, recall 20.5%, and F1 0.336**, while the RAG stage reaches **precision 98.2%, recall 99.1%, and F1 0.987**. The paper highlights this as a **78.6 percentage-point recall gain over symbolic-only detection on AI-generated threats** [2604.23563].

A key result is that **79.5% of phishing emails in the DataPhish2025 test set scored zero in Phase 1**, meaning they bypassed the symbolic rules entirely, yet **Phase 2 correctly classified 99.0% of those evaded emails**. This establishes the benchmark’s value as a failure case for purely rule-based detection and as a high-recall setting for semantic retrieval and ontology-guided reasoning [2604.23563].

DataPhish2025 is also the substrate for CyberCane’s formal reasoning components. The paper gives an ontology coverage score,
\[
c \leftarrow (m_{\forall} + m_{\exists}) / \text{den},
\]
with an acceptance threshold \(\theta = 0.3\). This score is used by **PhishOnt**, the OWL ontology that maps indicators like **missing MX/SPF/DMARC, URL obfuscation, urgency, and credential requests** into multi-label phishing attack types with verifiable reasoning chains. In this configuration, DataPhish2025 functions as both a semantic retrieval benchmark and an ontology-reasoning benchmark [2604.23563].

## 5. Robustness across creator sources and persuasion styles

A common misconception is that DataPhish2025 is only an AI-generated phishing dataset. The paper explicitly uses it to test whether CyberCane can detect **both human-written and AI-generated phishing**, and reports that performance remains high across both [2604.23563].

In the **creator-source robustness table**, Phase 2 RAG achieves **F1 between 0.976 and 1.000 across 18 creator sources**, including **0.985 for human-written** and **0.987 for LLM-generated emails**. The paper also reports creator-specific recall transitions from Phase 1 to Phase 2: **human-written 15.6% to 99.2%**, **OpenAI/GPT-4o 19.6% to 99.3%**, **DeepSeek-Chat 22.4% to 99.0%**, **Mistral Medium 3.1 7.7% to 97.4%**, **Gemini 1.5 Pro 12.5% to 100%**, and **GPT-5 Mini 0% to 100%** [2604.23563].

The **emotional-stratified breakdown** shows similarly sharp gains. Phase 1 recall is **9.8% for altruism, 5.0% for curiosity, 2.8% for greed, 4.2% for neutral, 35.9% for authority, 39.9% for fear, and 28.9% for urgency**. Phase 2 rises to **100%, 99.2%, 98.2%, 99.0%, 99.9%, 99.1%, and 99.1%** respectively. These results are significant because they show that the benchmark is not only mixed by author source but also stratified by socially engineered affective style [2604.23563].

The paper interprets the near-identical F1 for human-written and AI-generated phishing as evidence that the detector is **not merely fitting stylistic artifacts of a particular generator**. This suggests that DataPhish2025 tests semantic attack structure rather than only surface fluency.

## 6. Position within phishing dataset research

Within the broader 2024–2026 phishing literature, DataPhish2025 occupies a specific position. It is an email benchmark centered on **mixed human/LLM phishing**, privacy-preserving preprocessing, and phishing-only retrieval. This differentiates it from phishing website and multimodal resources such as the **2,063-sample** screenshot-based phishing intention ground-truth dataset and the larger **~9K** profiling dataset used by PhishIntentionLLM [2507.15419], as well as the released sample webpage-resource dataset containing **4,056 legitimate** and **5,666 phishing URLs** with associated **HTML, CSS, JavaScript, favicon, images, and screenshots** [2509.09592].

It also differs from older and more qualitatively coded email corpora. The **Phishing Codebook** benchmark is built from **503 phishing emails collected between 2015 and 2021** and is intended to capture descriptive, human-facing cues such as claimed sender, salutation, threatening language, urgency cues, and requested action [2408.08967]. By contrast, DataPhish2025 is explicitly contemporary and is repeatedly framed as a benchmark for **modern, AI-generated phishing** [2604.23563].

A further contrast appears in synthetic-data frameworks. PEEK is presented as a system for augmenting phishing email datasets with evolving LLM-generated phishing and reports increasing the proportion of usable phishing samples from **21.4% to 84.8%** [2411.11389]. DataPhish2025, by comparison, is used as an evaluation benchmark rather than as a generation framework. This suggests that it fills the role of a contemporary testbed against which privacy-preserving semantic detectors can be calibrated and compared.

Overall, DataPhish2025 is best understood as a modern phishing benchmark that operationalizes three research pressures simultaneously: **mixed human/LLM authorship**, **privacy-preserving preprocessing**, and **robustness evaluation across creator sources and emotional cohorts**. In the literature examined here, those properties make it a focal dataset for studying whether phishing detection systems can remain accurate when the attacks are semantically polished, psychologically varied, and unsuitable for unredacted transmission to external APIs [2604.23563].

Source: https://www.emergentmind.com/topics/dataphish2025