DataPhish2025: AI and Human Phishing Benchmark
- DataPhish2025 is a modern phishing benchmark comprising 12.3k emails with roughly 75% LLM-generated and 25% human-written attacks.
- It evaluates detection systems using a two-stage process with privacy-preserving PHI/PII redaction and semantic retrieval to enhance recall and precision.
- The benchmark supports robustness analysis across diverse creator sources and emotional cohorts, demonstrating performance gains from symbolic to RAG-based methods.
DataPhish2025 denotes, most explicitly in the current arXiv literature, a contemporary phishing benchmark used to evaluate modern phishing detection under mixed human-written and AI-generated threats. In CyberCane, it is described as a 12.3k-email corpus with a mixed authorship distribution of roughly 75% LLM-generated and 25% human-written phishing emails, and is used to test whether a privacy-preserving, retrieval-augmented phishing detector can generalize across both authoring sources (Hakim et al., 26 Apr 2026). In adjacent 2025 phishing-dataset discussions, the label also appears in connection with a phishing webpage resource collector and its released sample dataset, indicating that the term has been used in more than one dataset-centric context (Kulkarni et al., 11 Sep 2025).
1. Definition and benchmark role
In CyberCane, DataPhish2025 is the benchmark for evaluating phishing detection on modern, AI-generated phishing rather than only on older, template-like attacks. The paper repeatedly emphasizes that the dataset spans “template-based to GPT-generated attacks” and that it contains both human-written and AI-generated phishing emails. Its function is therefore not merely archival. It is a stress test for generalization under contemporary phishing conditions, especially where semantically coherent, technically clean messages evade lightweight symbolic heuristics (Hakim et al., 26 Apr 2026).
The benchmark is also structurally important to CyberCane’s architecture. The training split is indexed for retrieval, but only phishing training rows are used in the retrieval corpus. This makes DataPhish2025 not only an evaluation set but also the source of retrieval examples for the RAG stage. A plausible implication is that the benchmark is designed to support both discriminative evaluation and phishing-only semantic grounding.
2. Corpus composition and split structure
The paper states that DataPhish2025 has 12,300 emails total, and Table 3 breaks this into train, validation, and test partitions as follows (Hakim et al., 26 Apr 2026):
| Split | Benign | Phishing |
|---|---|---|
| Training | 4,932 | 3,063 |
| Validation | 269 | 131 |
| Test | 711 | 1,589 |
The same source states that this yields a test set of 2,300 phishing-related emails in the robustness analysis, with 37 messages lacking creator attribution excluded from creator-source breakdowns but still included in the overall evaluation. Separately, the paper reports that across the 2,337 test emails used in one evaluation view, 50.7% contained at least one redacted PII field. The article’s own reporting therefore preserves both numbers as stated (Hakim et al., 26 Apr 2026).
The appendix provides a more granular characterization of the test portion. In the creator-source breakdown, phishing emails include human-written emails (559) and LLM-generated emails from multiple model families, including GPT-4o/OpenAI (596), DeepSeek-Chat (602), Mistral Medium 3.1 (54), Gemini 1.5 Pro (9), GPT-5 Mini (9), and some grouped frontier model buckets. In the emotional cohort analysis, the test portion is stratified into 80 altruism, 976 curiosity, 432 greed, 888 neutral, 908 authority, 655 fear, and 1,177 urgency. These stratifications make DataPhish2025 useful for robustness studies that go beyond aggregate accuracy (Hakim et al., 26 Apr 2026).
3. Privacy-preserving preprocessing and retrieval configuration
All DataPhish2025 evaluations in CyberCane use the same privacy pipeline: PHI/PII redaction is applied before any external API calls. The redaction patterns cover email addresses, phone numbers, SSNs, and credit cards; the appendix gives explicit regex-based replacements such as masking emails to forms like a****[email protected], phones to ***-***-1234, SSNs to ***-**-6789, and credit cards to masked groups (Hakim et al., 26 Apr 2026).
The paper reports that, across the 2,337 test emails used in one evaluation view, 50.7% contained at least one redacted PII field, with a mean of 1.63 items per email. It further states that LLM-generated emails averaged 1.73 PII items/email versus 1.30 for human-written emails. These statistics matter because CyberCane’s central claim is that privacy-preserving preprocessing does not meaningfully degrade detection performance on DataPhish2025 (Hakim et al., 26 Apr 2026).
The retrieval subsystem is calibrated on the validation split. The threshold search ranges over Phase 1 thresholds and RAG similarity thresholds , and the selected operating point is Phase 1 threshold 2 and RAG threshold 0.70, chosen to minimize FPR while maintaining recall above 15%. The retrieval corpus is built from 2,297 labeled phishing emails from the training split, embedded with text-embedding-3-small and indexed using HNSW in PostgreSQL/pgvector. For each email, Phase 2 uses top- nearest neighbors. These details make DataPhish2025 a benchmark with a specified privacy-and-retrieval operating regime rather than a raw corpus alone (Hakim et al., 26 Apr 2026).
4. Benchmark results in CyberCane
On DataPhish2025, CyberCane’s main comparison is between its Phase 1 symbolic-only baseline and the Phase 2 RAG stage. Table 3 reports that Phase 1 gets precision 93.4%, recall 20.5%, and F1 0.336, while the RAG stage reaches precision 98.2%, recall 99.1%, and F1 0.987. The paper highlights this as a 78.6 percentage-point recall gain over symbolic-only detection on AI-generated threats (Hakim et al., 26 Apr 2026).
A key result is that 79.5% of phishing emails in the DataPhish2025 test set scored zero in Phase 1, meaning they bypassed the symbolic rules entirely, yet Phase 2 correctly classified 99.0% of those evaded emails. This establishes the benchmark’s value as a failure case for purely rule-based detection and as a high-recall setting for semantic retrieval and ontology-guided reasoning (Hakim et al., 26 Apr 2026).
DataPhish2025 is also the substrate for CyberCane’s formal reasoning components. The paper gives an ontology coverage score,
with an acceptance threshold . This score is used by PhishOnt, the OWL ontology that maps indicators like missing MX/SPF/DMARC, URL obfuscation, urgency, and credential requests into multi-label phishing attack types with verifiable reasoning chains. In this configuration, DataPhish2025 functions as both a semantic retrieval benchmark and an ontology-reasoning benchmark (Hakim et al., 26 Apr 2026).
5. Robustness across creator sources and persuasion styles
A common misconception is that DataPhish2025 is only an AI-generated phishing dataset. The paper explicitly uses it to test whether CyberCane can detect both human-written and AI-generated phishing, and reports that performance remains high across both (Hakim et al., 26 Apr 2026).
In the creator-source robustness table, Phase 2 RAG achieves F1 between 0.976 and 1.000 across 18 creator sources, including 0.985 for human-written and 0.987 for LLM-generated emails. The paper also reports creator-specific recall transitions from Phase 1 to Phase 2: human-written 15.6% to 99.2%, OpenAI/GPT-4o 19.6% to 99.3%, DeepSeek-Chat 22.4% to 99.0%, Mistral Medium 3.1 7.7% to 97.4%, Gemini 1.5 Pro 12.5% to 100%, and GPT-5 Mini 0% to 100% (Hakim et al., 26 Apr 2026).
The emotional-stratified breakdown shows similarly sharp gains. Phase 1 recall is 9.8% for altruism, 5.0% for curiosity, 2.8% for greed, 4.2% for neutral, 35.9% for authority, 39.9% for fear, and 28.9% for urgency. Phase 2 rises to 100%, 99.2%, 98.2%, 99.0%, 99.9%, 99.1%, and 99.1% respectively. These results are significant because they show that the benchmark is not only mixed by author source but also stratified by socially engineered affective style (Hakim et al., 26 Apr 2026).
The paper interprets the near-identical F1 for human-written and AI-generated phishing as evidence that the detector is not merely fitting stylistic artifacts of a particular generator. This suggests that DataPhish2025 tests semantic attack structure rather than only surface fluency.
6. Position within phishing dataset research
Within the broader 2024–2026 phishing literature, DataPhish2025 occupies a specific position. It is an email benchmark centered on mixed human/LLM phishing, privacy-preserving preprocessing, and phishing-only retrieval. This differentiates it from phishing website and multimodal resources such as the 2,063-sample screenshot-based phishing intention ground-truth dataset and the larger ~9K profiling dataset used by PhishIntentionLLM (Li et al., 21 Jul 2025), as well as the released sample webpage-resource dataset containing 4,056 legitimate and 5,666 phishing URLs with associated HTML, CSS, JavaScript, favicon, images, and screenshots (Kulkarni et al., 11 Sep 2025).
It also differs from older and more qualitatively coded email corpora. The Phishing Codebook benchmark is built from 503 phishing emails collected between 2015 and 2021 and is intended to capture descriptive, human-facing cues such as claimed sender, salutation, threatening language, urgency cues, and requested action (Saka et al., 2024). By contrast, DataPhish2025 is explicitly contemporary and is repeatedly framed as a benchmark for modern, AI-generated phishing (Hakim et al., 26 Apr 2026).
A further contrast appears in synthetic-data frameworks. PEEK is presented as a system for augmenting phishing email datasets with evolving LLM-generated phishing and reports increasing the proportion of usable phishing samples from 21.4% to 84.8% (Chen et al., 2024). DataPhish2025, by comparison, is used as an evaluation benchmark rather than as a generation framework. This suggests that it fills the role of a contemporary testbed against which privacy-preserving semantic detectors can be calibrated and compared.
Overall, DataPhish2025 is best understood as a modern phishing benchmark that operationalizes three research pressures simultaneously: mixed human/LLM authorship, privacy-preserving preprocessing, and robustness evaluation across creator sources and emotional cohorts. In the literature examined here, those properties make it a focal dataset for studying whether phishing detection systems can remain accurate when the attacks are semantically polished, psychologically varied, and unsuitable for unredacted transmission to external APIs (Hakim et al., 26 Apr 2026).