PhreshPhish: Realistic Phishing Dataset

• PhreshPhish is a webpage-level resource comprising 372k pages with real HTML content and binary labels for phishing detection.
• It employs a multi-stage cleaning and leakage-control pipeline, including temporal isolation and similarity filtering, to ensure low label noise.
• The benchmark suite supports evaluation under realistic base rates, challenging models to maintain high precision and recall in real-world conditions.

Searching arXiv for recent and directly relevant papers on PhreshPhish and adjacent phishing-detection systems. PhreshPhish is a large-scale, high-quality dataset of real phishing and benign websites, paired with a benchmark suite for evaluating phishing-detection models under more realistic conditions than are typical in prior work. In the formulation introduced in "PhreshPhish: A Real-World, High-Quality, Large-Scale Phishing Website Dataset and Benchmark" (Dalton et al., 14 Jul 2025), each example is a URL plus the full HTML of the rendered page, with a binary label and metadata, and the benchmark framework is explicitly designed to minimize leakage, increase task difficulty and diversity, and evaluate models at phishing base rates as low as $0.05\%$.

1. Definition, scope, and motivating problem

PhreshPhish is primarily a webpage-level resource for real-time phishing detection. The task setting is deliberately narrow: given a URL and its HTML, determine whether the page is phishing or benign. Host-based features such as WHOIS, IP, and SSL are explicitly out of scope because they add latency and are less suitable for real-time detectors (Dalton et al., 14 Jul 2025).

The central motivation is methodological rather than architectural. Existing public phishing datasets are described as often small, frequently of low label quality, vulnerable to train-test leakage, built with unrealistic phishing prevalences such as $10\%$ to $50\%$, and lacking standardized, leakage-resistant benchmark splits. PhreshPhish is intended to address those limitations by providing a modern corpus of approximately $372$k pages and a benchmark suite derived from a temporally separated, leakage-filtered test set (Dalton et al., 14 Jul 2025).

A common misconception is to treat PhreshPhish as a phishing detector. In the cited work, it is not a deployed classifier, browser extension, or warning layer; it is a dataset and benchmark intended to support standardized training and evaluation. This distinction matters because the paper’s primary contribution is not a new inference pipeline, but a more realistic experimental substrate for comparing such pipelines.

2. Data acquisition, labeling, and cleaning

PhreshPhish covers eight months of web data, from July 2024 through March 2025. Phishing URLs are sourced from PhishTank, APWG eCrime eXchange, and Netcraft. Benign URLs are drawn from anonymized browsing telemetry from over 6 million Webroot users worldwide, with additional benign pages from Google search results for heavily targeted brands used later to augment benchmarks (Dalton et al., 14 Jul 2025).

For each URL, the collection pipeline uses Chrome via Selenium in a Windows cluster. It fetches the fully rendered HTML, including JavaScript-generated content, and uses varied user agents and IP addresses to mitigate cloaking and geofencing. Phishing URLs are fetched within minutes of being reported in order to counter ephemerality. Only web pages that can be rendered in this environment are included (Dalton et al., 14 Jul 2025).

The paper emphasizes that raw feeds are not equivalent to clean ground truth. The reported scraping failure modes include takedown pages, cloaking, CAPTCHAs, JavaScript-heavy pages, HTTP errors such as $404$, $403$, and $500$, and geofencing or IP blocking. Consequently, PhreshPhish applies a multi-stage cleaning pipeline combining automated heuristics and human inspection (Dalton et al., 14 Jul 2025).

The first stage is URL canonicalization and deduplication. URLs are normalized for deduplication by removing trailing slashes and lowercasing, while preserving original case for released URLs. The second stage uses page titles to remove obvious scrape failures, including titles containing strings such as “400”, “403”, “404”, “bad request”, “denied”, “not found”, “forbidden”, “error”, “warning!”, “cloudflare”, “just a moment”, and “account suspended” (Dalton et al., 14 Jul 2025).

The third stage performs similarity-based grouping and manual annotation. HTML is vectorized with TF–IDF, random projections are used to obtain binary hashes and locality-sensitive hashing bins, pages are also grouped by title, and a human annotator inspects prototypes from groups processed in decreasing size order. If a prototype is marked for removal, its nearest neighbors in TF–IDF space are removed as well. The fourth stage is PII filtering: query parameters are removed from URLs; names in URLs are detected using a DistilBERT-based NER model; and a small LLM, SmolLM2-1.7B, is asked whether a detected name is a public or well-known figure. If not, the corresponding URL and page are removed (Dalton et al., 14 Jul 2025).

An important empirical claim of the paper is comparative label quality. Under the same standardized cleaning pipeline and a fixed manual inspection budget, Aljofey exhibits rejection rates of $65.5\%$ for phishing and $35.6\%$ for benign; Crawling2024 exhibits $43.8\%$ and $10\%$0; whereas PhreshPhish exhibits $10\%$1 for phishing and $10\%$2 for benign, totaling approximately $10\%$3 overall (Dalton et al., 14 Jul 2025). This is the principal basis for the authors’ claim of substantially higher quality.

3. Dataset composition and representation

The released corpus contains about $10\%$4 pages, comprising about $10\%$5 phishing and about $10\%$6 benign examples. The public dataset is described as a webpage-level dataset in which each row includes sha256, url, label, target, date, lang, lang_score, and html, and it is stored in Parquet format (Dalton et al., 14 Jul 2025).

The dataset is intentionally raw rather than featurized. There are no pre-computed feature vectors; instead, the release provides raw URL and HTML so that downstream work can implement arbitrary URL-, DOM-, or text-based feature extraction. This choice aligns with the benchmark’s focus on low-latency, browser-compatible detection using URL and HTML only (Dalton et al., 14 Jul 2025).

The paper reports exploratory differences between phishing and benign pages at both the URL and HTML levels. At the URL level, phishing pages have minimum length $10\%$7, median around $10\%$8, and maximum $10\%$9 characters, while benign pages have minimum $50\%$0, median around $50\%$1, and maximum $50\%$2. Domain length has median approximately $50\%$3 for phishing and approximately $50\%$4 for benign, while phishing subdomain length shows high variance, up to $50\%$5 characters (Dalton et al., 14 Jul 2025). At the HTML level, phishing pages can be as small as roughly $50\%$6 bytes, with the $50\%$7th percentile around $50\%$8k and maximum around $50\%$9M, while benign pages can reach around $372$0M; the sums of head, script, and image tag lengths also vary by class (Dalton et al., 14 Jul 2025).

The paper also notes that targeted brands follow a power-law distribution and that many phishing URLs leverage benign hosting platforms that allow user-generated content, such as GitHub and Blogspot, rather than obviously malicious domains (Dalton et al., 14 Jul 2025). This suggests that the dataset is designed to stress detectors on difficult, ecologically valid cases rather than on trivially malicious infrastructure.

4. Benchmark construction and leakage control

The benchmark protocol begins with a temporally stratified partition. The cleaned dataset is sorted by collection time; the earlier portion is used for training, and a strictly future portion is used for testing. In the paper’s notation,

$372$1

and

$372$2

This temporal split is intended to prevent models from training on future phishing campaigns or web templates (Dalton et al., 14 Jul 2025).

Temporal isolation alone is treated as insufficient because similar pages can persist across time. PhreshPhish therefore applies an additional leakage filter using TF–IDF and LSH. An LSH index is built over training HTML. For each candidate test example, candidate neighbors are retrieved from the index; cosine distances are computed; and highly similar test samples are discarded. The resulting test set is thus both later in time and distinct in content from the training set (Dalton et al., 14 Jul 2025).

From this cleaned test set, the authors derive a suite of 404 benchmark datasets. The first additional step is diversity enhancement. Benign augmentation adds benign pages for top targeted brands using Google search, thereby creating hard negatives. Phishing pruning removes highly similar phishing pages so that the benchmark is not dominated by near-duplicate kits (Dalton et al., 14 Jul 2025).

The second additional step is a difficulty filter. A baseline classifier is trained on the training set and used to score test samples. Easy examples are defined as correctly classified phishing pages with $372$3 and correctly classified benign pages with $372$4, where $372$5. Then $372$6 of the entire dataset is randomly dropped from among these easy examples (Dalton et al., 14 Jul 2025). The purpose is not to simulate deployment directly, but to make evaluation more discriminative among strong models.

The third additional step is base-rate adjustment. Benchmark suites are generated at phishing prevalences of $372$7, $372$8, $372$9, $404$0, and $404$1. Given $404$2 benign examples and target prevalence $404$3, the number of phishing examples is chosen as

$404$4

To choose how many independent benchmark instances to create at each base rate, the paper uses

$404$5

where $404$6 is the relevant z-score and $404$7 is the empirical variance from pilot sampling (Dalton et al., 14 Jul 2025). The final counts are 241 instances at $404$8, 124 at $404$9, 25 at $403$0, 12 at $403$1, and 2 at $403$2, totaling 404 benchmark datasets.

A key methodological implication is explicit in the reported precision–recall behavior: as base rate increases, precision–recall curves improve, and applying the difficulty filter shifts them downward at a fixed base rate (Dalton et al., 14 Jul 2025). This supports the paper’s broader claim that many earlier phishing-detection results are optimistic because they are measured on inflated prevalences and leaky or redundant splits.

5. Baseline models and empirical findings

PhreshPhish evaluates four baseline model families: a Linear SVM, a shallow feedforward neural network, a BERT-family encoder based on gte-large, and a zero-shot LLM classifier using gpt-4o-mini (Dalton et al., 14 Jul 2025). For the classic models, the input consists of URL and HTML features, including character-level $403$3-grams of lengths $403$4, $403$5, and $403$6, XPath $403$7-grams, and hand-crafted counts such as URL length, domain length, path length, subdomain length, and DOM graph statistics. For the FNN, about $403$8M binary features are initially extracted and then truncated to the $403$9k most frequent (Dalton et al., 14 Jul 2025).

The linear baseline is a LinearSVC with class balancing and $500$0-regularized squared hinge loss,

$500$1

The FNN uses a single hidden layer with $500$2 units and ReLU activation. The GTE model mean-pools token embeddings from gte-large and fine-tunes a linear classifier for two epochs using AdamW with learning rate $500$3 and weight decay $500$4. The zero-shot LLM baseline consumes a prompt containing the URL and simplified HTML and returns JSON with phishing and score fields (Dalton et al., 14 Jul 2025).

The paper emphasizes Average Precision and Precision at Recall $500$5. On the full temporally separated, leakage-reduced test set with base rate $500$6, the reported results are: Linear model, AP approximately $500$7 and $500$8 approximately $500$9; FNN, AP approximately $65.5\%$0 and $65.5\%$1 approximately $65.5\%$2; GTE, AP approximately $65.5\%$3 and $65.5\%$4 approximately $65.5\%$5; and the zero-shot LLM, AP approximately $65.5\%$6 and $65.5\%$7 approximately $65.5\%$8 (Dalton et al., 14 Jul 2025).

The more consequential result appears under realistic prevalences. For GTE, AP declines from $65.5\%$9 on the full $35.6\%$0 test set to $35.6\%$1 at $35.6\%$2 base rate, while $35.6\%$3 declines from $35.6\%$4 to $35.6\%$5. For the linear model, AP ranges from $35.6\%$6 to $35.6\%$7 as base rate increases, and $35.6\%$8 ranges from $35.6\%$9 at $43.8\%$0 to $43.8\%$1 at $43.8\%$2, reaching $43.8\%$3 at $43.8\%$4 (Dalton et al., 14 Jul 2025). The paper’s central empirical lesson is therefore not merely that some models perform well, but that performance degrades sharply when evaluation is leakage-resistant and prevalence is realistic.

This makes PhreshPhish a benchmark about calibration under skew as much as about representation learning. The reported example that GTE achieves only about $43.8\%$5 precision at $43.8\%$6 recall when the phishing base rate is $43.8\%$7 makes explicit how difficult high-recall blocking becomes in realistic traffic (Dalton et al., 14 Jul 2025).

6. Position within the phishing-research landscape

PhreshPhish occupies an infrastructural position in phishing research rather than solving the end-to-end problem by itself. Nearby work illustrates complementary axes of the field. "Explain, Don't Just Warn!" introduces PhishXplain, a real-time explainable phishing warning system that sits on top of existing detectors, uses a structured two-step prompt architecture with a local LLaMA 3.2 model, and improves understanding, detection accuracy, and satisfaction in a $43.8\%$8-participant user study (Roy et al., 11 May 2025). "AdaPhish" addresses phishing emails rather than phishing webpages, combining GPT-4o, text-embedding-3-small, Chroma, OCR, anonymization, and a confidence-weighted ensemble to support a phish bowl, trend analysis, and education (Meguro et al., 5 Feb 2025). "PhishParrot" addresses cloaked phishing by adaptively selecting crawling environments using similar-case retrieval, MMR, and LLM-based context analysis, improving downstream detector accuracy by up to $43.8\%$9 over standard analysis systems in a $10\%$00-day evaluation (Nakano et al., 4 Aug 2025). "WebPhish" is an earlier end-to-end detection model using raw URL and HTML with embeddings and CNNs, reporting $10\%$01 accuracy on its dataset (Opara et al., 2020).

These works are adjacent to, rather than instances of, PhreshPhish itself. PhishXplain concerns post-detection explanation; AdaPhish centers on email ingestion, anonymization, and phish-bowl analytics; PhishParrot addresses anti-cloaking crawling; and WebPhish is a concrete model architecture. PhreshPhish, by contrast, is a common substrate on which such models, crawlers, and explanation systems can be evaluated more realistically (Dalton et al., 14 Jul 2025).

A plausible implication is that PhreshPhish can serve as a standard URL+HTML benchmark for approaches that currently report results on incomparable or noisy datasets. Because the release is explicitly browser-oriented and excludes host-based features, it is particularly aligned with low-latency webpage detectors and with future additions of screenshots or network traces envisioned by the authors (Dalton et al., 14 Jul 2025).

7. Access, limitations, and broader significance

PhreshPhish is publicly hosted on Hugging Face at https://huggingface.co/datasets/phreshphish/phreshphish and released under Creative Commons Attribution 4.0 International. The associated GitHub repository, https://github.com/phreshphish/phreshphish, provides scraping and cleaning code, scripts for test and benchmark split construction, and baseline training and evaluation code. The authors recommend using the benchmark suite and reporting AP and $10\%$02 averaged across benchmark instances at given base rates, rather than relying on naive random splits or inflated prevalences (Dalton et al., 14 Jul 2025).

The paper also states several limitations. It does not claim zero label noise; the phishing pages that can be scraped may be systematically easier to scrape and thus easier to detect than the most sophisticated attacks; and the current release does not include screenshots or raw network traffic, although future versions may add such modalities. The scope is web-based phishing; email phishing, SMS phishing, and other channels are outside the release (Dalton et al., 14 Jul 2025).

Security and ethics are treated explicitly. The authors argue that openness and transparency benefit defenders more than attackers, but they also acknowledge dual-use risk. To reduce privacy exposure, the pipeline strips query parameters, removes URLs likely to contain individual-level PII in benign pages, and does not scrape login-only content. They nevertheless caution that benign pages may still contain pornographic or hateful content because such pages occur in real browsing telemetry (Dalton et al., 14 Jul 2025).

The broader significance of PhreshPhish lies in its attempt to raise the evidentiary standard for phishing-detection research. Its contribution is not simply scale, but the conjunction of scale, low estimated label noise, temporal isolation, content-based leakage mitigation, difficulty filtering, diversity enhancement, and prevalence-controlled evaluation. In that sense, PhreshPhish reframes phishing detection from an often simplified classification exercise into a benchmark regime intended to approximate deployment conditions more faithfully (Dalton et al., 14 Jul 2025).