Benchmarking Large Language Models for Zero-shot and Few-shot Phishing URL Detection

Abstract: The Uniform Resource Locator (URL), introduced in a connectivity-first era to define access and locate resources, remains historically limited, lacking future-proof mechanisms for security, trust, or resilience against fraud and abuse, despite the introduction of reactive protections like HTTPS during the cybersecurity era. In the current AI-first threatscape, deceptive URLs have reached unprecedented sophistication due to the widespread use of generative AI by cybercriminals and the AI-vs-AI arms race to produce context-aware phishing websites and URLs that are virtually indistinguishable to both users and traditional detection tools. Although AI-generated phishing accounted for a small fraction of filter-bypassing attacks in 2024, phishing volume has escalated over 4,000% since 2022, with nearly 50% more attacks evading detection. At the rate the threatscape is escalating, and phishing tactics are emerging faster than labeled data can be produced, zero-shot and few-shot learning with LLMs offers a timely and adaptable solution, enabling generalization with minimal supervision. Given the critical importance of phishing URL detection in large-scale cybersecurity defense systems, we present a comprehensive benchmark of LLMs under a unified zero-shot and few-shot prompting framework and reveal operational trade-offs. Our evaluation uses a balanced dataset with consistent prompts, offering detailed analysis of performance, generalization, and model efficacy, quantified by accuracy, precision, recall, F1 score, AUROC, and AUPRC, to reflect both classification quality and practical utility in threat detection settings. We conclude few-shot prompting improves performance across multiple LLMs.

• The paper evaluates three proprietary large language models (LLMs): GPT-4o, Claude-3.7, and Grok-3-Beta for phishing URL detection using zero-shot and few-shot prompting.
• Grok-3-Beta outperforms other models in the few-shot setting, achieving the highest accuracy (0.9405) and AUPRC (0.9573).
• The study finds that under class imbalance, adding in-context examples does not consistently improve performance across all models; Grok-3-Beta's performance declines with additional examples.

Motivation and problem setting

Phishing URL detection has historically been dominated by blacklist-based systems and supervised classifiers trained on handcrafted lexical, host-based, or structural features. Both families share a common weakness: they depend on prior knowledge of attack patterns, which degrades rapidly as adversaries generate novel or obfuscated URLs. The authors motivate their study with threat-intelligence figures indicating that phishing volume has escalated over 4,000% since 2022, with nearly half of attacks evading conventional filters. Their central premise is that when phishing tactics emerge faster than labeled data can be produced, prompt-only inference with instruction-tuned LLMs offers an adaptable alternative requiring minimal supervision.

The specific gap the paper targets is the absence of standardized comparisons among proprietary LLMs under unified zero-shot and few-shot prompting conditions. Prior work either fine-tunes a single model (e.g., PhishURLDetect with LoRA), benchmarks only open-source models with English prompts, or evaluates models under inconsistent datasets and metrics. This paper addresses that gap with a controlled benchmark of GPT-4o (OpenAI), Claude-3-7-sonnet-20250219 (Anthropic), and Grok-3-Beta (xAI).

Methodology

The evaluation uses the PhiUSIIL Phishing URL dataset from the UCI Machine Learning Repository, retaining URL strings and binary labels (0 = phishing, 1 = legitimate). Two experimental regimes are constructed:

• Balanced regime: 10,000 URLs sampled with equal class proportions (5,000 each, seed 42). Zero-shot runs evaluate all 10,000 samples; few-shot runs reserve 6 examples (3 per class) as $\mathcal{E}$ and evaluate on the remaining 9,994.
• Imbalanced regime: test sets of 1,000 URLs with phishing ratios of 1% and 10%, each replicated under two random seeds (S123, S456). Few-shot experiments at 10% imbalance vary the example count over $\mathcal{E} \in \{1, 3, 9\}$.

Prompts follow a model-agnostic template: a task instruction ("You are a cybersecurity expert. Respond only with 0 for phishing or 1 for legitimate."), optionally followed by formatted examples, then the query containing the target URL. GPT-4o and Grok-3-Beta receive system/user message structures; Claude-3.7 receives concatenated text separated by double newlines. Inference uses temperature 0 with output capped at 10 tokens, and unparseable responses are excluded. Metrics are macro-averaged accuracy, precision, recall, F1, AUROC, and AUPRC, computed via scikit-learn.

A notable design choice is the use of balanced data despite real-world imbalance; the authors justify this as enabling fair assessment of generalization, and they separately probe imbalanced distributions to capture deployment realism.

Results on the balanced dataset

Few-shot prompting with six examples improves performance across all three models. Grok-3-Beta leads in five of six metrics in the few-shot setting: accuracy 0.9405, precision 0.9492, F1 0.9399, AUROC 0.9405, and AUPRC 0.9573. Claude-3.7 attains the highest recall (0.9526) but lower precision (0.9027). GPT-4o shows consistent gains yet trails both competitors.

Model	Setting	Accuracy	Precision	Recall	F1	AUROC	AUPRC
GPT-4o	Zero-shot	0.8752	0.8421	0.9232	0.8808	0.8752	0.9018
GPT-4o	Few-shot	0.9050	0.8880	0.9270	0.9071	0.9050	0.9258
Claude-3.7	Zero-shot	0.8759	0.8778	0.8734	0.8756	0.8759	0.9072
Claude-3.7	Few-shot	0.9250	0.9027	0.9526	0.9270	0.9250	0.9395
Grok-3-Beta	Zero-shot	0.8914	0.8361	0.9735	0.8996	0.8914	0.9114
Grok-3-Beta	Few-shot	0.9405	0.9492	0.9307	0.9399	0.9405	0.9573

Confusion-matrix analysis reveals a clear precision–recall trade-off for Grok-3-Beta: moving from zero-shot to few-shot drops recall from 0.9735 to 0.9307 while raising precision from 0.8361 to 0.9492 — the few-shot examples induce stricter classification behavior. Few-shot prompting reduces false negatives across all models, most dramatically for Grok-3-Beta (from 950 to 248 on 10,000 samples), compared to GPT-4o (863 → 584) and Claude-3.7 (608 → 513). ROC and PR curves corroborate these scalar results, with Grok-3-Beta's few-shot configuration showing the steepest ROC rise and most stable PR curve.

An implication worth noting: because zero-shot Grok-3-Beta already achieves very high recall (0.9735) with few false positives (132), operators prioritizing catch-rate over false-alarm cost may prefer the zero-shot configuration even though its aggregate metrics are lower.

Results under class imbalance

Under imbalanced test sets, macro-F1 improves substantially as the phishing ratio rises from 1% to 10%: GPT-4o moves from 0.559 to 0.785, Claude-3.7 from 0.534 to 0.761, and Grok-3-Beta from 0.657 to 0.854. At 1% imbalance, all models show depressed precision (roughly 0.54–0.64) despite high accuracy (up to 0.976), illustrating how macro-averaged metrics expose minority-class weakness that raw accuracy conceals. Performance is stable across seeds S123 and S456, with F1 differences below 0.05 at 10% imbalance.

The few-shot ablation over $\mathcal{E} \in \{1, 3, 9\}$ yields non-monotonic and model-dependent behavior:

• Grok-3-Beta peaks at one example (F1 0.906, precision 0.949) and degrades with more examples (F1 0.821 at three, 0.831 at nine).
• Claude-3.7 improves monotonically, reaching F1 0.876 at nine examples.
• GPT-4o gains consistently from F1 0.709 (one example) to 0.861 (nine examples).

Grok-3-Beta also dominates zero-shot imbalanced settings, with accuracy 0.976 and recall 0.938 at S456/10%. The divergence in scaling behavior — where adding in-context examples helps two models but hurts the third — is a substantive finding: it indicates that optimal shot count is model-specific and cannot be assumed to increase monotonically.

Limitations and open questions

Several constraints qualify these results. First, the evaluation relies on a single dataset (PhiUSIIL); generalization to other phishing corpora, adversarially generated URLs, or AI-crafted deceptive URLs remains unverified, and the authors themselves call for validation across diverse datasets. Second, the few-shot example selection is random rather than curated, so the non-monotonic Grok-3-Beta result may reflect example sensitivity rather than an intrinsic capacity limit — the paper does not disentangle these factors. Third, unparseable responses are excluded from evaluation without reporting their frequency, leaving open whether refusal rates differ systematically across models. Fourth, the paper compares prompt-only inference against no fine-tuned baseline in this work, so the trade-off between prompting and parameter-efficient fine-tuning (e.g., LoRA-based approaches cited in related work) is asserted rather than measured. Finally, inference cost and latency are not reported, which matters for operational deployment claims.

Conclusion

This paper provides a controlled, metric-consistent benchmark of three proprietary LLMs for phishing URL classification under zero-shot and few-shot prompting. The principal findings are that six-shot prompting improves performance across all tested models on balanced data, with Grok-3-Beta achieving the strongest overall few-shot profile (accuracy 0.9405, AUPRC 0.9573) and Claude-3.7 the best recall (0.9526); and that under class imbalance, the benefit of additional in-context examples is model-dependent, with Grok-3-Beta peaking at a single example. The results support prompt-based LLM inference as a viable low-supervision approach to phishing URL detection, while leaving open the questions of cross-dataset generalization, example-selection sensitivity, and head-to-head comparison with fine-tuned alternatives.