A Hybrid, Multi-Layered Pipeline for Phishing and Threat Classification: Independently Validated URL and NLP Engines with a Calibrated Multi-Channel Fusion Stage

Abstract: Phishing is a multi-modal threat. We present a hybrid pipeline that scores each modality with its own engine and fuses the results. Three engines are built, deployed, and independently benchmarked: a four-stage URL stack (Domain Guard, lexical model, threat intelligence, and an asymmetric L2 fusion sidecar); a generalization-hardened DistilBERT NLP classifier whose held-out real-phishing recall rises from 0.8% to 87.3%; and a threat-intelligence synchronizer with end-to-end OpenTelemetry instrumentation confirming 1:1 message conservation. A decision-level fusion stage, characterized on a 10,677-email whole-system benchmark, reaches F1=0.914 with a calibrated probabilistic-OR over URL, header, and phishing-probability channels while reducing held-out real-spam false positives to 3.6%. Because that benchmark uses proxy URL and header channels and an operating point still needing recalibration, we present it as a preliminary integrated result. For deployable detection, the limiting factor is how well a model generalizes, not how accurately it scores data drawn from its own training distribution.

• The paper demonstrates that a hybrid pipeline combining independent URL and NLP engines with calibrated probabilistic-OR fusion significantly improves phishing classification accuracy.
• The URL engine leverages deterministic checks and a gradient-boosted model achieving near-perfect MCC and minimal false positives on benchmark datasets.
• The NLP classifier, built on DistilBERT with adversarial training, enhances phishing recall and generalizes effectively against unseen threats.

Hybrid Multi-Layered Phishing and Threat Classification Pipeline

System Overview and Architecture

The paper introduces a modular, multi-engine phishing and threat classification architecture, each modality handled by an independent engine and combined via a calibrated fusion layer (2606.21690). The pipeline comprises three core, benchmarked engines: a four-stage URL analysis stack (Domain Guard, lexical model, threat intelligence, and an asymmetric L2 fusion sidecar); a generalization-hardened DistilBERT-based NLP classifier; and a production-grade threat intelligence synchronizer. All engines are deployed as microservices over a Kafka data plane with end-to-end OpenTelemetry instrumentation.

Mail enters through an ingest service, is parsed for URLs, headers, and attachments, and then analyzed in parallel by modality-specific engines. Each engine emits a structured score envelope, which are aggregated and fused at a decision stage using a calibrated probabilistic-OR combiner. Channel scores are further mapped to discrete verdict bands.

Dataset Construction and Bias

The URL corpus merges PhiUSIIL and LegitPhish, post-cleaning yielding $299,306$ URLs ($45.2\%$ legitimate, $54.8\%$ phishing). Each URL is exhaustively labeled and subjected to detailed feature extraction. The NLP email corpus is constructed from six real-world corpora, including Kaggle's 3-class dataset, Nazario, Nigerian-419, and historical spam sources. Messages are deduplicated and fingerprinted to avoid campaign leakage across splits, with deliberate over-representation of phishing to anchor decision boundaries. The fundamental corpus bias is the marked difference between balanced training distributions and real-world traffic (roughly $1\%$ phishing for URLs and $5\%$ for email), necessitating conservative deployed thresholds and eventual recalibration.

URL Engine: Deterministic Guard and L2 Fusion

The URL stack begins with Domain Guard, which leverages apex extraction, a popularity allowlist (Cisco Umbrella), Levenshtein typosquatting, and brand-in-subdomain detection. Guard latency ranges from nanoseconds (allowlist/brand scan) to microseconds (typosquat check). URLs bypassing the guard advance to a lexical model: a gradient-boosted ensemble (XGBoost/LightGBM) trained on 30-dim feature vectors emphasizing domain entropy, character probabilities, TLD legitimacy, and continuation rate. On held-out URLs, LightGBM achieves MCC $0.99645$, accuracy $99.824\%$, FPR $0.099\%$.

The L2 fusion sidecar blends character-level URL probabilities with operational host features via histogram gradient boosting. Dedicated zone masks modulate the fusion weights, correcting for discordant classifier sensitivities (URL model over-flags banking domains; operational model over-flags CDN-fronted phishing). The deployed asymmetric rules yield robust detection, with clean held-out FPR at $1.3\%$ and detection bounds spanning $100\%$ on fresh OpenPhish to $45.2\%$0 on subtle generic-domain phishing.

Threat Intelligence Synchronization

The synchronizer ingests external feeds (PhishTank, OpenPhish, URLhaus, ThreatFox, MalwareBazaar) into persistent storage and maintains a low-latency blocklist cache. Risk scores are merged on upsert, with corroboration derived from feed counts. Complete cycles over thousands of indicators finish in seconds. Design prioritizes clear separation between bulk indicators, malware hashes, and costly enrichment tables.

NLP Email Engine: Generalization Hardened DistilBERT

The NLP classifier employs DistilBERT with three heads: classification (legitimate, spam, phishing) via softmax/focal loss ($45.2\%$1), multi-label intent taxonomy, and scalar urgency. Preprocessing strips URLs and senders, enforcing canonicalization, leetspeak normalization, and homoglyph folding; this ensures model focus is on intent rather than memorized link structures. Robustness is enforced through FGSM adversarial training and noise augmentation.

Major dataset redesign—campaign-aware split, paired contrastive buckets, matched legitimate/phishing twins—replaces overfit metrics with genuine generalization. Cycle-12 checkpoint attains macro $45.2\%$2, phishing recall $45.2\%$3, and crucially, held-out real-phishing recall increases from $45.2\%$4 (overfit) to $45.2\%$5 (cycle-12). Deployed model passes metamorphic invariants and deployability gates with zero observed evasions.

Quantization tests (INT8 dynamic/static) reveal unacceptable score perturbation for borderline cases; therefore, the deployed model retains full 32-bit precision (median CPU latency $45.2\%$6 ms).

Decision-Level Fusion and System Benchmark

Fusion is performed via a calibrated probabilistic-OR. This noisy-OR combines per-channel probabilities (URL, header, NLP/phishing-probability, attachment), ensuring single confident signals propagate unattenuated across verdict bands. A whole-system benchmark with $45.2\%$7 emails (realistic legitimate/spam/phishing mix) demonstrates that dropping the content-risk channel (which, due to confusion between spam and phishing, triggers high false positives) in favor of a dedicated phishing-probability head yields $45.2\%$8 and reduces real-spam false positives from $45.2\%$9 to $54.8\%$0.

Attempted learned fusion (gradient-boosted stackers) yielded superior in-corpus recall but failed to generalize due to family memorization; simple calibrated combiners are empirically best.

Numerical Results and Claims

• URL model (LightGBM): MCC $54.8\%$1, FPR $54.8\%$2, accuracy $54.8\%$3 (held-out).
• NLP model (DistilBERT cycle-12): held-out real-phishing recall $54.8\%$4, macro $54.8\%$5.
• L2 sidecar: clean-benign FPR $54.8\%$6, detection from $54.8\%$7 down to $54.8\%$8 depending on corpus.
• System fusion (calibrated OR): $54.8\%$9 on synthetic/real mix, real-spam misclassification reduced to $1\%$0.
• All latency numbers are well below service-level objectives.

Implications and Future Directions

This work asserts that modality separation and independent signal fusion outperforms monolithic classifiers, especially in phishing risk adjudication. Modality-specific preprocessing and robust adversarial training are essential for NLP generalization. Decision-level fusion—when calibrated—overcomes spam/phishing confusion and is robust against synthetic campaign generation.

Limitations include corpus biases (Latin-centric, synthetic data contamination, class imbalance), proxy channels for fusion benchmarking (URL/header are stand-ins), and thresholds needing recalibration on production traffic. The absence of a large, labeled multi-modal phishing corpus with live URLs and headers restricts retraining and validation of fusion strategies.

The theoretical implication is that calibration and independence of signals are key to adversarial robustness and generalization. Practically, the pipeline enables modular upgrades: replacing any channel or fusion engine is straightforward, provided channels remain independent. Once sufficient real data is available, fusion can be re-optimized as a learned meta-classifier, replacing hand-tuned rules.

Conclusion

The hybrid pipeline demonstrates authoritative modality separation, robust generalization (particularly for NLP), deterministic/probabilistic URL stacking, efficient threat intelligence synchronization, and decision-level fusion with strong $1\%$1 performance and low false-positive rates in realistic benchmarks. The primary bottleneck is not classifier accuracy but cross-modality generalization to unseen threats. Future work entails assembling comprehensive multi-channel corpora, recalibrating fusion thresholds live, and introducing learned fusion strategies contingent on real data availability.