KnowPhish: Large Language Models Meet Multimodal Knowledge Graphs for Enhancing Reference-Based Phishing Detection

Abstract: Phishing attacks have inflicted substantial losses on individuals and businesses alike, necessitating the development of robust and efficient automated phishing detection approaches. Reference-based phishing detectors (RBPDs), which compare the logos on a target webpage to a known set of logos, have emerged as the state-of-the-art approach. However, a major limitation of existing RBPDs is that they rely on a manually constructed brand knowledge base, making it infeasible to scale to a large number of brands, which results in false negative errors due to the insufficient brand coverage of the knowledge base. To address this issue, we propose an automated knowledge collection pipeline, using which we collect a large-scale multimodal brand knowledge base, KnowPhish, containing 20k brands with rich information about each brand. KnowPhish can be used to boost the performance of existing RBPDs in a plug-and-play manner. A second limitation of existing RBPDs is that they solely rely on the image modality, ignoring useful textual information present in the webpage HTML. To utilize this textual information, we propose a LLM-based approach to extract brand information of webpages from text. Our resulting multimodal phishing detection approach, KnowPhish Detector (KPD), can detect phishing webpages with or without logos. We evaluate KnowPhish and KPD on a manually validated dataset, and a field study under Singapore's local context, showing substantial improvements in effectiveness and efficiency compared to state-of-the-art baselines.

• The paper presents an automated multimodal approach integrating LLMs and large-scale brand knowledge graphs for enhanced phishing detection.
• It details the KnowPhish detector’s dual analysis of visual and textual data, improving recall, precision, and runtime efficiency.
• Empirical evaluations demonstrate significant gains over traditional methods, ensuring adaptable and robust detection in evolving threat landscapes.

KnowPhish: Enhancing Phishing Detection through Multimodal Knowledge Graphs

Reference-based phishing detectors (RBPDs) have advanced the state-of-the-art in automated phishing detection. However, existing methods face significant limitations due to their dependence on manually curated brand knowledge bases (BKBs) and image modality-exclusive approaches. The paper "KnowPhish: LLMs Meet Multimodal Knowledge Graphs for Enhancing Reference-Based Phishing Detection" proposes an innovative solution to these issues. KnowPhish is a large-scale multimodal BKB that allows for the integration of comprehensive brand information, significantly expanding the scope and capabilities of RBPDs to detect phishing webpages.

Automated Pipeline for Brand Knowledge Collection

The sophistication of KnowPhish arises from its automated knowledge collection pipeline, which compiles a BKB containing approximately 20,000 brands. This approach is driven by the empirical observation that phishing targets persistently belong to high-value industries, a trend supported by historical data analysis (Figure 1), ensuring stable and predictable target identification over time. KnowPhish taps into resources like Wikidata, leveraging categorical relationships such as the instance_of attribute to populate potential phishing targets within narrow and general categories.

Figure 2: An overview of our automated pipeline for constructing our large scale multimodal BKB, KnowPhish. We first collect (a) all brands from certain high-value industries, and (b) only popular brands from general categories. Then, the knowledge acquisition and augmentation steps collect logos, domains, and aliases for these brands.

KnowPhish Detector: A Multimodal Approach

The introduction of the KnowPhish Detector (KPD) enables phishing detection without being limited to the visual domain. KPD uniquely incorporates both visual and textual modalities by deploying a LLM-based approach for extracting brand information from webpage text. This advancement allows for substantial improvements in phishing detection efficiency, addressing cases where phishing websites are devoid of logos (Figure 3).

Figure 4: An overview of our phishing detector KPD.

KPD integrates with any existing RBPD through a plug-and-play mechanism, benefiting from comprehensive alias mapping and logo variants available in KnowPhish. The detector's multi-stage analysis employs both visual logo extraction and textual brand inference, significantly enhancing the coverage and precision of phishing detection.

Evaluation of Performance

The empirical evaluations conducted demonstrate the superior performance of KnowPhish and KPD over existing RBPDs. Key performance metrics such as accuracy, F1-score, and recall were substantially improved, with KnowPhish achieving better runtime efficiency due to preemptive knowledge compilation. KPD, in particular, exhibits the highest recall and precision among all tested configurations, with the ability to process a larger number of phishing pages effectively (Figure 5).

Figure 5: Top 20 phishing targets detected by KPD+KnowPhish and Phishpedia+DynaPhish on SG-SCAN.

Practical Implications and Future Directions

KnowPhish represents a paradigm shift in phishing detection by incorporating a scalable, multimodal knowledge graph that can be dynamically updated. This ensures the adaptability and robustness of RBPDs in the rapidly evolving landscape of phishing threats. Moving forward, additional integrations with other brand databases and knowledge-augmented LLMs could further enhance KnowPhish's effectiveness and broaden its applicability across different detection systems.

In conclusion, the integration of KnowPhish with existing RBPDs not only catalyzes improvements in detection performance but also addresses limitations inherent in manual and image-only detection methodologies. The multimodal capability facilitated by KnowPhish significantly extends the coverage and precision of phishing detection systems, marking a substantial advancement in cybersecurity practices.