LLMpatronous: Harnessing the Power of LLMs For Vulnerability Detection (2504.18423v1)

Abstract: Despite the transformative impact of AI across various sectors, cyber security continues to rely on traditional static and dynamic analysis tools, hampered by high false positive rates and superficial code comprehension. While generative AI offers promising automation capabilities for software development, leveraging LLMs for vulnerability detection presents unique challenges. This paper explores the potential and limitations of LLMs in identifying vulnerabilities, acknowledging inherent weaknesses such as hallucinations, limited context length, and knowledge cut-offs. Previous attempts employing machine learning models for vulnerability detection have proven ineffective due to limited real-world applicability, feature engineering challenges, lack of contextual understanding, and the complexities of training models to keep pace with the evolving threat landscape. Therefore, we propose a robust AI-driven approach focused on mitigating these limitations and ensuring the quality and reliability of LLM based vulnerability detection. Through innovative methodologies combining Retrieval-Augmented Generation (RAG) and Mixtureof-Agents (MoA), this research seeks to leverage the strengths of LLMs while addressing their weaknesses, ultimately paving the way for dependable and efficient AI-powered solutions in securing the ever-evolving software landscape.

This paper, "LLMpatronous: Harnessing the Power of LLMs For Vulnerability Detection" (Yarra, 25 Apr 2025 ), explores a novel approach to automating software vulnerability detection by leveraging LLMs while mitigating their inherent weaknesses like hallucinations and knowledge limitations. The core idea is to combine Retrieval-Augmented Generation (RAG) with a Mixture-of-Agents (MoA) architecture to create a more reliable and accurate analysis system compared to traditional tools or basic LLM prompting.

Traditional vulnerability detection methods like Static Application Security Testing (SAST) and Dynamic Application Security Testing (DAST) often suffer from high false positive rates and limited understanding of complex code semantics or evolving vulnerability patterns (Yarra, 25 Apr 2025 ). While LLMs show promise due to their code comprehension abilities, their application to security tasks is hampered by potential hallucinations (generating incorrect findings) and knowledge cutoffs (lacking information on recent vulnerabilities) (Yarra, 25 Apr 2025 ). Existing LLM-based approaches, especially those relying on simple prompts, have proven insufficient for reliable vulnerability detection (Yarra, 25 Apr 2025 ).

The proposed LLMpatronous system integrates three key components: RAG, Prompt Engineering, and MoA.

1. Retrieval-Augmented Generation (RAG): To overcome LLMs' knowledge limitations and ensure analyses are based on accurate, up-to-date information, RAG is used. Before analyzing a code snippet for a specific vulnerability, the system queries an external knowledge base (like a vector database) containing detailed information on known vulnerabilities (CWEs, CVEs), code examples, and mitigation techniques (Yarra, 25 Apr 2025 ). The retrieved relevant context is then provided to the LLM, enabling it to perform an "open-book" analysis grounded in factual knowledge (Yarra, 25 Apr 2025 ).
   • Implementation Detail: This involves indexing vulnerability information into a vector database (e.g., Pinecone is mentioned in the paper) and performing similarity searches based on the vulnerability type being checked and potentially code characteristics. The retrieved text segments are then included in the prompt to the LLM.
2. Prompt Engineering: Carefully crafted prompts are used to guide the LLM agents on their specific task: analyzing the provided code snippet, vulnerability information, and RAG context to determine the presence and nature of a potential vulnerability (Yarra, 25 Apr 2025 ). Prompts define the expected output format and reasoning steps.
3. Mixture-of-Agents (MoA): This architecture employs multiple LLMs collaboratively to analyze the inputs and refine the findings (Yarra, 25 Apr 2025 ). In the described setup, agents process information iteratively: a first agent generates an initial assessment, and subsequent agents take the original inputs plus the preceding agent's output to refine the analysis, correct errors, or provide further validation (Yarra, 25 Apr 2025 ). This iterative refinement acts as a collaborative verification process, significantly reducing the impact of hallucinations and improving the reliability of the final output (Yarra, 25 Apr 2025 ).
   • Implementation Detail: This can be implemented as a sequence of API calls to different LLMs (open-source like Llama 3.1, Qwen2, or closed-source like GPT-4o, Gemini 1.5 Pro) where the output of one call is fed into the prompt for the next call.

The overall workflow for LLMpatronous is: Code Input $\rightarrow$ Select Code Snippet & Vulnerability Type $\rightarrow$ RAG Query $\rightarrow$ Retrieve & Synthesize Context $\rightarrow$ Prepare Prompt (Code, Vulnerability Info, RAG Context) $\rightarrow$ MoA Pipeline (Agent 1 $\rightarrow$ Agent 2 $\rightarrow$ ... $\rightarrow$ Final Agent) $\rightarrow$ Aggregated/Final Assessment (Vulnerability Present/Absent, Details).

To evaluate this approach, the researchers used Vuldroid, a deliberately vulnerable Android application (Yarra, 25 Apr 2025 ). Experiments were conducted using various LLMs, including GPT-4o, Claude-3-Haiku, Gemini 1.5 Pro/Flash, Qwen2-72B, Llama-3.1-70B/405B, and DBRX-Instruct (Yarra, 25 Apr 2025 ).

The experiments demonstrated the benefits of the RAG+MoA approach:

• Experiment 1 (Basic Prompting, Focused List): Using a single LLM (Gemini 1.5 Pro) with a predefined list of Vuldroid vulnerabilities showed limited success, missing several known issues and providing somewhat inconsistent output (Yarra, 25 Apr 2025 ).
• Experiment 2 (Basic Prompting, Expanded List): Expanding the list of potential vulnerabilities for the single LLM helped identify some additional issues but also introduced likely false positives, highlighting the risk of hallucination when scanning a broader range (Yarra, 25 Apr 2025 ).
• Experiment 3 (RAG + MoA): Applying the RAG+MoA workflow to verify the findings from Experiment 2 successfully filtered out the likely false positive ("Insecure Design") while confirming the true positives (Yarra, 25 Apr 2025 ). This validates the hypothesis that collaborative analysis grounded in external knowledge can effectively reduce false positives.

Vulnerability Candidate	File(s)	Verified by RAG+MoA
Webview XSS via DeepLink	BlogsViewer.java	True
Webview XSS via Exported Activity	YoutubeViewer.java	True
Steal Files via WebView (XHR)	YoutubeViewer.java, NotesViewer.java	True
Steal Password Reset Tokens	ForgetPassword.java	True
Reading User Email via Broadcasts	EmailViewer.java, MyReceiver.java	True
Intent Sniffing	SendMsgtoApp.java	True
Insecure Activity Handling	RoutingActivity.java	True
Hardcoded Credentials	Login.java	True
Insecure Design	NotesViewer.java	False
Insecure Input Validation	NotesViewer.java	True

Table adapted from the paper's results showing verification using RAG+MoA.

The paper concludes that LLMpatronous offers a more robust and reliable method for vulnerability detection than basic LLM prompting by effectively addressing knowledge gaps and hallucination-induced false positives through the combination of RAG and MoA (Yarra, 25 Apr 2025 ). Using open-source models within the MoA framework was also shown to be viable (Yarra, 25 Apr 2025 ).

However, practical implementation faces challenges. The MoA architecture increases computational cost and latency compared to single-model approaches (Yarra, 25 Apr 2025 ). The effectiveness is dependent on maintaining a high-quality, up-to-date RAG knowledge base (Yarra, 25 Apr 2025 ). The system still exhibited false negatives, missing some known vulnerabilities (Yarra, 25 Apr 2025 ). Scaling to large, complex codebases and generalizing to different languages requires further work (Yarra, 25 Apr 2025 ).

Future work suggestions include optimizing MoA efficiency (e.g., parallelization, specialized agents), enhancing the RAG knowledge base, refining prompting and analysis techniques to reduce false negatives, potentially fine-tuning specialized models, improving MoA aggregation methods, and evaluating the system on broader datasets and integrating it into DevSecOps workflows (Yarra, 25 Apr 2025 ).