IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities

Abstract: Software is prone to security vulnerabilities. Program analysis tools to detect them have limited effectiveness in practice due to their reliance on human labeled specifications. LLMs (or LLMs) have shown impressive code generation capabilities but they cannot do complex reasoning over code to detect such vulnerabilities especially since this task requires whole-repository analysis. We propose IRIS, a neuro-symbolic approach that systematically combines LLMs with static analysis to perform whole-repository reasoning for security vulnerability detection. Specifically, IRIS leverages LLMs to infer taint specifications and perform contextual analysis, alleviating needs for human specifications and inspection. For evaluation, we curate a new dataset, CWE-Bench-Java, comprising 120 manually validated security vulnerabilities in real-world Java projects. A state-of-the-art static analysis tool CodeQL detects only 27 of these vulnerabilities whereas IRIS with GPT-4 detects 55 (+28) and improves upon CodeQL's average false discovery rate by 5% points. Furthermore, IRIS identifies 4 previously unknown vulnerabilities which cannot be found by existing tools. IRIS is available publicly at https://github.com/iris-sast/iris.

• The paper presents a neuro-symbolic approach that integrates LLMs with static analysis to improve vulnerability detection in Java projects.
• It uses LLM-based taint specification inference and contextual alert filtering to boost precision, achieving over 70% accuracy while reducing false positives by up to 80%.
• Performance benchmarks demonstrate IRIS outperforms traditional tools, detecting 69 vulnerabilities compared to 27 by CodeQL, showcasing its practical efficacy.

IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities

The paper "IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities" introduces a novel approach that integrates LLMs with static analysis techniques to improve the efficacy of vulnerability detection in Java projects. It highlights the challenges faced by traditional static analysis tools in detecting security vulnerabilities and proposes IRIS as a solution that leverages the strengths of LLMs for whole-repository reasoning.

Introduction to Vulnerability Detection Challenges

Traditional static analysis tools like GitHub CodeQL and others rely heavily on manually crafted taint specifications for API sources, sinks, and sanitizers, which are labor-intensive and often incomplete, leading to false negatives. Furthermore, these tools suffer from low precision due to imprecise reasoning and over-approximation, generating many false positives that burden developers during triaging. Despite advances in program analysis techniques, these challenges persist, hindering effective vulnerability detection in complex projects.

Overview of the IRIS Framework

IRIS presents a neuro-symbolic approach combining the reasoning capabilities of LLMs with static analysis to detect security vulnerabilities at the project level.

Figure 1: An overview of IRIS: our neuro-symbolic approach for vulnerability detection. Given a whole Java repository, IRIS checks whether it contains a certain type of vulnerability (CWE).

1. Candidate API Extraction: IRIS begins by using static analysis to extract candidate APIs from a given Java project, focusing on potential sources, sinks, and sanitizers.
2. LLM-Based Specification Inference: LLMs are utilized to automatically infer taint specifications for these APIs. This involves labeling APIs as sources or sinks relevant to a specified vulnerability class.
3. Static Analysis Integration: The inferred specifications are integrated with a static analysis engine, such as CodeQL, to perform taint analysis and identify vulnerable paths in the code.
4. Contextual Alert Filtering: IRIS implements a contextual analysis technique that uses LLMs to filter out false positives from detected paths, significantly reducing the triaging burden.

Practical Application and Dataset

IRIS was evaluated using a newly curated dataset, CWE-Bench-Java, consisting of 120 Java projects with manually validated vulnerabilities across four common classes (CWEs). This dataset serves as a robust benchmark for assessing the capability of vulnerability detection tools given the complexity and scale of real-world projects.

Figure 2: Steps for curating CWE-Bench-Java, and dataset statistics.

Performance Evaluation

The performance of IRIS was benchmarked against state-of-the-art static analysis tools and a variety of LLMs including GPT-4 and DeepSeekCoder.

Figure 3: Estimated precision of inferred source and sink specs by selected LLMs.

• Vulnerability Detection: IRIS outperforms top static analysis tools by detecting 69 vulnerabilities with GPT-4, compared to 27 by CodeQL, showcasing substantial improvements across all CWEs.
• Source/Sink Precision: The LLM-based inference achieves precision levels of over 70% for source and sink specifications, mitigating the limitations of traditional taint inference methods.
• False Positive Reduction: Contextual filtering using LLMs reduces false alarms by over 80% in some cases, easing the developer's workload during vulnerability triaging.

Conclusion

IRIS demonstrates the substantial benefits of integrating LLMs with static analysis to enhance vulnerability detection in software engineering. By effectively inferring taint specifications on the fly and minimizing false positives through context-based reasoning, IRIS addresses key limitations of existing tools, paving the way for more accurate and efficient security analyses. Future research may explore expanding IRIS to other programming languages and deeper integration of dynamic analysis components to handle more complex vulnerability patterns.