CVE-LLM: LLMs for Automated Vulnerability Analysis
- CVE-LLM is a research domain and industrial system that uses LLMs to evaluate vulnerabilities grounded in CVE records and related metadata.
- Systems in this field combine preprocessing, normalization, retrieval, static analysis, and multi-agent orchestration to enhance security reasoning.
- Empirical studies show that CVE-LLM methods improve the accuracy and scalability of detecting, localizing, and patching software vulnerabilities.
“CVE-LLM” denotes both a specific industrial system for automatic vulnerability evaluation and a broader research area in which LLMs are grounded in Common Vulnerabilities and Exposures records, CVE-linked patches, or related metadata to perform security analysis tasks. In the narrow sense, "CVE-LLM : Ontology-Assisted Automatic Vulnerability Evaluation Using LLMs" describes an MPT-7B-based system integrated into the Siemens Healthineers Cybersecurity Management System for product-specific assessment of vulnerability notifications (Ghosh et al., 21 Feb 2025). In the broader literature, the term has come to cover CVE-grounded vulnerability detection, statement localization, exploit generation and reproduction, patch localization, automated repair, CVSS scoring, ATT&CK mapping, malware-to-CVE association, secure-code preference testing, and adversarial code-review evaluation (Gonçalves et al., 2024). This body of work treats CVEs not merely as labels, but as anchors for reasoning over code, patches, environments, exploit traces, and organizational asset context.
1. Scope, definition, and problem setting
At its core, CVE-LLM research addresses the gap between public vulnerability disclosure and operational security decision-making. CVE records identify known vulnerabilities, but many downstream tasks remain unresolved: determining whether a product is affected, locating the fixing commit, finding sibling instances of the same bug pattern, generating a repair, reproducing an exploit, or inferring attacker behavior. The literature therefore uses LLMs as semantic mediators between sparse vulnerability descriptions and richer technical artifacts such as source code, patch diffs, commit histories, build environments, and program-analysis outputs (Ghosh et al., 21 Feb 2025).
The pressure for automation is explicit in several studies. One paper states that the National Vulnerability Database publishes over a thousand new vulnerabilities monthly, with a projected 25 percent increase in 2024 (Ghosh et al., 21 Feb 2025). Another reports a 38% increase from 2023 to 2024 and more than 135 new CVEs per day in the context of automatic CVSS scoring (Marchiori et al., 14 Apr 2025). In industrial deployment, Siemens Healthineers reports roughly 1.7K products and around 1.7 million vulnerability evaluations per month, framing CVE-level assessment as a scaling problem rather than a purely academic classification task (Ghosh et al., 21 Feb 2025).
A notable property of the field is task heterogeneity. Some systems operate on code-only inputs; others require CVE descriptions, CWE tags, CAPEC links, or full patch histories. Some predict binary labels, while others output environmental CVSS vectors, ATT&CK techniques, patch rankings, verifiable exploits, or pull-request approval decisions. This suggests that “CVE-LLM” is best understood as a family of CVE-grounded reasoning systems rather than a single benchmarkable capability.
2. Data foundations and benchmark evolution
The empirical trajectory of CVE-LLM research has been strongly shaped by dataset quality. Early function-level C/C++ vulnerability work based on CVEFixes motivated the "SCoPE: Evaluating LLMs for Software Vulnerability Detection" framework, which used ANTLR4-based normalization to expose flaws in the C/C++ subset of CVEFixes. From an original extraction of 15,649 entries, SCoPE identified 905 duplicate entries, including identical code with different comments, identical content, and identical code with different programmer-defined names; some duplicates had conflicting labels (Gonçalves et al., 2024). This finding established deduplication and normalization as first-order methodological concerns.
Later benchmarks explicitly targeted these weaknesses. "SecVulEval: Benchmarking LLMs for Real-World C/C++ Vulnerability Detection" defines a deduplicated, statement-level benchmark with 25,440 function samples covering 5,867 unique CVEs across 707 projects from 1999 to 2024 and 145 CWE types, together with patch-grounded statement labels and contextual metadata (Ahmed et al., 26 May 2025). "LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoning" introduces UniVul, which provides retrievable knowledge and context-supplementable code for controlled studies of intrinsic reasoning versus external assistance (Sun et al., 2024).
Other benchmarks move away from direct classification. TOSSS reframes CVE-grounded evaluation as secure-versus-vulnerable snippet selection built from MegaVul vulnerability-fix pairs (Damie et al., 11 Mar 2026). CVE-Bench evaluates whether LLM agents can exploit real-world web vulnerabilities in sandboxed deployments built from 40 critical-severity web-related CVEs (Zhu et al., 21 Mar 2025). SEVRA-BENCH reverses real CVE-linked fixes into 1,062 malicious pull requests and tests whether review agents can resist adversarial PR framing (Melo et al., 11 Jun 2026).
| Benchmark or framework | Task form | Distinctive property |
|---|---|---|
| SCoPE | Function-level vulnerability detection | 905 duplicates exposed in CVEFixes subset |
| SecVulEval | Statement-level detection and reasoning | 25,440 samples with rich context |
| TOSSS | Secure snippet selection | CVE-grounded secure vs vulnerable pairs |
| CVE-Bench | Real-world web exploitation | 40 critical-severity web CVEs |
| SEVRA-BENCH | Adversarial PR review | 1,062 malicious PRs from reversed fixes |
Across these resources, the benchmark trend is clear: later work increasingly penalizes superficial matching, leakage, weak labels, and context omission. A plausible implication is that much of the apparent progress in earlier CVE-grounded vulnerability detection depended on overly forgiving task formulations.
3. Methodological patterns
Several architectural patterns recur across CVE-LLM systems. The simplest is preprocessing-driven classification. SCoPE normalizes function-level C/C++ code by replacing programmer-defined function and variable names, replacing strings with a generic token, normalizing whitespace, removing comments, and tokenizing code, thereby reducing average token count per function from 374 to 300 before fine-tuning CodeBERT, CodeGPT, and NatGen for binary sequence classification (Gonçalves et al., 2024).
A second pattern is decoupled reasoning. LLM4Vuln explicitly separates target-code reasoning from retrieved vulnerability knowledge, tool-supplied code context, and prompt schemes. Its factorial design treats vulnerability reasoning as a controlled capability and distinguishes raw reasoning from enhanced reasoning under knowledge and context supplementation (Sun et al., 2024). This is methodologically important because several later systems rely heavily on retrieval, tool use, or structured decomposition.
A third pattern is structural grounding through program analysis. "LLMxCPG: Context-Aware Vulnerability Detection Through Code Property Graph-Guided LLMs" uses Joern and CPGQL to extract vulnerability-relevant slices before classification. Its two-stage design employs LLMxCPG-Q to generate CPG queries and LLMxCPG-D to classify the resulting slice, with reported code-size reductions ranging from 67.84% on PrimeVul to 90.93% on ReposVul (Lekssays et al., 22 Jul 2025). "One Bug, Hundreds Behind: LLMs for Large-Scale Bug Discovery" similarly grounds LLM reasoning in static enumeration: BugStone derives a security coding rule from one patched exemplar, enumerates candidate code pieces with lightweight static analysis, and asks the LLM to judge rule violations, rather than performing end-to-end raw detection (Wu et al., 15 Oct 2025).
A fourth pattern is multi-agent orchestration. SecVulEval uses five agents—normalization, planning, context, detection, and validation—for statement-level vulnerability localization (Ahmed et al., 26 May 2025). CVE-GENIE decomposes CVE reproduction into Processor, Builder, Exploiter, and CTF Verifier stages, with critic agents to catch shortcut behavior (Ullah et al., 1 Sep 2025). AutoPatch uses a Similarity Analyzer, Vulnerability Verifier, and Code Patcher in a verification-patching loop over a retrieval database of recent high-severity CVEs (Seo et al., 7 May 2025). In incomplete Rust CVE analysis, a four-agent architecture synthesizes KLEE-compatible wrappers around fragmentary unsafe-code snippets, then converts symbolic-execution outputs into a graph database for cross-CVE querying (Abdelrazek et al., 28 Apr 2026).
These patterns differ in surface form, but they share a common premise: direct prompting on raw CVE artifacts is usually insufficient. The field therefore combines LLMs with normalization, retrieval, static analysis, symbolic execution, graph structures, or task decomposition to stabilize reasoning.
4. Empirical findings across task families
Reported performance varies sharply by task, and direct cross-task comparison is generally invalid. Even so, several robust empirical regularities emerge.
For function-level vulnerability detection on refined CVEFixes data, NatGen achieved the best result in SCoPE with 53% accuracy and 53% F1-score, while the paper concludes that preprocessing improved data quality and reduced code size but did not significantly improve LLM performance (Gonçalves et al., 2024). SecVulEval makes the evaluation harder and more realistic: Claude-3.7-Sonnet achieved 23.83% statement-level F1 for detecting vulnerable statements with correct reasoning, while function-level F1 for the same models exceeded 50%, demonstrating a large gap between flagging suspicious code and identifying the actual vulnerable statement with causal justification (Ahmed et al., 26 May 2025).
Industrial vulnerability evaluation shows a different profile. In Siemens Healthineers deployment, CVE-LLM reports micro-F1 scores of 0.94 for VEXCategory, 0.90 for VEXJustification, and 0.96 for Vector on the test set, with domain-adaptive pretraining providing large gains over SFT-only variants (Ghosh et al., 21 Feb 2025). In automatic CVSS computation, however, vanilla LLMs are weaker: the best LLM, Gemma 3, performs strongly on AV, AC, and UI but not on subjective impact metrics, and the study reports mean accuracy of 0.738 for vanilla LLMs, 0.811 for embeddings, and 0.835 for a hybrid approach (Marchiori et al., 14 Apr 2025).
Patch-centric and repair-centric tasks often produce higher numbers because they use richer grounding. LLM-SPL, which augments VCMatch with LLM-derived CVE–commit and inter-commit signals, reports 92.74% Recall@10 and 87.33% NDCG@10 on the full dataset, with especially large gains in the 1-N setting where vulnerabilities require multiple collaborating patches (Yu et al., 2024). AutoPatch reports 90.4 percent accuracy in CVE matching, 89.5 percent F1-score for vulnerability verification, and 95.0 percent accuracy in patching over a database of 525 code snippets derived from 75 high-severity CVEs (Seo et al., 7 May 2025). LLM4CVE reports a human-verified correctness score of 8.51/10 and a 20% increase in ground-truth code similarity with Llama 3 70B in the guided+feedback setting (Fakih et al., 7 Jan 2025).
Security-offense and benchmark-generation tasks show both capability and constraint. An agentic GPT-4 system exploited 87% of a 15-vulnerability one-day benchmark when given the CVE description, but only 7% without the description (Fang et al., 2024). In CVE-Bench, the best agent framework resolved up to 13% of vulnerabilities in the zero-day setting and up to 25% in the one-day setting (Zhu et al., 21 Mar 2025). CVE-GENIE reproduced approximately 51% (428 of 841) CVEs published in 2024–2025, complete with verifiable exploits, at an average cost of $2.77 per CVE (Ullah et al., 1 Sep 2025).
Selection and review benchmarks reveal another aspect of capability. TOSSS reports security scores between 0.48 and 0.89 across 14 models on C/C++ and Java secure-snippet choice, with scores defined on a 0–1 scale where 1 means always selecting the secure implementation (Damie et al., 11 Mar 2026). SEVRA-BENCH, by contrast, evaluates refusal rather than selection and reports 97.6% overall refusal rate for Claude Opus 4.7 and 95.2% for GPT-5.5 on malicious PRs in the retained challenge split, with much lower refusal rates for weaker models (Melo et al., 11 Jun 2026).
5. Failure modes, controversies, and recurrent misconceptions
The literature repeatedly warns against equating headline benchmark scores with dependable vulnerability reasoning. The most immediate issue is data quality. SCoPE shows that duplicate detection and label inconsistencies in CVEFixes can materially distort training and evaluation, especially when semantically identical code appears with opposite labels (Gonçalves et al., 2024). SecVulEval extends this critique by arguing that function-level success is too forgiving because a model can label a function as vulnerable without locating the vulnerable statement or explaining the root cause (Ahmed et al., 26 May 2025).
A second issue is contextual incompleteness. LLM4Vuln shows that retrieved knowledge, tool-supplied context, and prompt schemes can substantially change outcomes, making it difficult to isolate intrinsic reasoning from external assistance (Sun et al., 2024). The incomplete-code problem is even more severe for Rust CVE snippets, where 31/31 files in one study lacked imports and Cargo manifests, causing all state-of-the-art formal verification tools tested there to achieve 0% compilation success before multi-agent wrapper synthesis was introduced (Abdelrazek et al., 28 Apr 2026).
A third issue is over-flagging and spurious reasoning. SecVulEval documents aggressive false-positive behavior, especially around Null Dereference, Use-After-Free, and Integer Overflow, and reports that no model identifies environment-level context in its context-identification analysis (Ahmed et al., 26 May 2025). BugStone explicitly states that its outputs are not guaranteed sound or complete and still require manual validation; among 400 sampled findings in the Linux kernel, 246 were confirmed valid, but false positives remained attributable to caller-side cleanup, flags or macros, design-specific exceptions, and limited interprocedural reasoning (Wu et al., 15 Oct 2025).
A fourth issue is evaluation mismatch. TOSSS measures secure-code preference rather than code generation, and the paper explicitly presents it as complementary rather than substitutive (Damie et al., 11 Mar 2026). SEVRA-BENCH shows that vulnerability recognition in code does not guarantee robustness under adversarial PR text, since framing strategies such as Prior Approval Signal or Fake Bug Fix can sharply reduce refusal rates for weaker reviewer agents (Melo et al., 11 Jun 2026). This undermines the common assumption that secure coding, vulnerability detection, and code-review security are interchangeable competencies.
Taken together, these studies reject a simplistic view in which an LLM “understands CVEs” merely because it performs well on one benchmark. The evidence instead indicates fragmented competence: some systems classify, some retrieve, some localize, some exploit, some repair, and each capability is highly sensitive to data curation, task formulation, and tool support.
6. Applications, operationalization, and research direction
The practical reach of CVE-LLM systems is broad. In enterprise product security, CVE-LLM supports human-in-the-loop vulnerability evaluation inside a Cybersecurity Management System, generating VEXCategory, VEXJustification, internal and customer comments, and environmental vectors, with deterministic post-processing rules and continuous retraining from expert corrections (Ghosh et al., 21 Feb 2025). In vulnerability management and triage, TRIAGE maps CVEs to ATT&CK techniques by combining methodology-guided prompting with many-shot in-context learning, distinguishing exploitation technique, primary impact, and secondary impact (Høst et al., 25 Aug 2025).
For code maintenance, LLM-SPL localizes security patches among OSS commits, especially in 1-N cases where a vulnerability is resolved through multiple related commits (Yu et al., 2024). AutoPatch and LLM4CVE target repair rather than localization, using retrieval-backed prompts, verification loops, or guided+feedback iterations to patch real-world CVE-style vulnerabilities in generated or legacy code (Seo et al., 7 May 2025). BugStone pushes in the opposite direction, scaling from one fixed bug to project-wide discovery of recurring pattern bugs (Wu et al., 15 Oct 2025).
Benchmark generation and exploit operationalization form another branch. CVE-GENIE turns CVE entries into reconstructed vulnerable environments, verified exploits, and machine-checkable CTF-style verifiers, explicitly positioning the resulting corpus as useful for fuzzer evaluation, vulnerability patching, and assessing AI security capabilities (Ullah et al., 1 Sep 2025). MalCVE associates decompiled Java JAR malware with likely CVEs through decompilation, light deobfuscation, LLM summarization, semantic retrieval in Milvus, BM25 reranking, and final LLM-based selection, reporting up to 65% Recall@10 for CVE association (Cristea et al., 17 Oct 2025).
The offensive-capability literature adds a policy dimension. One study reports that a GPT-4-based agent can autonomously exploit one-day vulnerabilities when given CVE descriptions (Fang et al., 2024), while CVE-Bench provides a broader real-world web benchmark for such behavior (Zhu et al., 21 Mar 2025). This suggests that CVE-LLM systems are simultaneously defensive infrastructure, evaluation machinery, and dual-use capability probes.
A plausible synthesis of the field is that CVE-grounding improves relevance but does not eliminate brittleness. The most successful systems rarely rely on an LLM alone; they combine the model with ontologies, retrieval, static analysis, code property graphs, symbolic execution, or multi-agent feedback loops. The resulting research program is therefore less about replacing security engineering with generative models than about embedding LLMs into CVE-centered workflows where reasoning can be constrained, audited, and cross-checked.
