Revisiting Vul-RAG: Reproducibility and Replicability of RAG-based Vulnerability Detection with Open-Weight Models

Abstract: LLMs have shown strong potential for automated software vulnerability detection, particularly in retrieval-augmented generation (RAG) settings. However, for approaches relying on proprietary models and APIs, reproducibility and replicability remain largely unexplored, raising the question of whether reported results generalize or depend primarily on specific model choices. In this work, we present a reproducibility study of Vul-RAG, a RAG-based framework for source code vulnerability detection that enhances LLMs with high-level vulnerability knowledge. We first replicate the results in a fully local and open-weights setting using the reported open-weight baseline models. We then extend the evaluation to a diverse set of recent open-weight LLMs, including code-specialized, general-purpose, and reasoning models of varying parameter sizes. The results confirm that the findings of Vul-RAG are reproducible under local deployment, but with minor deviations. Across all evaluated models, we observe a performance plateau at approximately 0.30 pairwise accuracy (code pairs for which both the vulnerable and the patched function are correctly classified). Notably, this plateau persists even for more recent and advanced models, indicating that improvements in model capacity alone do not substantially enhance performance. Finally, we discuss practical implications and trade-offs between detection effectiveness, model capabilities, and model scale. Implementation and evaluation artifacts are publicly available at https://github.com/hs-esslingen-it-security/revisiting-Vul-RAG.

• The paper demonstrates that Vul-RAG reproduces baseline results with Qwen2.5, achieving a pairwise accuracy of 0.26 with open-weight models.
• The methodology integrates offline knowledge extraction, BM25-based retrieval, and iterative reasoning to detect vulnerabilities in Linux kernel code.
• The study reveals that model scale and specialization have limited impact, highlighting the need for optimized knowledge extraction and retrieval strategies.

Reproducibility and Model Sensitivity in Retrieval-Augmented LLM-Based Vulnerability Detection

Introduction

The paper "Revisiting Vul-RAG: Reproducibility and Replicability of RAG-based Vulnerability Detection with Open-Weight Models" (2606.04739) conducts a rigorous reproducibility and model sensitivity study of Vul-RAG, a retrieval-augmented generation (RAG) framework for source code vulnerability detection leveraging LLMs. Given the proliferation of proprietary LLMs and closed-source pipelines, the study focuses on open-weight models and local inference, eliminating reliance on external APIs. The authors systematically examine the influence of model specialization, reasoning capabilities, parameter scales, and generational advances, quantifying performance on the PairVul dataset of Linux kernel vulnerabilities. The analysis clarifies limitations and practical prospects of integrating RAG techniques into vulnerability detection pipelines.

Vul-RAG Framework Overview

Vul-RAG augments LLMs with semantic vulnerability knowledge through a multi-stage pipeline:

• Offline Knowledge Base Construction: Extraction and abstraction of semantic/causal vulnerability knowledge from CVEs and patches, represented along functional semantics, causes, and fixes.
• Online Retrieval: Functional abstraction of the input code is performed via the LLM. Relevant knowledge items are then retrieved using BM25 ranking and re-ranking, based on semantic proximity.
• Iterative Knowledge-Augmented Detection: Instead of composing a single prompt, the LLM sequentially reasons over retrieved knowledge snippets via structured prompts for vulnerability cause inspection and fixing solution matching, terminating when a decision is reached or maximum iterations are exhausted.

The study preserves the original GPT-3.5-turbo-0125-generated knowledge base for all experiments to ensure controlled comparisons.

Figure 1: Vul-RAG’s architecture integrates knowledge retrieval and iterative vulnerability reasoning for source code.

Experimental Methodology

Dataset and Metrics

The evaluation uses PairVul, comprising 2,903 Linux kernel function pairs (vulnerable & patched) with CWE labels, split into train/validation. Pairwise accuracy is the primary metric, requiring both function versions to be correctly classified, complemented with balanced recall and precision. This metric is stringent, as source-to-patch differences are minimal and security-relevant.

Studied Models

Experiments span open-weight code-specialized, general-purpose, and reasoning models—primarily Qwen, DeepSeek, and Llama families—ranging from 3B to 32B parameters. Reasoning capacity is analyzed via reinforcement learning-distilled variants. Only models processing all prompt outputs successfully were included, as compliance to strict output formatting was required.

Reproducibility: Baseline Results

Replication of Vul-RAG's reported results with Qwen2.5-Coder-32B-Instruct confirms the originally claimed performance (pairwise accuracy: 0.26), with negligible deviations in recall/precision. DeepSeek-Coder-V2-Instruct, however, suffers reduced performance (pairwise accuracy drop: 0.08), predominantly due to GPU out-of-memory failures on samples with long code sequences. Failed samples were predominantly high-token outliers, further emphasizing practical deployment concerns.

Figure 2: Token distributions and OOM failure points across the PairVul dataset, highlighting limits of scaling inference.

Model Sensitivity and Performance Plateau

Generational Advances

Evaluation of Qwen3-Coder-30B-A3B-Instruct, a newer baseline, yields only a modest uptick (+0.02) in pairwise accuracy compared to Qwen2.5. The performance plateau persists across generational upgrades, echoing the limited impact of mere model scaling.

Domain Specialization

Comparisons between code-specialized and general-purpose models (Qwen2.5-Coder-32B-Instruct vs. Qwen2.5-32B-Instruct) reveal virtually no difference in performance. This suggests that Vul-RAG’s abstraction strategies compensate for lack of code-specific pretraining, reframing detection as a semantic reasoning task.

Reasoning Enhancement

Explicit reasoning models (e.g., QwQ-32B, DeepSeek-R1-Distill-Qwen-32B) achieve the highest pairwise accuracy (0.29), but at substantial computational cost—token output volumes increase 2.6-fold, producing heavy-tailed output distributions. Gains are most pronounced in the early stages of detection, with diminishing value from later iterative reasoning.

Figure 3: Reasoning models exhibit increased output length and variance compared to baselines, correlating with deeper reasoning steps.

Figure 4: First-iteration reasoning contributes the majority of successful classifications, with marginal gains in subsequent steps.

Parameter Scale

Parameter scale is not a predictor of performance; smaller models (e.g., Qwen3-4B, DeepSeek-R1-8B) often match larger models (30B–32B) in pairwise accuracy. The absence of monotonic accuracy scaling underscores the preeminent role of architectural and training refinements, especially reasoning integration, over brute parameter count.

Detection Effectiveness, CWE Variation, and Practical Considerations

The measured pairwise accuracy (0.26–0.29) remains only marginally above the random baseline (0.25), reaffirming a ceiling observed in other studies of RAG-based vulnerability detection on disparate languages and datasets. Performance is highest for access control (CWE-264), lowest for input validation (CWE-20), and unstable for resource-related vulnerabilities. Reasoning models yield the most consistent per-CWE results, mitigating retrieval noise and knowledge base deficiencies.

Knowledge base quality, shaped by initial knowledge extraction (originating from GPT-3.5-turbo-0125), appears to restrict further gains. False negatives are attributable to vague or irrelevant retrieval; false positives often originate from fix description mismatch. Thus, future research should prioritize optimization of knowledge extraction and retrieval strategies, rather than simply scaling LLMs.

Practical deployment is further constrained by extreme GPU requirements for inference, highlighting the utility of smaller models with competitive accuracy for on-premise vulnerability scanning. However, binary output, lack of reasoning trace, and limited coverage impede direct integration with production pipelines.

Threats to Validity

Potential threats include data leakage (given recent publication of PairVul and lack of transparency in LLM training sets), restrictive output formatting bias (only instruction-following models included), and hardware limitations (exclusion of samples for DeepSeek-Coder-V2). The repeatability of results under more relaxed prompt parsing remains an open question. The gap between experimental and real-world deployment hardware accentuates the need for further efficiency studies.

Conclusion

The Vul-RAG reproducibility study confirms the replicability of results for Qwen2.5-Coder-32B but not for DeepSeek-Coder-V2, reflecting practical scaling limitations. Across a spectrum of open-weight models, reasoning-enhanced variants offer the highest accuracy but at substantial token cost, and neither increased parameter scale nor generational improvement breaks the performance plateau. These findings underscore a fundamental limitation in pairwise source-vulnerability discrimination using RAG-augmented LLMs and prioritize knowledge base quality and retrieval optimization as next research steps. Deployment-wise, small models suffice for parity with larger variants, promoting resource-efficient vulnerability detection pipelines. Future work should systematically decouple the role of base model capability from knowledge extraction and reasoning integration to address the observed accuracy ceiling.