RealVuln: Benchmarking Rule-Based, General-Purpose LLM, and Security-Specialized Scanners on Real-World Code

Abstract: How do security scanners perform on real-world code? We present RealVuln, the first open-source benchmark comparing Rule-Based SAST, General-Purpose LLMs, and Security-Specialized scanners on 26 intentionally vulnerable Python repositories (educational and Capture-The-Flag applications) with 796 hand-labeled entries (676 vulnerabilities, 120 false-positive traps). We test 15 scanners (3 Rule-Based SAST, 10 General-Purpose LLM, 2 Security-Specialized) and rank them by F3 score (beta=3, weighting recall 9x over precision). A clear three-tier ranking emerges under all metrics. Under F3, the Security-Specialized scanner Kolega.Dev (73.0) leads, followed by the best General-Purpose LLM, Claude Sonnet 4.6 (51.7), which in turn scores nearly 3x higher than the best Rule-Based tool, Semgrep (17.7). Under F1, Sonnet 4.6 leads (60.9) with Kolega.Dev at 52.4. Rankings within tiers shift with beta, but the three-tier hierarchy holds across all weightings. All code, ground-truth data, scanner outputs, and scoring scripts are released under an open-source license. An interactive dashboard is at https://realvuln.kolega.dev/. RealVuln is a living benchmark: versioned, community-driven, with a roadmap toward multi-language coverage.

• The paper introduces RealVuln, a benchmark evaluating rule-based, LLM-driven, and security-specialized scanners on intentionally vulnerable Python repositories.
• It reveals that security-specialized systems, like Kolega.Dev, achieve superior recall with an F3 score of 73.0 compared to GP-LLMs and rule-based tools.
• The study underscores the importance of domain-aware architectures and robust false-positive traps for accurate and cost-effective vulnerability detection.

Authoritative Summary of "RealVuln: Benchmarking Rule-Based, General-Purpose LLM, and Security-Specialized Scanners on Real-World Code" (2604.13764)

Context and Motivation

The RealVuln benchmark addresses a prominent gap in the evaluation landscape for code vulnerability scanners. Prior work either relies on synthetic datasets, lacks public ground truth, evaluates only a narrow set of scanner types, or fails to measure false positive rates rigorously. Industry reports flag persistent issues: substantial noise (false positives) and low recall in SAST outputs, leading practitioners to disregard scanner findings. RealVuln is specifically constructed to enable systematic comparison of Rule-Based SAST tools, General-Purpose LLMs, and Security-Specialized systems on a unified suite of real-world, intentionally vulnerable Python repositories with comprehensive hand-labeled vulnerabilities and false-positive traps.

Benchmark Structure and Methodology

RealVuln v1.0 comprises 26 multi-file Python repositories designed for educational and CTF purposes, aggregated to cover five major web frameworks and 18 distinct CWE families. The dataset includes 796 ground-truth findings, with 676 confirmed vulnerabilities and 120 false-positive traps—serving to quantitatively assess both recall and specificity. All entries are hand-labeled, minimizing label noise and maximizing auditability.

Scanner output matching is implemented by considering file path normalization, CWE classification (including acceptable alternatives), and proximity in line numbers, explicitly handling multi-location reports common in agentic LLM-based approaches. Ground truth, scoring scripts, scanner outputs, and all evaluation procedures are open-sourced for fully reproducible results.

The primary metric is strict $F_3$, giving recall nine times the weight of precision (reflecting security risk asymmetry), alongside secondary metrics $F_2$ and $F_1$ for broader interpretability. Micro-averaging and strict-micro modes account for scanner completion rates and robustness.

Comparative Evaluation of Scanner Categories

Fifteen scanners were evaluated across three categories:

• Rule-Based SAST: Semgrep, Snyk Code, SonarQube (pattern-matching/static analyzers)
• General-Purpose LLMs: Ten agentic runs spanning seven model families (Anthropic Claude, Google Gemini, xAI Grok, Moonshot Kimi, GLM, Minimax, Qwen)
• Security-Specialized: Kolega.Dev (proprietary, purpose-built with LLM reasoning and domain-targeted architecture) and GitHub SecLab Taskflow Agent (multi-stage agentic analysis driven by GPT-5.4).

All LLMs operated in agentic mode, exploiting iterative file exploration, tool use, and multi-stage reasoning, standardized via a shared prompt and output schema.

Key Empirical Results

A clear hierarchy is observed under all weighting schemes, with the principal findings in recall-weighted strict $F_3$ (best for production security contexts):

• Security-Specialized Scanner (Kolega.Dev): Achieves $F_3 = 73.0$, recall $= 0.809$, precision $= 0.388$; highest coverage among all scanners.
• Best General-Purpose LLM (Claude Sonnet 4.6): $F_3 = 51.7$, recall $= 0.498$, precision $= 0.785$.
• Rule-Based SAST (Semgrep): $F_2$0, recall $F_2$1, precision $F_2$2.

This ordering persists across $F_2$3 and $F_2$4, though the dominance of General-Purpose LLMs appears under metrics prioritizing precision.

Figure 1: Precision vs. recall for all 15 scanners, highlighting the superior recall of Kolega.Dev and the clusterings of GP-LLMs and Rule-Based SAST tools.

The second Security-Specialized tool, SecLab Taskflow Agent, returns high precision ($F_2$5) but extremely low recall ($F_2$6, $F_2$7), underscoring that architectural specialization is necessary but not sufficient for effective coverage; broad vulnerability detection is critical.

Figure 2: Strict F3 versus cost per 100k LOC; Kolega.Dev achieves the highest F3 at the lowest cost, with GP-LLMs trailing by both recall and cost.

Cost-Effectiveness and Practical Deployment

Cost-normalized analysis reveals that Kolega.Dev outperforms all other paid scanners: $F_2$8 at \$F_2$9F_3 = 51.7$F_1$083/100k LOC, while others are less economical and less performant. Rule-Based SAST tools are largely open-source or free but offer an order of magnitude lower recall than modern AI-powered tools.

Reliability is a critical factor: several frontier LLMs suffer from non-trivial timeout/failure rates on benchmark repositories. Coverage gaps from GP-LLMs (e.g., Opus 4.6, Minimax M2.7), and their non-deterministic outputs, reinforce the necessity of reliability-adjusted evaluation.

Discussion and Implications

The empirical results establish:

• Three-tier taxonomy: Security-Specialized scanners with domain-aware pipelines outperform GP-LLMs, which in turn outperform Rule-Based SAST.
• Recall is paramount: For security contexts where missed vulnerabilities pose severe risk, high recall (even at moderate precision) is preferable—justifying the recall-heavy strict $F_1$1 metric.
• Domain-specific architecture matters: Kolega.Dev's outperformance of GP-LLMs (even those based on frontier models) demonstrates the importance of integrating LLM reasoning into purpose-built vulnerability detection pipelines, beyond prompt engineering.
• Agentic LLMs improve over basic prompting: GP-LLMs deliver higher recall than SAST, particularly for semantic vulnerabilities and multi-file dataflows, but lack sufficient reliability and comprehensive coverage for production use.
• False positives and traps: The incorporation of false-positive traps provides a robust negative set, validating scanner specificity and preventing inflated recall via memorization.

Practical implications include recommending Security-Specialized AI scanners for compliance and high-risk production code, using recall-weighted metrics for procurement and risk analysis, and transitioning away from legacy SAST tools for vulnerability detection.

Limitations and Threats to Validity

• Language and benchmark scope: The benchmark currently covers only Python and intentionally vulnerable code (Type 1). Real-world generalizability will require extension to production codebases, memory-unsafe languages, and broader vulnerability types.
• Potential training data contamination: Some repositories may have appeared in LLM training sets, although false-positive trap analysis mitigates memorization as a confounding factor.
• Scanner selection bias: Security-Specialized representation is limited to two systems; broader inclusion will be necessary for comprehensive category analysis.
• LLM completion and stochastic variation: Run failures and nondeterminism among GP-LLMs may skew results, highlighting need for repeated and reliability-adjusted benchmarks.

Theoretical and Future Directions

The results imply that further advancements in vulnerability detection will come from:

• Deeper integration of LLM reasoning with domain-aware pipelines for detection coverage.
• Expansion of benchmarks to multi-language, real-world, and low-vulnerability-density codebases (planned for RealVuln v2).
• Agentic LLM approaches are promising but require architectural adaptation and greater reliability for production viability.
• Community-driven benchmark evolution, with versioned ground truth, transparent scoring, and reproducible dashboards, will be essential for rigorous comparative assessment as tools evolve.

Conclusion

RealVuln establishes a rigorous, open, reproducible framework for evaluating code vulnerability scanners, providing strong evidence that Security-Specialized systems with domain-aware LLM architectures substantially outperform both general-purpose LLMs and legacy static analysis tools on real-world code. The benchmark's design and transparency enable ongoing community-driven improvement in both dataset and scanner coverage. Future developments should focus on architectural innovation, broader codebase representation, and robust methodology for evaluating evolving AI-based vulnerability detection systems.