Snyk VulnBench JS 1.0: Can LLMs Find the Same Bugs Twice?

Abstract: We ran 300 repeated vulnerability-finding scans to measure how repeatable agentic LLM security review is on the same JavaScript code, prompt, and benchmark harness. The headline result is that LLM security findings were unevenly repeatable: reference-matched findings were stable, but extra model reports varied heavily from run to run. Across 250 model runs, 80 of 161 unique unmatched findings appeared in only one of five identical repetitions, while only 22 appeared in all five. By contrast, when Claude matched a Snyk Code reference finding, the behavior was much more stable: 134 of 158 unique reference-matched findings appeared in all five repetitions. The benchmark also shows complementarity. Models consistently found familiar, high-signal exploit shapes, and in one case surfaced a likely Snyk Code product gap. Snyk Code static application security testing (SAST) was deterministic and better at systematically enumerating repeated data-flow sinks. The results support combining agentic LLM review with deterministic SAST rather than treating either technique as a replacement for the other.

• The paper presents a benchmark that systematically evaluates repeatability of agentic LLM vulnerability reviews versus deterministic SAST in JavaScript codebases.
• The study finds that LLMs reliably rediscover known vulnerabilities but generate highly unstable unmatched findings, impacting operational reliability.
• The analysis highlights a cost-quality tradeoff, where higher-cost LLM configurations may not yield commensurate improvements in vulnerability detection.

Snyk VulnBench JS 1.0: A Systematic Study of Agentic LLM Vulnerability Review Repeatability

Introduction and Motivation

The increasing integration of agentic LLMs into the SDLC, especially for security reviews, has raised critical questions regarding their reliability and consistency. Traditional SAST tools are characterized by strict determinism—unmodified code and rules yield invariant results. In contrast, LLMs provide context-aware, linguistic, and sometimes inventive reasoning about code, but their stochasticity can introduce operational unreliability. "Snyk VulnBench JS 1.0: Can LLMs Find the Same Bugs Twice?" (2606.15762) presents a meticulous benchmark designed to evaluate the repeatability of agentic LLM security review on JavaScript codebases and to contrast this behavior with deterministic SAST.

Experimental Design

The benchmark suite consists of 10 Express/JavaScript fixtures with 44 Snyk Code–derived reference findings. Six configurations, including Snyk Code and five variants across the Claude Opus and Sonnet lines, were evaluated. Each configuration performed five repeated reviews per task, amounting to 300 total runs under tightly controlled conditions. Agreement with the SAST reference was evaluated primarily using a lenient Snyk-reference F1 score, defined as the harmonic mean of precision and recall against the reference findings, but conferred as a measure of repeatability and operational agreement rather than true-grounded detection accuracy.

Repeatability and Agreement Analysis

A principal outcome is the stark variance in LLM outputs on repeated runs. Snyk Code SAST, by construction, is perfectly stable. Among LLMs, Claude Opus 4.6 Medium demonstrates the best repeatability (F1 = 75.4%, σ = 0.2pp), with higher-variance configurations (e.g., Claude Sonnet 4.6 High, σ = 3.5pp) producing less reliable queues.

Figure 1: Snyk-reference F1 versus standard deviation, highlighting the determinism of Snyk Code and the varying stability across LLM configurations.

Stability diverges significantly between findings that match the reference set and additional (unmatched) model-generated reports. Reference-matched findings are consistently rediscovered by the same model across runs—84.8% of such findings recur in all five repetitions, indicating robust identification of “known” vulnerabilities.

Figure 2: Proportion of Snyk Code reference findings consistently matched across repeated runs for each model configuration.

In stark contrast, unmatched reports generated solely by the models are highly unstable. Across all configurations, only 13.7% of unmatched finding signatures appeared in all five runs; nearly half (49.7%) are one-off discoveries, raising concerns about the operational reliability of model-only triage.

Figure 3: Fraction of unmatched finding signatures that occurred only once across five repeated runs by model configuration.

Figure 4: Fraction of unmatched finding signatures present in every run, showing weak stability for most LLMs' "extra" reports.

This pronounced unreliability in non-reference reports could translate to inconsistent developer experiences, elevated triage overhead, and increased risk of both noise and missed detections depending on the stochastic outcome of any given run.

Coverage, Complementarity, and Failure Modes

Routine SAST and agentic LLM review exhibit complementary strengths and weaknesses. LLMs are effective on certain high-signal vulnerabilities (e.g., command injection, SSRF, hardcoded credentials, ReDoS), while their recall diminishes sharply for systematic SAST-oriented classes such as resource limit checking, path traversal enumeration, and framework-specific information disclosure.

Figure 5: Mean recall by vulnerability type and configuration, illustrating where LLMs align with or fall short of SAST.

A qualitative review reveals that LLMs sometimes misclassify non-exploitable, vulnerability-shaped code as true positives (e.g., mock SQL helpers that cannot be attacked), yet also reveal SAST product gaps—instances where potential vulnerabilities outside the reference set are surfaced by the model.

Moreover, SAST systematically identifies every repeated vulnerable sink while LLMs are prone to spotting a representative instance and failing to comprehensively enumerate duplicative flows. In larger, app-like fixtures, the top LLM configuration achieved only 40.0% F1, missing critical patterns detected by SAST.

Figure 6: Benchmark scores for the largest multi-file fixture, indicating persistent recall gaps in LLMs relative to SAST.

Cost-Quality Tradeoffs

A salient outcome is the decoupling between inference cost and security review efficacy. Claude Opus 4.7 Max, the highest-cost configuration (5.7x the cost, 1.9x the tokens as Opus 4.6 Medium), underperformed both in F1 (68.8% vs 75.4%) and precision. This is critical for deployment at scale where the marginal utility per dollar may determine feasibility.

Figure 7: Scatterplot of Snyk-reference F1 against model session cost, illustrating diminishing returns for expensive configurations.

Practical and Theoretical Implications

The findings underscore several operational and research-relevant themes:

• Agentic LLM review is not sufficiently stable to replace deterministic SAST for production gating: High variance in unmatched findings can compromise triage reliability.
• Combining LLM review with SAST offers tangible benefit: LLMs surface high-signal vulnerabilities missed by SAST and product-gap cases, while SAST excels in comprehensive, systematic detection.
• Current benchmarks reliant on internal SAST reference findings are valuable for repeatability studies, but broader accuracy claims require independent, exhaustively adjudicated ground truth: Implications for future benchmarking efforts suggest integration with publicly curated datasets (e.g., BaxBench (Vero et al., 17 Feb 2025)) and expanded fixture complexity.
• Inference cost must be critically balanced with detection gain: Marginal improvements in recall or precision do not always justify higher operational expenses.

Conclusion

Snyk VulnBench JS 1.0 (2606.15762) provides a rigorous, reproducible evaluation of the repeatability and operational reliability of agentic LLMs for vulnerability discovery in JavaScript projects. The study demonstrates clear limitations in both the determinism and coverage of LLM-only workflows; while reference findings are generally stable, the volatility of additional reports introduces significant uncertainty. Deterministic SAST retains critical value for exhaustive, predictable reviews, but hybrid workflows leveraging the strengths of both paradigms are substantiated. Advancing agentic review requires expanded benchmarks, incorporation of independent ground truths, and methodical assessment of combined agentic and static analyses.