LiveCVEBench: Continuous CVE Benchmark

• LiveCVEBench is a continuously updated benchmark that automatically reproduces, exploits, and patches real-world software vulnerabilities using multi-stage automation.
• It utilizes a decoupled, six-stage pipeline known as CVE-Factory to convert raw CVE data into executable security tasks validated by rigorous testing.
• The benchmark covers 14 programming languages and 153 repositories, demonstrating high accuracy and scalability through dynamic, test-driven verification.

LiveCVEBench is a continuously updated, multi-language benchmark for evaluating and advancing the capability of AI agents to automatically reproduce, exploit, and patch real-world software vulnerabilities. Its core architecture and data pipeline leverage CVE-Factory, a fully automated, multi-agent reproduction framework, which enables the daily ingestion, transformation, and rigorous verification of high-severity Common Vulnerabilities and Exposures (CVEs) as executable agentic security tasks. LiveCVEBench directly addresses the need for scalable, expert-level, and temporally current benchmarks in code security, with a scope encompassing diverse programming languages, vulnerability classes, and real-world repositories, including the rising class of AI-tooling vulnerabilities (Luo et al., 3 Feb 2026).

1. Automated CVE Task Generation Pipeline

The foundation of LiveCVEBench is a six-stage, decouple–couple automation pipeline operationalized in CVE-Factory. This system transforms raw, often sparse CVE entries into robust, executable tasks in the Terminal-Bench format. Each stage is executed by a specialized agent under orchestration, with explicit separation of context, file ownership, and feedback to maximize both autonomy and reliability.

1. Information Collection (Analyzer): Parses raw CVE JSON, strips organizational fields, enriches with reference URLs, PoCs, patch commits, and generates role-specific Markdown files. If the CVE lacks reproducible, open-source code, the pipeline aborts at this stage.
2. File Generation (Generator): Creates task.yaml (first-person, bug-report style), a reference exploit (solution.sh), functional and vulnerability pytest suites (test_func.py, test_vuln.py), and a requirements manifest.
3. Environment Construction (Builder): Independently generates a Dockerfile and, if warranted, a docker-compose.yaml for multi-container tasks, validating that services build and start in isolation.
4. Vulnerability Verification (Validator): Executes the functional and vulnerability test suites pre-patch. The orchestrator defines task readiness as

$$(\text{run test\_vuln.py} \rightarrow \text{fail}) \land (\text{run test\_func.py} \rightarrow \text{pass}).$$

Feedback and fixes iterate up to three times.

1. Solution Verification (Solver): Patches via solution.sh and reruns all tests; the patched state is validated by

$$(\text{run test\_vuln.py} \rightarrow \text{pass}) \land (\text{run test\_func.py} \rightarrow \text{pass}).$$

2. Holistic Validation (Checker): Ensures the environment is free of mock constructs and extractable data-leakage; successful tasks are marked reproduced and added to the benchmark set.

Central orchestration routes precise signals (continue, pause, error) and tracks file-level provenance, ensuring targeted repairs and context minimization. Stage runtimes are balanced, with per-agent median wall-clock between 6–15 minutes per CVE, confirming highly parallel, bottleneck-resistant throughput.

2. Benchmark Composition and Coverage

After an initial three-phase filtering regime—consisting of heuristic reproducibility scoring (as defined in Table 5), monthly sampling to enforce MITRE CWE diversity with per-repo and per-CWE caps, and LLM-as-Judge semantic deduplication—CVE-Factory attempts reproduction on a curated set of recent CVEs.

LiveCVEBench v1.0 features the following properties, based on May–December 2025 data:

Attribute	Value (v1.0)
Number of tasks	190
Programming languages	14 (PHP, JavaScript, Python, Shell, C, TypeScript, Go, Java, C++, Ruby, Rust, C#, Lua, Erlang)
Source repositories	153 (across GitHub, npm, WordPress, etc.)
CWE coverage	74 categories (injection, memory safety, path traversal, access control, AI-tooling, etc.)
Environment topology	137 single-container, 46 two-container, 7 three-plus containers
AI-tooling CVEs	10% (e.g., vulnerabilities in frameworks such as LangChain or PyTorch)

Tasks encode system-level, end-to-end exploits, validated via rigorous test harnesses aligned with real-world pentesting workflows. The inclusion of first-person, bug-report–style specifications and holistic verification replicates the complexity faced by professional red-teamers (Luo et al., 3 Feb 2026).

3. Verification Metrics and Reproduction Quality

LiveCVEBench adopts cross-validation against the PatchEval expert dataset using strict, test-driven metrics:

• Solution Correctness: Fraction of solutions ($C_s$) from CVE-Factory that pass expert PatchEval environment/tests ($P_e, P_t$):

$$\text{SolutionCorrectness} = \frac{\#\{\,(P_e, P_t, C_s)\text{ that pass all tests}\}}{N} \times 100\%.$$

Empirically, $N=215$, with $9$ failures, yielding $95.35\%$ correctness.

• Environment Fidelity: Fraction of environments ($C_e$) and tests ($C_t$) from CVE-Factory that validate expert solutions ($$(\text{run test\_vuln.py} \rightarrow \text{pass}) \land (\text{run test\_func.py} \rightarrow \text{pass}).$$0):

$$(\text{run test\_vuln.py} \rightarrow \text{pass}) \land (\text{run test\_func.py} \rightarrow \text{pass}).$$1

• Test Quality: LLM-based pairwise comparative grading finds $$(\text{run test\_vuln.py} \rightarrow \text{pass}) \land (\text{run test\_func.py} \rightarrow \text{pass}).$$2 of CVE-Factory test suites as "equal or better" than PatchEval analogs.
• Verified Success Rate: For curated benchmark tasks, $$(\text{run test\_vuln.py} \rightarrow \text{pass}) \land (\text{run test\_func.py} \rightarrow \text{pass}).$$3 of LiveCVEBench's 190 tasks pass all dynamic verification; the observed upstream hit rate for raw CVEs is $$(\text{run test\_vuln.py} \rightarrow \text{pass}) \land (\text{run test\_func.py} \rightarrow \text{pass}).$$4, due to manual culling of failures (dominant causes include infeasible test logic and mock PoCs).

This test-driven approach, centered on dynamic execution rather than static or pattern-matching heuristics, underpins the observed expert-level fidelity (Luo et al., 3 Feb 2026).

4. Continuous Update and Temporal Adaptation

LiveCVEBench maintains real-time relevance by tracking temporal shifts in vulnerability distributions, with support for emergent exploit classes such as AI-specific threats.

The update process is as follows:

a) Automated daily/weekly ingestion of new entries from CVEListV5. b) Reproducibility scoring and filtering, as per Section A.1. c) Diversity enforcement via a composite sampler score

$$(\text{run test\_vuln.py} \rightarrow \text{pass}) \land (\text{run test\_func.py} \rightarrow \text{pass}).$$5

d) Parallel, auto-reproduction in Kubernetes-based agent clusters. e) Lightweight manual spot-checks for authenticity. f) Public dataset and leaderboard update within days.

The system is designed to surface novel CVE patterns rapidly, as demonstrated with prototype-pollution attacks in major AI libraries (e.g., LangChain). This continuous ingestion and curation mechanism ensures the benchmark remains current with emerging software and threat paradigms (Luo et al., 3 Feb 2026).

5. Model Evaluation and Training Impact

LiveCVEBench enables comparative analysis of agent frameworks and LLMs on realistic, temporally representative vulnerability exploitation tasks.

Recent results (Table 4, (Luo et al., 3 Feb 2026)):

Model/Framework	Pass Rate
Terminus-2 + Claude Opus 4.5	42.33 %
Terminus-2 + Claude Sonnet 4.5	38.10 %
Claude Code + Claude Opus 4.5	27.78 %
MiniMax-M2	30.16 %
Qwen3-Coder-480B	19.58 %

Fine-tuning studies using >1,000 reproduction-derived training tasks demonstrate substantial performance lift:

• Qwen3-32B (zero-shot): 5.29%
• • SETA (4k trajectories): 21.69%
• • CVE-Factory (3k): 31.05%
• • CVE-Factory (4k): 35.79%

Thus, fine-tuning on CVE-Factory output yields a 6.8× improvement on LiveCVEBench and parallel generalization gains on PatchEval and Terminal-Bench.

Trajectory analysis shows fine-tuned agents systematically explore more solution steps (57.5 on average), with improved patch validation. Agent framework ("scaffold") differences produce >14 percentage point swings—structured JSON-RPC prompts (analysis → plan → command → reflection) consistently outperform monolithic free-text scaffolds (Luo et al., 3 Feb 2026).

6. Lessons Learned and Best Practices

• Staged decouple–couple designs reduce cognitive and computational load per agent, minimizing runtime and avoiding hallucinatory behavior seen in monolithic, context-heavy approaches.
• Context isolation with targeted feedback (file-level repair and explicit orchestration signals) prevents context bloat and facilitates efficient debugging and repair.
• Pre-reproduction filtering based on reproducibility and diversity drastically improves downstream hit rate, conserving compute resources.
• Continuous refresh and automation are essential to avoid benchmark staleness, particularly as AI-specific vulnerabilities rise in prevalence.
• Agent scaffold quality is as decisive as the underlying model in determining success rates; disciplined, structured prompting is critical.
• Dynamic, test-driven verification is mandatory. Static pattern-matching or mock implementations are a dominant source of reproduction failure.
• Breadth and diversity constraints ensure the benchmark remains representative and avoids overconcentration in a single class (e.g., XSS in WordPress plugins), as evidenced by enforced per-repo and per-CWE caps (74 CWE categories in v1.0).

This suggests that LiveCVEBench, through automation, diversity enforcement, dynamic verification, and continuous update, offers a robust platform for tracking and improving the real-world exploit capabilities of code-focused AI agents, while maintaining alignment with expert pentesting standards (Luo et al., 3 Feb 2026).

7. Comparison to Prior Work and Community Integration

LiveCVEBench supersedes earlier static or manually curated benchmarks, such as CVE-Bench (Zhu et al., 21 Mar 2025), by supporting broader language and repository coverage, timely updates, and higher task fidelity. CVE-Bench made foundational advances in isolating real, open-source web app vulnerabilities in containerized sandboxes with standardized attack taxonomies and grader-based exploit verification, but its scale and temporal responsiveness were limited to periodic manual curation (e.g., 40 critical CVEs, single-month snapshot).

LiveCVEBench automates ingestion (daily/weekly), scales task construction across 14 languages and 153 repositories, and explicitly adapts to contemporary trends such as AI-tooling exploits. Community-facing features, including a public leaderboard, open dataset, and challenge API, foster benchmarking, training, and red-teaming by external researchers and model developers (Luo et al., 3 Feb 2026, Zhu et al., 21 Mar 2025).

The platform provides an empirically grounded, reproducible, and extensible yardstick for measuring both the security risk posed by LLM-driven agents and the evolving sophistication of contemporary code security models.