---
title: 'LiveCVEBench: Continuous CVE Benchmark'
url: https://www.emergentmind.com/topics/livecvebench
type: topic
---

# LiveCVEBench: Continuous CVE Benchmark

LiveCVEBench is a continuously updated, multi-language benchmark for evaluating and advancing the capability of AI agents to automatically reproduce, exploit, and patch real-world software vulnerabilities. Its core architecture and data pipeline leverage CVE-Factory, a fully automated, multi-agent reproduction framework, which enables the daily ingestion, transformation, and rigorous verification of high-severity Common Vulnerabilities and Exposures (CVEs) as executable agentic security tasks. LiveCVEBench directly addresses the need for scalable, expert-level, and temporally current benchmarks in code security, with a scope encompassing diverse programming languages, vulnerability classes, and real-world repositories, including the rising class of AI-tooling vulnerabilities [2602.03012].

## 1. Automated CVE Task Generation Pipeline

The foundation of LiveCVEBench is a six-stage, decouple–couple automation pipeline operationalized in CVE-Factory. This system transforms raw, often sparse CVE entries into robust, executable tasks in the Terminal-Bench format. Each stage is executed by a specialized agent under orchestration, with explicit separation of context, file ownership, and feedback to maximize both autonomy and reliability.

1. **Information Collection (Analyzer):** Parses raw CVE JSON, strips organizational fields, enriches with reference URLs, PoCs, patch commits, and generates role-specific Markdown files. If the CVE lacks reproducible, open-source code, the pipeline aborts at this stage.
2. **File Generation (Generator):** Creates `task.yaml` (first-person, bug-report style), a reference exploit (`solution.sh`), functional and vulnerability pytest suites (`test_func.py`, `test_vuln.py`), and a requirements manifest.
3. **Environment Construction (Builder):** Independently generates a `Dockerfile` and, if warranted, a `docker-compose.yaml` for multi-container tasks, validating that services build and start in isolation.
4. **Vulnerability Verification (Validator):** Executes the functional and vulnerability test suites pre-patch. The orchestrator defines task readiness as 
   $$
   (\text{run test\_vuln.py} \rightarrow \text{fail}) \land (\text{run test\_func.py} \rightarrow \text{pass}).
   $$
   Feedback and fixes iterate up to three times.
5. **Solution Verification (Solver):** Patches via `solution.sh` and reruns all tests; the patched state is validated by 
   $$
   (\text{run test\_vuln.py} \rightarrow \text{pass}) \land (\text{run test\_func.py} \rightarrow \text{pass}).
   $$
6. **Holistic Validation (Checker):** Ensures the environment is free of mock constructs and extractable data-leakage; successful tasks are marked reproduced and added to the benchmark set.

Central orchestration routes precise signals (`continue`, `pause`, `error`) and tracks file-level provenance, ensuring targeted repairs and context minimization. Stage runtimes are balanced, with per-agent median wall-clock between 6–15 minutes per CVE, confirming highly parallel, bottleneck-resistant throughput.

## 2. Benchmark Composition and Coverage

After an initial three-phase filtering regime—consisting of heuristic reproducibility scoring (as defined in Table 5), monthly sampling to enforce MITRE CWE diversity with per-repo and per-CWE caps, and LLM-as-Judge semantic deduplication—CVE-Factory attempts reproduction on a curated set of recent CVEs.

**LiveCVEBench v1.0** features the following properties, based on May–December 2025 data:

| Attribute                | Value (v1.0)                                                                          |
|--------------------------|---------------------------------------------------------------------------------------|
| Number of tasks          | 190                                                                                   |
| Programming languages    | 14 (PHP, JavaScript, Python, Shell, C, TypeScript, Go, Java, C++, Ruby, Rust, C#, Lua, Erlang) |
| Source repositories      | 153 (across GitHub, npm, WordPress, etc.)                                             |
| CWE coverage             | 74 categories (injection, memory safety, path traversal, access control, AI-tooling, etc.) |
| Environment topology     | 137 single-container, 46 two-container, 7 three-plus containers                        |
| AI-tooling CVEs          | 10% (e.g., vulnerabilities in frameworks such as LangChain or PyTorch)                |

Tasks encode system-level, end-to-end exploits, validated via rigorous test harnesses aligned with real-world pentesting workflows. The inclusion of first-person, bug-report–style specifications and holistic verification replicates the complexity faced by professional red-teamers [2602.03012].

## 3. Verification Metrics and Reproduction Quality

LiveCVEBench adopts cross-validation against the PatchEval expert dataset using strict, test-driven metrics:

- **Solution Correctness:** Fraction of solutions ($C_s$) from CVE-Factory that pass expert PatchEval environment/tests ($P_e, P_t$):
  $$
  \text{SolutionCorrectness} = \frac{\#\{\,(P_e, P_t, C_s)\text{ that pass all tests}\}}{N} \times 100\%.
  $$
  Empirically, $N=215$, with $9$ failures, yielding $95.35\%$ correctness.
- **Environment Fidelity:** Fraction of environments ($C_e$) and tests ($C_t$) from CVE-Factory that validate expert solutions ($P_s$):
  $$
  \text{EnvironmentFidelity} = \frac{\#\{\,(C_e, C_t, P_s)\text{ that pass all tests}\}}{N} \times 100\% = 96.13\%.
  $$
- **Test Quality:** LLM-based pairwise comparative grading finds $73.91\%$ of CVE-Factory test suites as "equal or better" than PatchEval analogs.
- **Verified Success Rate:** For curated benchmark tasks, $100\%$ of LiveCVEBench's 190 tasks pass all dynamic verification; the observed upstream hit rate for raw CVEs is $66.2\%$, due to manual culling of failures (dominant causes include infeasible test logic and mock PoCs).

This test-driven approach, centered on dynamic execution rather than static or pattern-matching heuristics, underpins the observed expert-level fidelity [2602.03012].

## 4. Continuous Update and Temporal Adaptation

LiveCVEBench maintains real-time relevance by tracking temporal shifts in vulnerability distributions, with support for emergent exploit classes such as AI-specific threats.

The update process is as follows:

a) Automated daily/weekly ingestion of new entries from CVEListV5.
b) Reproducibility scoring and filtering, as per Section A.1.
c) Diversity enforcement via a composite sampler score
   $$
   S_\mathrm{final} = S_\mathrm{base} + \frac{\mathrm{CWE}_\mathrm{danger}}{57} \times 30 + 2 \times \mathrm{CVSS} + S_\mathrm{div} + S_\mathrm{nov}.
   $$
d) Parallel, auto-reproduction in Kubernetes-based agent clusters.
e) Lightweight manual spot-checks for authenticity.
f) Public dataset and leaderboard update within days.

The system is designed to surface novel CVE patterns rapidly, as demonstrated with prototype-pollution attacks in major AI libraries (e.g., LangChain). This continuous ingestion and curation mechanism ensures the benchmark remains current with emerging software and threat paradigms [2602.03012].

## 5. Model Evaluation and Training Impact

LiveCVEBench enables comparative analysis of agent frameworks and large language models (LLMs) on realistic, temporally representative vulnerability exploitation tasks.

Recent results (Table 4, [2602.03012]):

| Model/Framework                   | Pass Rate        |
|-----------------------------------|------------------|
| Terminus-2 + Claude Opus 4.5      | 42.33 %          |
| Terminus-2 + Claude Sonnet 4.5    | 38.10 %          |
| Claude Code + Claude Opus 4.5     | 27.78 %          |
| MiniMax-M2                        | 30.16 %          |
| Qwen3-Coder-480B                  | 19.58 %          |

Fine-tuning studies using >1,000 reproduction-derived training tasks demonstrate substantial performance lift:

- Qwen3-32B (zero-shot): 5.29%
- + SETA (4k trajectories): 21.69%
- + CVE-Factory (3k): 31.05%
- + CVE-Factory (4k): 35.79%

Thus, fine-tuning on CVE-Factory output yields a 6.8× improvement on LiveCVEBench and parallel generalization gains on PatchEval and Terminal-Bench.

Trajectory analysis shows fine-tuned agents systematically explore more solution steps (57.5 on average), with improved patch validation. Agent framework ("scaffold") differences produce >14 percentage point swings—structured JSON-RPC prompts (analysis → plan → command → reflection) consistently outperform monolithic free-text scaffolds [2602.03012].

## 6. Lessons Learned and Best Practices

- **Staged decouple–couple designs** reduce cognitive and computational load per agent, minimizing runtime and avoiding hallucinatory behavior seen in monolithic, context-heavy approaches.
- **Context isolation with targeted feedback** (file-level repair and explicit orchestration signals) prevents context bloat and facilitates efficient debugging and repair.
- **Pre-reproduction filtering** based on reproducibility and diversity drastically improves downstream hit rate, conserving compute resources.
- **Continuous refresh and automation** are essential to avoid benchmark staleness, particularly as AI-specific vulnerabilities rise in prevalence.
- **Agent scaffold quality** is as decisive as the underlying model in determining success rates; disciplined, structured prompting is critical.
- **Dynamic, test-driven verification** is mandatory. Static pattern-matching or mock implementations are a dominant source of reproduction failure.
- **Breadth and diversity constraints** ensure the benchmark remains representative and avoids overconcentration in a single class (e.g., XSS in WordPress plugins), as evidenced by enforced per-repo and per-CWE caps (74 CWE categories in v1.0).

*This suggests that LiveCVEBench, through automation, diversity enforcement, dynamic verification, and continuous update, offers a robust platform for tracking and improving the real-world exploit capabilities of code-focused AI agents, while maintaining alignment with expert pentesting standards* [2602.03012].

## 7. Comparison to Prior Work and Community Integration

LiveCVEBench supersedes earlier static or manually curated benchmarks, such as CVE-Bench [2503.17332], by supporting broader language and repository coverage, timely updates, and higher task fidelity. CVE-Bench made foundational advances in isolating real, open-source web app vulnerabilities in containerized sandboxes with standardized attack taxonomies and grader-based exploit verification, but its scale and temporal responsiveness were limited to periodic manual curation (e.g., 40 critical CVEs, single-month snapshot).

LiveCVEBench automates ingestion (daily/weekly), scales task construction across 14 languages and 153 repositories, and explicitly adapts to contemporary trends such as AI-tooling exploits. Community-facing features, including a public leaderboard, open dataset, and challenge API, foster benchmarking, training, and red-teaming by external researchers and model developers [2602.03012][2503.17332].

The platform provides an empirically grounded, reproducible, and extensible yardstick for measuring both the security risk posed by LLM-driven agents and the evolving sophistication of contemporary code security models.

Source: https://www.emergentmind.com/topics/livecvebench