VulnRepairEval: Exploit-Based Repair Evaluation

• The paper presents VulnRepairEval, a framework that counts repair success only when the original PoC exploit fails after patching.
• It employs a containerized, differential execution pipeline to assess repairs on authentic Python CVEs, ensuring security remediation over superficial fixes.
• Empirical evaluation reveals low end-to-end repair success for current LLMs, highlighting significant challenges in automated vulnerability patching.

VulnRepairEval is an exploit-based evaluation framework for assessing LLM vulnerability repair capabilities. It is designed around a stringent security criterion: a repair is counted as successful only if the original functional Proof-of-Concept (PoC) exploit succeeds on the vulnerable baseline and fails after patching. The framework couples this criterion with a containerized, differential execution pipeline and a curated benchmark of authentic Python CVEs, with the stated goal of measuring end-to-end security remediation rather than superficial patch plausibility or test-suite survival alone (Wang et al., 3 Sep 2025).

1. Concept and evaluation philosophy

VulnRepairEval is explicitly positioned against validation regimes that treat compilation, patch application, or ordinary regression passing as sufficient evidence of repair. Its central claim is that such criteria can overestimate security effectiveness, because a code change may remain exploitable even when it appears syntactically valid or functionally non-disruptive. Accordingly, the framework defines repair success through exploit failure after patching, not merely through program executability or textual similarity to a reference patch (Wang et al., 3 Sep 2025).

This design places VulnRepairEval within a broader shift in automated vulnerability repair toward stronger validation oracles. The Systematization of Knowledge on automated vulnerability repair argues that trustworthy security patch validation should require that a patch fix the vulnerability, avoid introducing new bugs or vulnerabilities, and preserve original functionality (Li et al., 31 Jan 2025). VulnRepairEval narrows that general principle to an operational exploit-grounded condition. Its distinctiveness lies in elevating the original attack artifact—the functional PoC—to the primary evaluation oracle.

The framework also distinguishes itself from adjacent benchmark families. General automated program repair datasets such as Defects4J, BugsInPy, and SWE-Bench are characterized as test-driven rather than exploit-driven. Security-oriented datasets such as CVEfixes, ReposVul, MegaVul, and MoreFixes are described as patch corpora that do not enforce exploit-based validation. Even more security-focused benchmarks such as Vul4J and VUL4C are presented as weaker in this specific sense, because PoV or exploit-like triggers are not treated as equivalent to a functional PoC exploit under the VulnRepairEval criterion (Wang et al., 3 Sep 2025).

2. Benchmark construction and corpus curation

The benchmark was built through a multi-stage curation pipeline intended to trade scale for evaluability. The starting corpus contained 449 CVEs from 192 open-source GitHub projects and 862 security-fixing commits. After canonicalizing commit references by keeping the longest available hash and deduplicating, the corpus became 448 CVEs across 164 projects with 684 unique commits. The framework then narrowed its scope to Python projects, because Python environments were described as comparatively easier to reconstruct and containerize reproducibly (Wang et al., 3 Sep 2025).

Candidate exploit material was mined primarily from NVD reference links. The pipeline crawled 2,493 candidate URLs, rendered each page with a headless browser, converted the resulting HTML to Markdown, and used DeepSeek-V3 to classify pages as Executable, Descriptive, or Brief. The model flagged over 120 web links corresponding to 109 unique CVEs as containing executable PoCs, after which the authors manually or operationally reconstructed and executed the candidate exploits against vulnerable targets. The paper reports only one false positive among the LLM-selected candidates (Wang et al., 3 Sep 2025).

The final benchmark contains 23 distinct Python CVEs from 2017 to 2024, spanning 9 vulnerability types, all with working PoC exploits. Inclusion required a real CVE, a verifiable vulnerable commit, a working executable PoC, and suitability for automated, containerized evaluation. The benchmark intentionally gives models only the specific vulnerable files, not entire repositories, in order to reduce context burden while retaining realistic exploit validation (Wang et al., 3 Sep 2025).

Aspect	Value
Initial corpus	449 CVEs, 192 projects, 862 commits
Post-dedup corpus	448 CVEs, 164 projects, 684 unique commits
Candidate exploit URLs	2,493
LLM-flagged executable PoC links	Over 120 links, 109 unique CVEs
Final benchmark	23 Python CVEs, 2017–2024, 9 vulnerability types

This construction methodology reflects a deliberate emphasis on authentic, automatically testable vulnerability instances rather than maximal benchmark size. A plausible implication is that VulnRepairEval prioritizes high-confidence exploit reproducibility over broad language or CWE coverage.

3. Containerized differential execution pipeline and metrics

VulnRepairEval uses a controlled, differential evaluation setup. For each case, the vulnerable baseline is reconstructed by identifying the official security-fixing commit and checking out its parent commit. Dependency reconstruction is handled by statically inspecting project packaging files in priority order: pyproject.toml, setup.py, and requirements.txt. The framework then generates a custom Docker environment for the project and executes the same PoC in two parallel containers: one containing the vulnerable code and one containing the patched code (Wang et al., 3 Sep 2025).

The runtime module standardizes execution by launching the PoC as a subprocess, capturing stdin, stdout, and stderr, installing dependencies automatically, recording execution time and error conditions, and formatting runtime output for later analysis. This differential design treats the patch as the only intended variable between the two executions. A repair is counted only if the patch can be applied and the original PoC succeeds on the vulnerable baseline but fails on the patched version (Wang et al., 3 Sep 2025).

The paper defines the primary success quantity as PoC-validated repair success, denoted $P_{\text{succ}}$. It also reports two secondary diagnostics: $P_{\text{corr}}$, measuring format correctness or patch applicability, and $V_{\text{dnf}}$, the “did not find” rate for cases in which the model abstains or declares the code safe. For compact leaderboard ranking, the framework introduces a composite score:

$$S_p \;=\; \frac{(1+\beta^2)\,\big(P_{\text{amend}}\cdot P_{\text{succ}}\big)}{\beta^2 \cdot P_{\text{amend}} + P_{\text{succ}}}\cdot \left(1 - 0.5 \cdot V_{\text{dnf}}\right),$$

with

$$P_{\text{amend}} = \ln(P_{\text{corr}} + 1), \qquad \beta = 2.$$

The score is described as $F_\beta$-style, with $\beta=2$, thereby placing greater emphasis on end-to-end repair success than on format correctness alone (Wang et al., 3 Sep 2025).

This metric design encodes a specific methodological view: applicability matters, but applicability is subordinated to exploit blocking, and excessive abstention is penalized. In effect, VulnRepairEval treats repair evaluation as a security-sensitive decision problem rather than a pure patch-generation task.

4. Empirical evaluation of LLMs

The framework evaluates 12 models under a unified prompt and runtime setup. The proprietary models are Gemini 2.5 Pro, Gemini 2.5 Flash, Gemini 2.0 Flash, GPT-4o, GPT-o4 mini, and GPT-3.5 Turbo. The open-source models are DeepSeek V3 671B, DeepSeek R1 671B, Qwen3 8B, Qwen3 235B, Qwen3 8B Thinking, and Qwen3 235B Thinking. SWE-Agent is also used in some experiments as an agentic wrapper rather than as a standalone model (Wang et al., 3 Sep 2025).

The headline result is low absolute performance. The best-performing model, Gemini 2.5 Pro, achieves $P_{\text{succ}} = 0.217$, corresponding to 5/23 CVEs, with $P_{\text{corr}} = 0.304$, $V_{\text{dnf}} = 0.000$, and $P_{\text{corr}}$0. The paper states that the overall average end-to-end success is about 7%, and that several models have zero success rate. It also reports that “thinking” variants often outperform their non-thinking counterparts, particularly DeepSeek R1 671B over DeepSeek V3 671B and Qwen3 235B Thinking over Qwen3 235B (Wang et al., 3 Sep 2025).

The evaluation therefore exposes a large gap between code-generation fluency and exploit-blocking remediation. Even among the strongest models, successful outcomes remain a minority of benchmark instances. The paper interprets this as evidence that current LLMs are not reliable vulnerability repair systems when judged under a functional exploit oracle.

A representative case study is CVE-2022-21797. The official fix removed dangerous evaluation of a user-controlled string by replacing it with constrained parsing and whitelisted arithmetic over n_jobs. On this case, Gemini 2.5 Pro generated an applicable patch that partially mitigated risk but still relied on eval; GPT-4o mini produced a syntactically invalid patch; and Qwen3 8B Thinking patched the wrong region and added irrelevant edits that did not eliminate the vulnerability. The case is used to illustrate the difference between applicability, plausibility, and true exploit-blocking repair (Wang et al., 3 Sep 2025).

5. Failure modes, prompting studies, and agentic variants

The dominant failure mode reported by VulnRepairEval is inaccurate vulnerability localization. Across models, roughly 60%–78% of failures are attributed to missing the vulnerable code entirely or refusing to patch it. This is followed by patch synthesis and applicability errors, including malformed unified diffs, incorrect context or line numbers, and patches that cannot be applied even under fuzzy patching. A third class of failures consists of semantic errors: patches that are syntactically valid and applicable but alter the wrong region, add irrelevant sanitization, preserve the exploitable path, or otherwise fail to close the attack surface (Wang et al., 3 Sep 2025).

The framework also identifies overly conservative abstention as a specific risk. Some models frequently return “no vulnerability found,” thereby reducing false positives at the cost of many false negatives. VulnRepairEval treats this as hazardous in a security setting because it can create unwarranted confidence that vulnerable code is safe (Wang et al., 3 Sep 2025).

The paper evaluates several prompt enhancements: adding vulnerability type information, adding step-by-step Chain-of-Thought instructions, and including an example patch. These changes produce only minimal or inconsistent gains. Vulnerability-type hints help some models and hurt others; Chain-of-Thought yields only slight improvement for one smaller thinking model; and example patches do not meaningfully improve end-to-end repair success. Multi-agent workflows based on SWE-Agent provide somewhat larger gains in detection and localization—for example, DeepSeek V3 671B sees its “did not find” rate drop from 87% to 26%—but overall exploit-blocking effectiveness remains low (Wang et al., 3 Sep 2025).

These findings align with a parallel methodological critique in a differentiated replication study on LLM-based vulnerability repair, which argues that exact or near-exact localization can inflate apparent repair ability and that evaluation should test sensitivity to localization error (Camporese et al., 28 Jul 2025). VulnRepairEval reaches a related conclusion from the opposite direction: even when only the vulnerable files are supplied, identifying the actual exploitable region remains the principal bottleneck.

6. Position within vulnerability-repair evaluation research

VulnRepairEval belongs to a broader research movement that treats patch validation as a first-class scientific problem rather than an afterthought. PVBench argues that conventional validation based on existing tests plus the original PoC is systematically optimistic, reporting 1250 false positives out of 2952 initially validated patches, for an overall false discovery rate of 42.3%, and showing that over 40% of patches judged correct by basic validation fail developer-authored PoC$P_{\text{corr}}$1 tests (Yu et al., 6 Mar 2026). PATCHEVAL similarly concludes that dynamic testing with both security tests and functionality tests is the most trustworthy validation method, and reports that even the best evaluated system repairs only 23.0% of its PoC-validated subset (Wei et al., 14 Nov 2025).

Taken together, these results suggest that VulnRepairEval’s exploit-failure criterion is a stricter and more security-relevant oracle than patch similarity, compilation, or static plausibility, but not necessarily a complete oracle for semantic correctness. That interpretation is also consistent with the broader AVR literature summarized in the SoK, which argues that benchmarks should not reward patches that merely compile or pass a narrow test, but should assess whether they actually fix the vulnerability without regressions (Li et al., 31 Jan 2025).

The principal significance of VulnRepairEval is therefore methodological. It provides a concrete, containerized framework in which evaluation is tied to real exploitation behavior, not to a proxy divorced from attack execution. Its main empirical message is negative but clarifying: under exploit-grounded evaluation, current LLMs repair only a small fraction of authentic vulnerabilities, enhanced prompting changes little, and agentic scaffolding does not remove the core difficulty. In that sense, VulnRepairEval functions less as a leaderboard artifact than as a calibration instrument for the field, exposing the distance between plausible patch generation and dependable vulnerability remediation (Wang et al., 3 Sep 2025).