Vul4J: Reproducible Java Vulnerability Benchmark
- Vul4J is a benchmark dataset of reproducible Java vulnerabilities featuring original projects, developer patches, and vulnerability-witnessing tests.
- It supports automated repair research by integrating both security-specific PoV tests and functional developer test suites within a Dockerized environment.
- The dataset underpins evaluations in automated vulnerability repair, test mining, and context engineering, despite its limited scale and variability across vulnerabilities.
Vul4J is a benchmark dataset of reproducible Java vulnerabilities designed for research on program repair and automated vulnerability repair. In descriptions reused across later studies, it comprises real-world Java vulnerabilities embedded in their original projects, together with the vulnerable version, the developer patch, proof-of-vulnerability tests, and project test suites, all packaged in a reproducible environment (Al-Maamari, 10 Mar 2026). Subsequent work also characterizes it as a manually curated collection spanning 79 vulnerabilities in 51 open-source Java projects, with 108 manually validated vulnerability-witnessing unit tests and broad CWE coverage, making it a central experimental substrate for security-focused repair, test mining, and benchmark extension in the Java ecosystem (Iannone et al., 5 Feb 2025, Wu et al., 2023).
1. Origin, purpose, and benchmark design
Vul4J was introduced by Bui et al. in 2022 as a dataset of reproducible Java vulnerabilities geared toward the study of program repair techniques. In the form cited by later papers, it contains 79 Java security vulnerabilities, each embedded in its original Java project and accompanied by the vulnerable code version, the human-written patch, one or more PoV tests, and the project’s developer test suite (Al-Maamari, 10 Mar 2026). Another line of work describes the same benchmark as covering 79 vulnerabilities affecting 51 Java projects and providing 108 manually validated unit tests matched to those vulnerabilities (Iannone et al., 5 Feb 2025).
The benchmark is security-specific rather than bug-generic. One study explicitly positions it as complementary to Defects4J by focusing on security vulnerabilities with explicit exploits, while another uses it as the Java-side counterpart to a C/C++ benchmark in a systematization of automated vulnerability repair (Al-Maamari, 10 Mar 2026, Hu et al., 13 Jun 2025). This specialization is operationally important: PoV tests act as a security oracle, while the developer test suite acts as a functional oracle, allowing researchers to ask not only whether a patch compiles, but whether it both blocks exploitation and preserves expected behavior (Al-Maamari, 10 Mar 2026).
Later descriptions emphasize reproducibility as a core design property. Vul4J is packaged with Docker-based reproduction, so that each vulnerability can be built and tested in a controlled environment (Al-Maamari, 10 Mar 2026). In a test-generation study, this before/after structure is treated as the enabling mechanism for defining a vulnerability-witnessing test as one that fails on the vulnerable version and passes on the fixed version (Antal et al., 13 Jun 2025).
2. Dataset contents and the role of witnessing tests
A central feature of Vul4J is its explicit treatment of vulnerability-witnessing tests, also called PoV tests. One study defines such a test as a unit test that passes on the patched version, fails on the vulnerable version, and whose failure specifically indicates that the vulnerability is present (Iannone et al., 5 Feb 2025). Another study adopts the same operational criterion and evaluates generated tests entirely through the FAIL–PASS pattern on Vul4J’s vulnerable and fixed project states (Antal et al., 13 Jun 2025).
The process used to obtain these tests is itself part of the benchmark’s significance. In the description reused by VUTECO, the benchmark authors built the vulnerable and patched versions, ran the post-patch test suite on both versions, selected tests that passed on the patched version but failed on the vulnerable version, and then manually inspected them to confirm that they truly witnessed the vulnerability (Iannone et al., 5 Feb 2025). That study further notes that only 79 out of 899 inspected vulnerabilities yielded such witnessing tests, underscoring the cost of assembling reproducible, security-relevant unit tests at scale (Iannone et al., 5 Feb 2025).
Because of this curation effort, Vul4J is treated not merely as a repository of vulnerable code but as a labeled corpus connecting vulnerabilities, tests, and fixes. One paper calls it “the primary reference collection of witnessing tests,” and states that it is, to date, the only dataset that provides working vulnerability-witnessing tests for Java (Iannone et al., 5 Feb 2025). This makes it usable for tasks that extend beyond patch generation, including identifying whether a test is security-related, matching a test to the exact vulnerability it witnesses, and generating new witnessing tests from code and patch context (Iannone et al., 5 Feb 2025, Antal et al., 13 Jun 2025).
3. Evaluation substrate for automated vulnerability repair
Vul4J’s most visible role is as an evaluation benchmark for automated vulnerability repair. In one failure-analysis study, the authors begin from the full dataset of 79 vulnerabilities, filter for buildability and reproducibility, and retain 64 vulnerabilities spanning 21 CWE categories; for each vulnerability they generate five Gemini 3.0 Flash patches and evaluate 319 patches with a tri-axis protocol covering compilation, security via PoV tests and Semgrep, and functionality via the developer test suite (Al-Maamari, 10 Mar 2026). That work also introduces the Security Repair Score,
thereby turning Vul4J into the basis for a continuous measure of partial repair success rather than a pure pass/fail benchmark (Al-Maamari, 10 Mar 2026).
A different tradition uses a standardized single-hunk subset. “How Effective Are Neural Networks for Fixing Security Vulnerabilities” restricts Vul4J to 35 reproducible single-hunk vulnerabilities because its deep-learning APR baselines are limited to single-hunk bugs (Wu et al., 2023). VulKey explicitly follows that setup and evaluates on 35 single-hunk vulnerabilities from Vul4J using Exact Match against the ground-truth patch (Li et al., 3 May 2026). SGAgent-Vul adopts the same 35-vulnerability subset for repository-level vulnerability repair, again following prior single-hunk settings for comparability (Zhang et al., 27 Feb 2026).
The benchmark is also used in broader assessment studies. In the SoK on automated vulnerability repair, Vul4J is the third-party Java benchmark against which Java AVR tools such as Seader and SeqTrans, and APR tools such as ThinkRepair and SRepair, are evaluated under a unified protocol that measures patch restoration, patch compilation, test pass rate, consistency with the human patch, and success under manual semantic review (Hu et al., 13 Jun 2025). In that setting, Vul4J functions as the Java analogue of a deliberately constructed C/C++ benchmark, reinforcing its role as the main standardized dataset for Java vulnerability repair research (Hu et al., 13 Jun 2025).
4. Empirical behavior revealed by Vul4J
The benchmark has enabled several distinct empirical pictures of LLM- and APR-based repair. In the tri-axis study on 64 reproducible vulnerabilities, 79 of 319 LLM-generated patches, or 24.8%, are classified as “Correct & Secure,” while 164, or 51.4%, are “Insecure & Breaking”; the same study reports a mean Security Score of 0.251, a mean Functionality Score of 0.832, and a mean SRS of 0.542, leading the authors to argue that semantic misunderstanding is the dominant failure mode on Vul4J (Al-Maamari, 10 Mar 2026). In the older neural-repair comparison on the 35 single-hunk subset, Codex fixes 6.2 vulnerabilities correctly and yields 10.9 plausible fixes on Vul4J, outperforming the other evaluated LLMs and DL-based APR techniques, but still repairing only a small fraction of the benchmark (Wu et al., 2023).
More recent agentic and pattern-guided systems report stronger results on the same 35-bug regime. VulKey reports 18 successful repairs on Vul4J with its StarCoder-based configuration, exceeding the figures listed for VulRepair, VulMaster, NTR, and Codex-12B under its Exact Match protocol (Li et al., 3 May 2026). SGAgent-Vul reports 17 resolved vulnerabilities out of 35, or 48.6%, on Vul4J, outperforming FSV-Codex, FSV-finetuned, NTR, and VRPILOT in its comparison table (Zhang et al., 27 Feb 2026). EvoRepair uses Vul4J differently, as a transfer target rather than a training benchmark, and reports 34 fixed vulnerabilities out of 79 for same-model transfer and 25 out of 79 for cross-model transfer when experiences learned on PATCHEVAL are reused on Vul4J (Hu et al., 28 May 2026).
These figures are generated under different subsets, objectives, and validation protocols. This suggests that Vul4J is less a single leaderboard than a common experimental substrate. Some studies evaluate exact patch match on 35 single-hunk instances, others evaluate PoC-and-unit-test success on all 79 CVEs, and still others decompose repair into compilation, security, and functionality (Li et al., 3 May 2026, Hu et al., 28 May 2026, Al-Maamari, 10 Mar 2026). The benchmark’s value lies partly in supporting all of these views without changing the underlying project-level vulnerabilities.
5. Uses beyond patch generation
Vul4J has also become a core resource for research on vulnerability-witnessing tests. VUTECO uses Vul4J as the sole source of ground truth for two tasks: finding whether a JUnit test case is security-related, and matching a test case to the exact vulnerability it witnesses (Iannone et al., 5 Feb 2025). From the 51 projects in Vul4J, that work extracts 62,635 test cases: 108 witnessing tests and 62,527 non-witnessing tests. On Vul4J-derived test sets, its best Finder configuration achieves precision 1.00 and , while its integrated Matching model reaches precision 0.86 and (Iannone et al., 5 Feb 2025).
A complementary line of work studies whether LLMs can synthesize witnessing tests from Vul4J’s vulnerable and fixed code. A GPT-4-based study selects 50 VUL4J entries, each under four AST-derived context levels, and reports overall syntactic correctness of 66.5%, automatic semantic correctness of 7.5%, and subjective usability of 68.5% for the generated tests (Antal et al., 13 Jun 2025). The same work treats the FAIL–PASS pattern on Vul4J’s before/after project states as the decisive semantic criterion for a vulnerability-witnessing test (Antal et al., 13 Jun 2025).
Prompt-context research on vulnerability repair also uses Vul4J as a controlled environment for studying how metadata should be presented to LLMs. A study on GPT-4o evaluates 42 Vul4J vulnerabilities, three runs per prompt, and reports that raw CVE information improves repair rates more than CWE descriptions, while the best-performing combined strategy—“Short + CVE Hints + Manual”—reaches an average repair rate of 38.9%; using the top three prompts together, GPT-4o repairs 26 of 42 vulnerabilities at least once (Antal et al., 13 Jun 2025). Here Vul4J acts not only as a benchmark, but as a testbed for context engineering.
6. Extensions, limitations, and position in the broader ecosystem
Several later datasets and protocols are explicitly defined in relation to Vul4J. VJBench and VJBench-trans extend Vul4J with additional Java vulnerabilities and transformed variants intended to mitigate training–test overlap risks for LLMs (Wu et al., 2023, Camporese et al., 28 Jul 2025). The differentiated replication study on LLM repair treats VJBench and VJBench-trans as extensions of Vul4J and proposes to perturb line-level fault localization on these Vul4J-derived benchmarks in order to study whether reported successes depend on perfect localization or memorized fixes (Camporese et al., 28 Jul 2025). ManyVuls4J, by contrast, is a manually curated extension built from a later ProjectKB snapshot; it expands the number of vulnerabilities from 79 to 103, increasing scale by about 30% while preserving the same reproducibility criteria (Lin et al., 2024).
The benchmark’s limitations are repeatedly acknowledged. One study notes that Vul4J’s original scale is only 79 vulnerabilities and that, in its environment, only 64 were reproducible and includable (Al-Maamari, 10 Mar 2026). Another emphasizes the small number of witnessing tests—108 total—and the uneven coverage of vulnerability classes, giving the example that Vul4J contains eight witnessing tests for CWE-835 but only one for CWE-78 (Iannone et al., 5 Feb 2025). The SoK reports that, in its own environment, only 68 of the 79 vulnerabilities were compilable for Seader, and that tool-specific structural constraints further shrink the usable subset to 15 vulnerabilities for SeqTrans, 33 for ThinkRepair, and 63 for SRepair (Hu et al., 13 Jun 2025). These are not contradictions so much as reminders that Vul4J is a project-level benchmark whose effective size depends on the repair pipeline and reproduction environment.
Despite those constraints, later work consistently treats Vul4J as the primary Java benchmark for reproducible vulnerability repair and closely related tasks. It is used as the Java-side benchmark for cross-language evaluation in VulKey, as the independent transfer target in EvoRepair, as one of two security benchmarks in SGAgent, as the witness-test corpus in VUTECO, and as the test-generation substrate in GPT-4 studies (Li et al., 3 May 2026, Hu et al., 28 May 2026, Zhang et al., 27 Feb 2026, Iannone et al., 5 Feb 2025, Antal et al., 13 Jun 2025). A plausible implication is that Vul4J’s lasting importance comes less from raw size than from the uncommon combination of real projects, developer patches, PoV tests, developer test suites, and reproducible execution environments. That combination makes it simultaneously a repair benchmark, a security oracle, and a corpus for studying how vulnerabilities, tests, and patches relate in Java.