VJTrans: Transformed Java Vulnerability Benchmark
- VJTrans is a transformed Java vulnerability benchmark that applies refactoring techniques, including control-flow changes and identifier renaming, to reduce LLM memorization.
- It extends datasets like VJBench and Vul4J by generating transformed single-hunk vulnerabilities and incorporating localization perturbation to simulate unseen repair scenarios.
- Its design serves as a leakage-resistant stress test for automated vulnerability repair experiments, though its limited size and Java-only focus pose challenges for broader generalization.
VJTrans denotes, in the most direct documented usage presently available on arXiv, a transformed Java vulnerability benchmark associated with the VJBench/VJBench-trans line introduced by Wu et al. and used in a registered-report study of LLM-based automated vulnerability repair (Camporese et al., 28 Jul 2025). In that study, the benchmark is not reintroduced as a standalone artifact in full detail; rather, it is treated as a transformed derivative of Vul4J-style instances whose purpose is to mitigate the advantage for LLMs already exposed to the testing data during training. The same record also shows that the label is ambiguous across neighboring research areas: several technically relevant systems have related names, but the papers do not support treating them as identical to VJTrans. The most precise current usage, therefore, is as a leakage-mitigating transformed benchmark for Java vulnerability repair experiments (Camporese et al., 28 Jul 2025).
1. Naming, lineage, and scope
The most explicit description appears in "Repairing vulnerabilities without invisible hands. A differentiated replication study on LLMs" (Camporese et al., 28 Jul 2025). There, the terminology is not fully stable: the paper mainly uses “VJBench-trans” and, in one threats-to-validity passage, “VJTransBench”; these are presented as referring to the transformed version of VJBench that the query label “VJTrans” designates.
Within that account, Wu et al. are said to have extended the Vul4J benchmark and proposed VJBench-trans, described as a dataset of transformed versions of collected single-hunk vulnerabilities (Camporese et al., 28 Jul 2025). The stated rationale is explicit: the transformations are intended to mitigate the advantage for LLMs that were already exposed to the testing data during their training. The benchmark is therefore not framed as a model, a repair method, or a repair baseline; it is a transformed evaluation corpus.
The paper does not fully redefine the benchmark itself. It states that the dataset choice falls on VJBench and VJBench-trans because these extend Vul4J, contain Java vulnerabilities, and include at least 50 single-hunk vulnerabilities. It also states that the study aims to widen the selection to single-function vulnerabilities and to use the scripts of the refactoring process of VJBench-trans to generate vulnerabilities “that the LLMs have never seen” (Camporese et al., 28 Jul 2025). The exact final count of VJTrans instances used in the planned experiments is not given.
2. Transformations and benchmark rationale
The transformations attached to VJTrans are described only briefly but consistently. The paper mentions control-flow changes and identifier renaming as examples, and it highlights a stronger renaming variant in which identifiers are renamed into a language different than English (Camporese et al., 28 Jul 2025). The benchmark’s methodological significance lies precisely in these refactorings: they are intended to make exact memorization of known vulnerable code and corresponding developer fixes less useful.
This use of transformed code is tightly coupled to a specific research question about LLM-based automated vulnerability repair. The study hypothesizes that strong AVR performance may be partly driven by “invisible hands” such as training-data leakage or exact fault localization. VJTrans is introduced into that setting as a leakage-resistant stress test: if repair performance survives the transformations, that would be more consistent with generalization; if it degrades sharply, that would be more consistent with memorization-sensitive behavior (Camporese et al., 28 Jul 2025).
The inclusion criteria adopted for dataset selection reinforce this role. The chosen datasets are required to have been used for LLM AVR evaluation, to contain real-world data, to include regression and PoV tests, to include diverse entries, to contain single function vulnerabilities, and to contain refactored vulnerabilities for which the corresponding developer fix is not available (Camporese et al., 28 Jul 2025). In that sense, VJTrans is defined less by a taxonomy of vulnerability types than by a benchmark design principle: transformed, testable, Java repair instances intended to weaken exact-match leakage.
3. Instance preparation, prompts, and localization perturbation
For selected vulnerabilities, including VJTrans instances, the repair pipeline supplies the LLM with the full vulnerable function (Camporese et al., 28 Jul 2025). The study defines four prompt settings:
| Prompt | Contents |
|---|---|
| P1 | general information + vulnerable function + output request |
| P2 | P1 + vulnerable lines marked with suffix // BUG |
| P3 | P1 + vulnerability description |
| P4 | P3 + vulnerable lines marked with suffix // BUG |
A notable design choice is that the code remains functionally valid: the line marker is appended as a suffix rather than replacing code with a hole (Camporese et al., 28 Jul 2025). This matters because the study is not testing masked completion in isolation; it is testing repair under varying localization and information conditions.
The paper’s core manipulation is fault localization perturbation. For prompts with line-level localization, the marked line is intentionally shifted by lines from the ground truth, with six variants: and baseline $0$ for exact localization (Camporese et al., 28 Jul 2025). Variants that fall outside the function’s scope are discarded. The rebuttal further notes that the study deliberately avoids a 1-line displacement because the difference from exact localization would not be sufficient to distinguish the hypothesized behaviors under the git diff window (Camporese et al., 28 Jul 2025).
The paper also anticipates an asymmetry between negative and positive offsets. A negative offset, placed before the vulnerable line, is hypothesized to allow a “cheater” LLM still to emit the memorized repair, whereas a positive offset may lead to hallucination or poorer repair behavior (Camporese et al., 28 Jul 2025). This is not an empirical result in the paper; it is part of the planned interpretation framework.
4. Validation pipeline and statistical methodology
VJTrans instances are embedded in a multi-stage AVR validation workflow. For each generated patch, the patched function replaces the vulnerable one in the original project, the project is compiled or built, regression tests are run, and PoV tests are run (Camporese et al., 28 Jul 2025). The paper defines a PoV test as a test that flips between the vulnerable and non-vulnerable code fragments.
A second-LLM review stage is then applied. The second LLM receives the original input together with the generated patched function and outputs “TRUE” if it judges the patch to fix the vulnerability while preserving functionality, otherwise “FALSE” (Camporese et al., 28 Jul 2025). This review step is part of both the exact-localization and localization-perturbation analyses.
The statistical plan is unusually explicit for a benchmark usage study. For prompt and localization comparisons, the paper uses TOST (two one-sided tests) for equivalence, with Mann-Whitney U (MWU) as the underlying directional test and Helmert contrast for multiple comparisons (Camporese et al., 28 Jul 2025). Several hypotheses are formulated around equivalence, including prompt-information equivalence, function-level versus line-level localization equivalence, and equivalence between small and larger localization offsets.
Manual inspection is also planned for patches that survive testing. The manual labels are Correct, Partially correct, and Wrong, and two authors are to inspect patches independently, reconcile disagreements, and additionally mark whether the LLM reasoning is right or wrong (Camporese et al., 28 Jul 2025). Confidence intervals on proportions are to be estimated with the Agresti-Coull-Wilson method. Sample sizing is stated using Cochran’s formula: with , , and worst-case , yielding patches for a 95\% confidence interval with a 10\% margin of error (Camporese et al., 28 Jul 2025).
5. Reported status, limitations, and validity threats
The study that directly uses VJTrans is a registered report / study plan, and this sharply constrains what can be claimed. No completed empirical VJTrans outcomes are reported: there are no actual repair success rates, no plausible-patch counts, no correct-patch counts, no review precision/recall, and no completed leakage findings specific to VJTrans in the available text (Camporese et al., 28 Jul 2025). Nor does the paper report completed comparisons between VJTrans and Vul4J.
The paper also makes several caveats explicit. First, benchmark size is limited: Vul4J and its derivatives (e.g., VJBench, VJTransBench) contain only 50–100 manually curated, reproducible vulnerabilities with tests (Camporese et al., 28 Jul 2025). This constrains statistical robustness and generalizability. Second, leakage is mitigated but not eliminated. Even with transformed code and non-English identifier renaming, the rebuttal notes that the probability of a memorization-based mapping is not zero, only highly unlikely (Camporese et al., 28 Jul 2025).
Third, the scope is Java-only. The paper’s conclusions are therefore bound to a Java evaluation setting, and it does not claim transfer to C/C++, Python, or other ecosystems (Camporese et al., 28 Jul 2025). Fourth, the study depends on the VJBench-trans refactoring scripts. This makes the benchmark methodologically useful, but it also means that claims about truly unseen vulnerabilities depend on whether those transformations preserve the vulnerability-repair structure while genuinely weakening memorization channels.
A further limitation is definitional rather than experimental: because the paper does not fully redefine the benchmark itself, VJTrans is best understood through its operational role in the study rather than through a complete benchmark specification (Camporese et al., 28 Jul 2025). This suggests that any fuller encyclopedia treatment of VJTrans would eventually need the original Wu et al. source as a primary reference.
6. Disambiguation from similarly named systems
The arXiv record around the string “VJTrans” is heterogeneous, and the distinction between direct naming and nearby technical relevance is essential. "VNJPTranslate: A comprehensive pipeline for Vietnamese-Japanese translation" explicitly states that it does not mention any system named “VJTrans”; it is relevant only if the label is being used informally for Vietnamese–Japanese translation technology (Phan et al., 1 Apr 2025). A cautious reading is therefore that VNJPTranslate is related background, not evidence of identity.
Several papers belong to a separate cluster concerned with video transitions rather than vulnerability repair. "V-Trans4Style: Visual Transition Recommendation for Video Production Style Adaptation" formulates style-aware transition recommendation as a sequential editing problem over ordered clips and transition classes (Guhan et al., 14 Jan 2025). "AutoTransition: Learning to Recommend Video Transition Effects" defines automatic video transition recommendation as a multimodal retrieval problem over neighboring shots and audio (Shen et al., 2022). "Versatile Transition Generation with Image-to-Video Diffusion" treats transition generation as endpoint-conditioned video synthesis from first frame, last frame, and text (Yang et al., 3 Aug 2025). These works are highly relevant only if “VJTrans” is being used informally for a video-transition system; the papers themselves do not identify their methods as VJTrans.
Other near matches are also distinct. "VJT: A Video Transformer on Joint Tasks of Deblurring, Low-light Enhancement and Denoising" uses the name VJT, not VJTrans, for a multi-tier video transformer for joint restoration (Hui et al., 2024). "TransientViT: A novel CNN - Vision Transformer hybrid real/bogus transient classifier for the Kilodegree Automatic Transient Survey" concerns astronomical transient classification and is not presented as VJTrans (Chen et al., 2023). The cumulative implication is straightforward: across the supplied arXiv material, VJTrans has one direct benchmark-centered meaning and several merely adjacent or potentially confusable name neighbors.
Taken together, this makes VJTrans an example of a term whose technical meaning currently depends on disciplinary context. In the strongest documented sense, it refers to a transformed Java vulnerability benchmark used to probe whether LLM repair performance persists under refactoring and localization perturbation (Camporese et al., 28 Jul 2025). In other contexts, similar strings designate unrelated systems in machine translation, video editing, transition synthesis, video restoration, and transient astronomy, but the papers do not justify collapsing those usages into a single entity.
