---
title: 'VJTrans: Transformed Java Vulnerability Benchmark'
url: https://www.emergentmind.com/topics/vjtrans
type: topic
---

# VJTrans: Transformed Java Vulnerability Benchmark

VJTrans denotes, in the most direct documented usage presently available on arXiv, a transformed Java vulnerability benchmark associated with the VJBench/VJBench-trans line introduced by Wu et al. and used in a registered-report study of LLM-based automated vulnerability repair [2507.20977]. In that study, the benchmark is not reintroduced as a standalone artifact in full detail; rather, it is treated as a transformed derivative of Vul4J-style instances whose purpose is to mitigate the advantage for LLMs already exposed to the testing data during training. The same record also shows that the label is ambiguous across neighboring research areas: several technically relevant systems have related names, but the papers do not support treating them as identical to VJTrans. The most precise current usage, therefore, is as a leakage-mitigating transformed benchmark for Java vulnerability repair experiments [2507.20977].

## 1. Naming, lineage, and scope

The most explicit description appears in "Repairing vulnerabilities without invisible hands. A differentiated replication study on LLMs" [2507.20977]. There, the terminology is not fully stable: the paper mainly uses **“VJBench-trans”** and, in one threats-to-validity passage, **“VJTransBench”**; these are presented as referring to the transformed version of VJBench that the query label **“VJTrans”** designates.

Within that account, Wu et al. are said to have **extended the Vul4J benchmark** and proposed **VJBench-trans**, described as a dataset of **transformed versions** of collected **single-hunk vulnerabilities** [2507.20977]. The stated rationale is explicit: the transformations are intended to **mitigate the advantage for LLMs that were already exposed to the testing data during their training**. The benchmark is therefore not framed as a model, a repair method, or a repair baseline; it is a transformed evaluation corpus.

The paper does not fully redefine the benchmark itself. It states that the dataset choice falls on **VJBench and VJBench-trans** because these extend Vul4J, contain Java vulnerabilities, and include **at least 50 single-hunk vulnerabilities**. It also states that the study aims to widen the selection to **single-function vulnerabilities** and to use the **scripts of the refactoring process of VJBench-trans** to generate vulnerabilities **“that the LLMs have never seen”** [2507.20977]. The exact final count of VJTrans instances used in the planned experiments is not given.

## 2. Transformations and benchmark rationale

The transformations attached to VJTrans are described only briefly but consistently. The paper mentions **control-flow changes** and **identifier renaming** as examples, and it highlights a stronger renaming variant in which identifiers are renamed into **a language different than English** [2507.20977]. The benchmark’s methodological significance lies precisely in these refactorings: they are intended to make exact memorization of known vulnerable code and corresponding developer fixes less useful.

This use of transformed code is tightly coupled to a specific research question about LLM-based automated vulnerability repair. The study hypothesizes that strong AVR performance may be partly driven by **“invisible hands”** such as training-data leakage or exact fault localization. VJTrans is introduced into that setting as a **leakage-resistant stress test**: if repair performance survives the transformations, that would be more consistent with generalization; if it degrades sharply, that would be more consistent with memorization-sensitive behavior [2507.20977].

The inclusion criteria adopted for dataset selection reinforce this role. The chosen datasets are required to have been used for **LLM AVR evaluation**, to contain **real-world data**, to include **regression and PoV tests**, to include **diverse entries**, to contain **single function vulnerabilities**, and to contain **refactored vulnerabilities** for which the corresponding developer fix is not available [2507.20977]. In that sense, VJTrans is defined less by a taxonomy of vulnerability types than by a benchmark design principle: transformed, testable, Java repair instances intended to weaken exact-match leakage.

## 3. Instance preparation, prompts, and localization perturbation

For selected vulnerabilities, including VJTrans instances, the repair pipeline supplies the LLM with the **full vulnerable function** [2507.20977]. The study defines four prompt settings:

| Prompt | Contents |
|---|---|
| **P1** | general information + vulnerable function + output request |
| **P2** | P1 + vulnerable lines marked with suffix `// BUG` |
| **P3** | P1 + vulnerability description |
| **P4** | P3 + vulnerable lines marked with suffix `// BUG` |

A notable design choice is that the code remains functionally valid: the line marker is appended as a suffix rather than replacing code with a hole [2507.20977]. This matters because the study is not testing masked completion in isolation; it is testing repair under varying localization and information conditions.

The paper’s core manipulation is **fault localization perturbation**. For prompts with line-level localization, the marked line is intentionally shifted by \(n\) lines from the ground truth, with six variants:
\[
-2,\ +2,\ -4,\ +4,\ -8,\ +8
\]
and baseline \(0\) for exact localization [2507.20977]. Variants that fall outside the function’s scope are discarded. The rebuttal further notes that the study deliberately avoids a 1-line displacement because the difference from exact localization would not be sufficient to distinguish the hypothesized behaviors under the git diff window [2507.20977].

The paper also anticipates an asymmetry between negative and positive offsets. A negative offset, placed before the vulnerable line, is hypothesized to allow a “cheater” LLM still to emit the memorized repair, whereas a positive offset may lead to hallucination or poorer repair behavior [2507.20977]. This is not an empirical result in the paper; it is part of the planned interpretation framework.

## 4. Validation pipeline and statistical methodology

VJTrans instances are embedded in a multi-stage AVR validation workflow. For each generated patch, the patched function replaces the vulnerable one in the original project, the project is compiled or built, **regression tests** are run, and **PoV tests** are run [2507.20977]. The paper defines a PoV test as a test that flips between the vulnerable and non-vulnerable code fragments.

A second-LLM review stage is then applied. The second LLM receives the original input together with the generated patched function and outputs **“TRUE”** if it judges the patch to fix the vulnerability while preserving functionality, otherwise **“FALSE”** [2507.20977]. This review step is part of both the exact-localization and localization-perturbation analyses.

The statistical plan is unusually explicit for a benchmark usage study. For prompt and localization comparisons, the paper uses **TOST (two one-sided tests)** for equivalence, with **Mann-Whitney U (MWU)** as the underlying directional test and **Helmert contrast** for multiple comparisons [2507.20977]. Several hypotheses are formulated around equivalence, including prompt-information equivalence, function-level versus line-level localization equivalence, and equivalence between small and larger localization offsets.

Manual inspection is also planned for patches that survive testing. The manual labels are **Correct**, **Partially correct**, and **Wrong**, and two authors are to inspect patches independently, reconcile disagreements, and additionally mark whether the **LLM reasoning is right or wrong** [2507.20977]. Confidence intervals on proportions are to be estimated with the **Agresti-Coull-Wilson method**. Sample sizing is stated using **Cochran’s formula**:
\[
n=z_\alpha^2\cdot Pr_C(1-Pr_C)/\epsilon^2
\]
with \(\alpha=5\%\), \(\epsilon=10\%\), and worst-case \(Pr_C=1-Pr_C=0.5\), yielding **\(n=96\)** patches for a **95\% confidence interval** with a **10\% margin of error** [2507.20977].

## 5. Reported status, limitations, and validity threats

The study that directly uses VJTrans is a **registered report / study plan**, and this sharply constrains what can be claimed. No completed empirical VJTrans outcomes are reported: there are **no actual repair success rates**, **no plausible-patch counts**, **no correct-patch counts**, **no review precision/recall**, and **no completed leakage findings** specific to VJTrans in the available text [2507.20977]. Nor does the paper report completed comparisons between VJTrans and Vul4J.

The paper also makes several caveats explicit. First, benchmark size is limited: **Vul4J and its derivatives (e.g., VJBench, VJTransBench) contain only 50–100 manually curated, reproducible vulnerabilities with tests** [2507.20977]. This constrains statistical robustness and generalizability. Second, leakage is mitigated but not eliminated. Even with transformed code and non-English identifier renaming, the rebuttal notes that the probability of a memorization-based mapping is **not zero**, only **highly unlikely** [2507.20977].

Third, the scope is Java-only. The paper’s conclusions are therefore bound to a **Java evaluation setting**, and it does not claim transfer to C/C++, Python, or other ecosystems [2507.20977]. Fourth, the study depends on the **VJBench-trans refactoring scripts**. This makes the benchmark methodologically useful, but it also means that claims about truly unseen vulnerabilities depend on whether those transformations preserve the vulnerability-repair structure while genuinely weakening memorization channels.

A further limitation is definitional rather than experimental: because the paper does not fully redefine the benchmark itself, VJTrans is best understood through its operational role in the study rather than through a complete benchmark specification [2507.20977]. This suggests that any fuller encyclopedia treatment of VJTrans would eventually need the original Wu et al. source as a primary reference.

## 6. Disambiguation from similarly named systems

The arXiv record around the string “VJTrans” is heterogeneous, and the distinction between direct naming and nearby technical relevance is essential. "VNJPTranslate: A comprehensive pipeline for Vietnamese-Japanese translation" explicitly states that it does **not** mention any system named **“VJTrans”**; it is relevant only if the label is being used informally for Vietnamese–Japanese translation technology [2504.00339]. A cautious reading is therefore that VNJPTranslate is related background, not evidence of identity.

Several papers belong to a separate cluster concerned with **video transitions** rather than vulnerability repair. "V-Trans4Style: Visual Transition Recommendation for Video Production Style Adaptation" formulates style-aware transition recommendation as a sequential editing problem over ordered clips and transition classes [2501.07983]. "AutoTransition: Learning to Recommend Video Transition Effects" defines automatic video transition recommendation as a multimodal retrieval problem over neighboring shots and audio [2207.13479]. "Versatile Transition Generation with Image-to-Video Diffusion" treats transition generation as endpoint-conditioned video synthesis from first frame, last frame, and text [2508.01698]. These works are highly relevant only if “VJTrans” is being used informally for a video-transition system; the papers themselves do not identify their methods as VJTrans.

Other near matches are also distinct. "VJT: A Video Transformer on Joint Tasks of Deblurring, Low-light Enhancement and Denoising" uses the name **VJT**, not **VJTrans**, for a multi-tier video transformer for joint restoration [2401.14754]. "TransientViT: A novel CNN - Vision Transformer hybrid real/bogus transient classifier for the Kilodegree Automatic Transient Survey" concerns astronomical transient classification and is not presented as VJTrans [2309.09937]. The cumulative implication is straightforward: across the supplied arXiv material, **VJTrans** has one direct benchmark-centered meaning and several merely adjacent or potentially confusable name neighbors.

Taken together, this makes VJTrans an example of a term whose technical meaning currently depends on disciplinary context. In the strongest documented sense, it refers to a transformed Java vulnerability benchmark used to probe whether LLM repair performance persists under refactoring and localization perturbation [2507.20977]. In other contexts, similar strings designate unrelated systems in machine translation, video editing, transition synthesis, video restoration, and transient astronomy, but the papers do not justify collapsing those usages into a single entity.

Source: https://www.emergentmind.com/topics/vjtrans