RevisitVD: PLM Evaluation for Vulnerability Detection

• RevisitVD is an evaluation framework that redefines PLM-based vulnerability detection using curated datasets, time-aware partitioning, and realistic testing conditions.
• It compares fine-tuning and prompt-engineering approaches, revealing that fine-tuning outperforms prompting by approximately 15% under practical scenarios.
• Robustness analyses in RevisitVD, including normalization and abstraction tests, highlight critical failure modes and the importance of realistic evaluation protocols.

RevisitVD is an evaluation framework and empirical study for function-level vulnerability detection with pre-trained LLMs (PLMs). Introduced in “Revisiting Pre-trained LLMs for Vulnerability Detection” (Li et al., 22 Jul 2025), it re-examines how vulnerability detectors should be assessed by combining newly curated datasets, time-aware partitioning, fine-tuning and prompt-engineering comparisons, out-of-distribution testing, and robustness analyses under normalization, abstraction, semantic-preserving transformations, and truncation. The study evaluates 17 PLMs spanning code-specific small PLMs, code LLMs, and general-purpose LLMs, and argues that realistic evaluation protocols materially change conclusions about model quality.

1. Motivation and evaluation philosophy

RevisitVD is motivated by the claim that earlier PLM evaluations for vulnerability detection often gave misleadingly optimistic or incomplete results because they handled data preparation, train/test partitioning, and experimental settings inadequately (Li et al., 22 Jul 2025). The study identifies three recurring problems.

First, earlier datasets and splits could introduce leakage. Random partitioning could place pre-patch and post-patch versions of related code across train and test, and similar fixes from the same commit could also be separated across splits. The paper also criticizes the common heuristic that “pre-patch function = vulnerable,” because unrelated changed code may be labeled vulnerable simply for appearing in a vulnerable commit.

Second, earlier evaluations often used narrow model sets or unrealistic class distributions. RevisitVD argues that balanced test sets are not representative of deployment conditions, because realistic vulnerability detection is strongly imbalanced. It therefore keeps validation and test sets imbalanced and studies the effect of balanced versus imbalanced training separately.

Third, earlier work often emphasized topline scores while neglecting practical failure modes. RevisitVD treats code normalization, code abstraction, semantic-preserving transformations, and truncation-induced label inconsistency as first-class evaluation dimensions. This shifts the benchmark from a single-score comparison toward a broader realism-oriented assessment.

2. Dataset construction and partitioning

RevisitVD uses two principal datasets: a reconstructed dataset based on PrimeVul and a self-collected NVD-based dataset (Li et al., 22 Jul 2025). The reconstructed dataset is used for train/validation/test experiments, while the NVD-based dataset functions as a temporally newer holdout set intended to reduce pretraining-data leakage.

Dataset	Role	Size
Reconstructed PrimeVul	Train/validation/test	184,920 imbalanced-train functions; 10,862 balanced-train functions; 23,112 validation; 23,144 test
Self-collected NVD	Test-only holdout	25,536 functions; 646 vulnerable; 24,890 non-vulnerable; 99 projects; 28 CWE types

For the reconstructed PrimeVul dataset, RevisitVD applies a fine-grained time-based partitioning strategy. Data are grouped by CWE type, each CWE group is sorted by commit date, and each group is split into train/validation/test with an 8:1:1 ratio (Li et al., 22 Jul 2025). This preserves vulnerability-type distribution while reducing leakage. The paper reports that it examined the reconstructed dataset and found no data leakage for the two leakage cases it explicitly targeted.

The self-collected dataset is built from all C/C++ vulnerability-related GitHub commits in NVD dated from October 2023 to October 2024. Functions before patching are labeled vulnerable, and functions after patching and unchanged functions are labeled non-vulnerable (Li et al., 22 Jul 2025). The date range is chosen to be later than the pretraining cutoffs of all evaluated PLMs, so the dataset also serves as a knowledge-cutoff-aware evaluation set.

3. Model families, adaptation methods, and evaluation metrics

RevisitVD evaluates 17 PLMs across several architectural families (Li et al., 22 Jul 2025). The code-specific small PLMs are CodeBERT, GraphCodeBERT, UniXCoder, PDBERT, and CodeT5. The code LLMs are Code Llama-7B, Code Llama-13B, Code Llama-34B, DeepSeek-Coder-6.7B, DeepSeek-Coder-34B, StarChat-β-16B, WizardCoder-15B, and WizardCoder-34B. The general-purpose LLMs are Mistral-7B, GPT-3.5, GPT-4, and GPT-4o Mini.

The study compares two adaptation paradigms. Fine-tuning uses AdamW, a linear schedule, a 10% warm-up ratio, batch size 32, 10 epochs, and learning rate $2\times10^{-5}$ (Li et al., 22 Jul 2025). For Code SLMs, the classifier is a RobertaClassificationHead with two linear layers, hidden size 768, dropout 0.1, and tanh. For LLMs, the study uses AutoModelForSequenceClassification with LoRA adapters on all linear layers; the LoRA rank is 64, alpha is 16, and dropout is 0.05.

Prompt engineering is evaluated in zero-shot and 4-shot in-context settings. The system role is “You are a code security expert who excels at detecting vulnerabilities,” and the user prompt asks, “Is the following function vulnerable? Please answer Yes or No” (Li et al., 22 Jul 2025). Four prompt representations are used: raw code, flattened AST, code with API calls, and code with data flow. Prompted inference uses $top\text{-}p = 0.9$, temperature $= 0$, and max new tokens $= 10$.

The evaluation metrics are standard binary-classification quantities, with a particular emphasis on balanced accuracy under class imbalance:

$\text{Accuracy} = (TP + TN)/(TP + TN + FP + FN)$

$\text{Recall} = TP/(TP + FN)$

$\text{Precision} = TP/(TP + FP)$

$\text{True Negative Rate (TNR)} = TN/(TN + FP)$

$$F1 = 2 \cdot (\text{Precision}\cdot\text{Recall})/(\text{Precision} + \text{Recall})$$

$$\text{Balanced Accuracy} = ((\text{Recall} + \text{TNR}))/2$$

4. Main empirical findings

A central RevisitVD result is that balanced versus imbalanced training produces qualitatively different conclusions (Li et al., 22 Jul 2025). Under imbalanced training, Code SLMs attain very high TNR and high raw accuracy by predicting mostly negatives, but their recall often falls below 10%. Under balanced training, recall increases sharply and balanced accuracy becomes substantially more informative.

PDBERT is the strongest Code SLM. On the reconstructed test set under balanced training, it reaches Accuracy 76.91, Balanced Accuracy 78.33, F1 17.18, Precision 9.62, Recall 79.83, and TNR 76.82 (Li et al., 22 Jul 2025). On the self-collected NVD holdout set, PDBERT reaches Accuracy 81.47, Balanced Accuracy 71.27, F1 14.18, Precision 8.03, Recall 60.53, and TNR 82.01. The paper reports that PDBERT outperforms the other SLMs by about 3.2%–4.7% balanced accuracy on average across the two test sets.

The study also finds that fine-tuning substantially outperforms prompting. It summarizes the gap as about 15% overall (Li et al., 22 Jul 2025). Among fine-tuned LLMs, DeepSeek-Coder-34B is the strongest, with Balanced Accuracy 70.48 on the self-collected dataset, close to but slightly below PDBERT. Prompting performs much worse and can become pathological for some models. For example, prompted Code Llama-13B reaches Accuracy 2.80, TNR 0.04, and Recall 100.0, indicating near-universal positive predictions (Li et al., 22 Jul 2025).

These results support the paper’s main thesis: PLMs incorporating pre-training tasks designed to capture the syntactic and semantic patterns of code outperform both general-purpose PLMs and those solely pre-trained or fine-tuned on large code corpora (Li et al., 22 Jul 2025). In RevisitVD, scale alone does not dominate.

5. Robustness analyses and failure modes

RevisitVD’s robustness analyses show that current PLMs remain brittle in several practically important ways (Li et al., 22 Jul 2025). The first is normalization sensitivity. The study evaluates three normalization settings—CodeXGLUE cleaner, PDBERT cleaner, and no normalization—and finds that performance is highest when training and testing normalization match. For PDBERT, removing newline characters causes about a 6.9% performance drop.

The second is abstraction. Using Joern, the study abstracts variables, parameters, and strings into placeholders such as VAR0, PARAM0, and STRING0. After abstraction, false-positive rate decreases and false-negative rate increases, while average balanced accuracy drops by only about 1% (Li et al., 22 Jul 2025). The paper interprets this as evidence that PLMs are not relying solely on names and strings, although lexical cues still matter.

The third is semantic-preserving transformation. RevisitVD applies conditional branch negation, conditional branch expansion, loop branch conversion, and relational operator reversal to the reconstructed test set. From 23,144 original functions, it generates 48,182 semantically equivalent transformed functions (Li et al., 22 Jul 2025). Most PLMs degrade under these transformations. Conditional branch expansion is the easiest case, with an average performance decrease of 1.3%, while conditional branch negation is the hardest, with an average performance decrease of 15.8%. UniXCoder is the most robust model in this setting, with an average performance drop of only 0.1%.

A fourth failure mode is truncation-induced labeling error. RevisitVD observes that Code SLMs with a 512-token limit can receive identical visible prefixes for vulnerable and fixed functions, while the labels remain opposite after truncation (Li et al., 22 Jul 2025). Among 5,480 patch pairs in PrimeVul, 1,473 pairs—27%—have this contradictory-label issue. To mitigate it, the paper uses code slicing with tree-climber, anchored on API calls, array usage, pointer usage, and arithmetic expressions, and slices bidirectionally until the 512-token budget is reached. Training on the sliced set improves Code SLM performance by an average of 4%.

6. Interpretation, limitations, and future directions

RevisitVD interprets its results as evidence that vulnerability detection depends strongly on explicit modeling of code structure and dependency semantics (Li et al., 22 Jul 2025). This is why PDBERT, with statement-level control dependency prediction and token-level data dependency prediction in pretraining, performs especially well. The same pattern appears in per-CWE recall: models generally perform better—often over 70% recall on average—on CWE-119, CWE-399, CWE-189, and CWE-787, which often involve recognizing missing sanity or bounds checks, and worse on CWE-264 and CWE-476, which more often require complex dependencies or external context.

The study is explicit about its limitations. It focuses on function-level vulnerability detection rather than cross-function or repository-level analysis (Li et al., 22 Jul 2025). Undetectable leakage may still remain even though the self-collected dataset is later than the knowledge cutoffs of all evaluated PLMs. Chain-of-Thought prompting is excluded because of budget constraints, and the latest reasoning models are also excluded because of leakage concerns and limited post-cutoff vulnerability data.

The broader significance of RevisitVD lies in its recalibration of what PLM-based vulnerability detection should mean in practice. It argues that realistic evaluation requires time-aware splitting, imbalanced testing, out-of-distribution assessment, and robustness checks against formatting, abstraction, semantic-preserving rewrites, and truncation (Li et al., 22 Jul 2025). A plausible implication is that future progress will depend less on generic scaling and more on leakage-resistant benchmarks, long-context or sliced-context handling, and pretraining objectives that explicitly encode program dependencies and vulnerability-relevant semantics.