VulTegra: Evaluation Framework for Vulnerability Detectors
- The paper introduces VulTegra, an evaluation framework that challenges single-score benchmarking by testing detectors across CWE categories, real-world data, and emerging vulnerabilities.
- VulTegra distinguishes between scratch-trained and pre-trained models, highlighting scenario-dependent strengths and limitations in detection performance.
- It reveals that incorporating hidden code-level factors beyond CWE labels can significantly improve recall and F1 scores in vulnerability detection.
Searching arXiv for the cited papers to ground the article. VulTegra is an evaluation framework for deep-learning-based vulnerability detectors proposed in “It Only Gets Worse: Revisiting DL-Based Vulnerability Detectors from a Practical Perspective” (Wang et al., 13 Jul 2025). Rather than introducing a new detector, it is designed to examine whether current state-of-the-art detectors are reliable under practical deployment conditions. Its central premise is that vulnerability detection quality is multidimensional and cannot be reduced to a single benchmark score. To that end, VulTegra evaluates detectors along five research questions: consistency across different CWE categories, effectiveness on real-world vulnerabilities, scalability to newly emerged vulnerabilities, comparative behavior of scratch-trained and pre-trained-based models, and the impact of hidden code-level factors beyond CWE labels (Wang et al., 13 Jul 2025).
1. Conceptual scope and motivation
VulTegra was proposed to address several unresolved questions in the evaluation of deep-learning vulnerability detectors. The motivating concerns are whether detectors are consistent across the CWE types they claim to support, whether they operate effectively on real-world code rather than only curated datasets, whether they generalize to newly emerged vulnerabilities, whether pre-trained code models are truly better than models trained from scratch, and whether CWE labels alone are too coarse to explain detector behavior (Wang et al., 13 Jul 2025).
The framework explicitly distinguishes two model families. “Scratch-trained models” are trained from the ground up on task-specific vulnerability datasets, whereas “pre-trained-based models” are initialized from large-scale code pretraining and then fine-tuned for vulnerability detection (Wang et al., 13 Jul 2025). This distinction is central because VulTegra rejects the assumption that pre-trained models are uniformly superior. Instead, it frames model capability as scenario-dependent.
A second conceptual motivation is the critique of CWE-only evaluation. VulTegra argues that CWE is useful for taxonomy but insufficient as the sole organizing principle for dataset construction and evaluation. The paper’s motivating example contrasts two CWE-121 samples where only the data passing mechanism differs: one is detected correctly and the other is missed. This suggests that detectors are sensitive to latent code characteristics not encoded by CWE labels (Wang et al., 13 Jul 2025).
2. Framework structure and evaluated systems
At a high level, VulTegra organizes its study around five research questions:
- consistency across different CWE categories
- effectiveness on real-world vulnerabilities
- scalability to newly emerged vulnerabilities
- comparison between scratch-trained and pre-trained-based models
- impact of hidden code-level factors beyond CWE labels (Wang et al., 13 Jul 2025)
The framework’s stated objectives are to produce a “deep and systematic understanding” of state-of-the-art detector capability, compare scratch-trained and pre-trained-based detectors, reveal consistency, real-world effectiveness, and scalability limitations, and identify hidden factors that significantly influence performance (Wang et al., 13 Jul 2025).
VulTegra evaluates seven detectors, selected for public availability, reproducibility, and relevance. Four are scratch-trained: SySeVR, VulDeeLocator, DeepWuKong, and VulCNN. Three are pre-trained-based: LineVul, PILOT, and PDBert. Their reported architectural and granularity descriptions are RNN/function-level for SySeVR, Bi-LSTM/function-level for VulDeeLocator, GCN/function-level for DeepWuKong, CNN/function-level for VulCNN, Transformer/function/line-level for LineVul, and Transformer/function-level for both PILOT and PDBert (Wang et al., 13 Jul 2025). All seven were reproduced using their original code and datasets to ensure fairness.
This design positions VulTegra as an assessment framework rather than a detection pipeline. A plausible implication is that its main contribution lies in evaluation protocol design and comparative analysis, not in proposing a new detection architecture.
3. Data composition and experimental protocol
The dataset used in VulTegra has three components. Real-world data were drawn from NVD, Devign, Reveal, and Fan, then cleaned by deduplication, formatting, and removal of irrelevant samples. Synthetic data came from SARD, mainly to supplement coverage with explicit CWE labels. A third component, hidden-factor data, was constructed manually to test code-level factors beyond CWE (Wang et al., 13 Jul 2025).
The full dataset contains 31,673 vulnerability samples and 208,451 non-vulnerability samples, with 80% sourced from real-world scenarios (Wang et al., 13 Jul 2025). VulTegra then defines research-question-specific subsets. For RQ1, the subset contains 4,591 vulnerable and 6,732 non-vulnerable samples across 28 CWE categories. For RQ2, the real-world effectiveness subset contains 27,418 vulnerable and 203,317 non-vulnerable samples. For RQ3, the scalability experiment evaluates 20 vulnerabilities across 12 CWE categories and four popular open-source projects, each with more than 8,000 forks. For RQ5, the hidden-factor analysis uses 500 vulnerable and 500 non-vulnerable samples organized into five sub-datasets, although the paper also refers elsewhere to controlled modifications derived from 100 real-world vulnerable and 100 non-vulnerable samples; the phrasing is explicitly noted as somewhat inconsistent in the source description (Wang et al., 13 Jul 2025).
The framework uses four standard classification metrics: Accuracy, Precision, Recall, and F1 score (Wang et al., 13 Jul 2025). The source explicitly does not provide formal metric equations, loss functions, or statistical significance tests.
4. Consistency across CWE ranges
One of VulTegra’s most distinctive elements is its treatment of consistency within a detector’s declared vulnerability scope. Rather than relying on aggregate performance, the framework evaluates whether a detector maintains stable performance across the vulnerability classes it claims to handle (Wang et al., 13 Jul 2025).
A detector is considered consistent for a particular CWE if its accuracy exceeds 90% for that vulnerability type (Wang et al., 13 Jul 2025). This criterion is operationalized with scope-specific evaluation rules. If a detector claims more than 10 vulnerability types, VulTegra evaluates the top 10 most prevalent CWE types within its declared range. If it claims 10 or fewer types, all declared types are evaluated. If it makes no specific claim, the top 10 most common CWE types are evaluated. If there are too few samples for a top-10 CWE, the framework moves to the next CWE until enough samples exist. The ranking reference is CWE TOP25 (Wang et al., 13 Jul 2025).
Across models, results are reported for 28 CWEs: CWE-17, CWE-20, CWE-22, CWE-78, CWE-79, CWE-89, CWE-119, CWE-121, CWE-122, CWE-124, CWE-125, CWE-126, CWE-134, CWE-190, CWE-254, CWE-269, CWE-284, CWE-285, CWE-311, CWE-358, CWE-399, CWE-400, CWE-415, CWE-416, CWE-476, CWE-617, CWE-787, and CWE-862 (Wang et al., 13 Jul 2025).
Several per-CWE accuracies illustrate the heterogeneity of model behavior. On CWE-20, SySeVR, VulDeeLocator, and DeepWuKong achieve 0.790, 0.761, and 0.678 respectively, while PILOT and PDBert reach 0.900 and 0.929. On CWE-89, SySeVR and VulDeeLocator obtain 0.562 and 0.605, whereas PILOT and PDBert reach 0.913 and 0.928. On CWE-119, however, the pattern reverses: SySeVR, VulDeeLocator, and DeepWuKong achieve 0.928, 0.918, and 0.916, while PILOT and PDBert achieve 0.819 and 0.836 (Wang et al., 13 Jul 2025).
The main conclusion is that consistency is poor. SySeVR achieved accuracy above 90% for only 20% of its declared vulnerability types, while LineVul reached consistent detection on 70% of its declared types (Wang et al., 13 Jul 2025). More generally, consistency among the seven detectors remains below 80%, and some important CWEs remain difficult across the board. In particular, CWE-78 was not detected above 90% by any of the six models evaluated on it (Wang et al., 13 Jul 2025). VulTegra therefore argues that declared coverage and actual dependable coverage should not be conflated.
5. Real-world effectiveness and temporal scalability
VulTegra’s second and third evaluation dimensions examine deployability under realistic and temporally shifted conditions. For real-world effectiveness, all seven detectors were tested on the real-world subset using Precision, Recall, and F1-score, and tested values were compared with values reported in original papers when available (Wang et al., 13 Jul 2025).
The reported real-world results show substantial degradation relative to benchmark-style claims. PDBert had the highest tested precision, 0.59, and the highest tested F1, 0.56, while LineVul had the highest tested recall, 0.53 (Wang et al., 13 Jul 2025). At the same time, SySeVR dropped from reported precision/F1 of 0.90/0.92 to tested 0.32/0.38, and LineVul dropped from 0.97/0.86/0.91 to 0.56/0.53/0.54 (Wang et al., 13 Jul 2025). PILOT and PDBert were described as relatively closer to their reported performance, which the authors attribute to transferability from pretraining.
The framework’s interpretation is that all detectors remain suboptimal for real-world use (Wang et al., 13 Jul 2025). This assessment is grounded in the fact that even the best detector reaches only precision 0.59, recall 0.53, and F1 0.56 on the real-world dataset (Wang et al., 13 Jul 2025).
For scalability to newly emerged vulnerabilities, VulTegra evaluates whether detectors can identify vulnerabilities disclosed after model training assumptions. The setup collects 20 CVE vulnerabilities disclosed in 2024 from widely used open-source projects with more than 8,000 forks, covering 12 CWE categories. The metric is the number correctly detected out of 20 rather than Precision, Recall, or F1 (Wang et al., 13 Jul 2025).
Performance on this task is also poor. SySeVR achieves 10% accuracy, approximately 2 out of 20, while PDBert reaches 55%, approximately 11 out of 20 (Wang et al., 13 Jul 2025). PILOT is described as relatively higher as well, though no exact number is given in the provided material. The central conclusion is that none of the detectors performed well enough for practical deployment on new CVEs, although pre-trained models degraded less severely than scratch-trained ones (Wang et al., 13 Jul 2025).
6. Scratch-trained versus pre-trained-based models
The comparison between scratch-trained and pre-trained-based detectors is a central theme of VulTegra. The framework explicitly states that pre-trained models are not universally superior (Wang et al., 13 Jul 2025). Instead, the two model families exhibit different strengths across vulnerability classes and evaluation settings.
The per-CWE analysis provides the key evidence. Pre-trained models perform better on semantically rich vulnerabilities such as CWE-20 and CWE-89. On CWE-20, PDBert and PILOT reach 0.929 and 0.900, while scratch-trained models range from 0.678 to 0.790. On CWE-89, PDBert and PILOT obtain 0.928 and 0.913, while SySeVR and VulDeeLocator achieve 0.562 and 0.605 (Wang et al., 13 Jul 2025). By contrast, scratch-trained models perform better on more structured, pattern-driven vulnerabilities such as CWE-119, where SySeVR, VulDeeLocator, and DeepWuKong achieve 0.928, 0.918, and 0.916, compared with 0.819 and 0.836 for PILOT and PDBert (Wang et al., 13 Jul 2025).
At the family level, pre-trained-based models show better precision and F1 on real-world data, stronger stability under domain shift, and stronger scalability to new vulnerabilities. Scratch-trained models are better on structured, pattern-centric vulnerabilities and remain competitive or superior on memory-related or bounds-related patterns such as CWE-119 (Wang et al., 13 Jul 2025).
This comparative result is methodologically important because it contradicts a simple “pre-trained beats scratch-trained” narrative. VulTegra instead argues for distinct strengths and application scenarios. The source further suggests hybrid designs that combine model families through ensemble voting or layered architectures, using scratch-trained models for structured, pattern-heavy vulnerabilities and pre-trained-based models for semantic and context-heavy vulnerabilities (Wang et al., 13 Jul 2025). This suggests a research direction in which detector selection or composition is conditioned on vulnerability characteristics rather than architecture family alone.
7. Hidden factors beyond CWE and broader significance
VulTegra’s fifth research question examines hidden code-level factors that affect detector behavior beyond CWE labels. After manually analyzing over 500 samples, the authors identify four hidden factors: indirect assignment, external function calls, dynamic variable values, and control flow structures (Wang et al., 13 Jul 2025).
To test these factors, five sub-datasets were constructed: an Original/No-factor baseline; indirect memory allocations removed or indirect assignment simplified; external function calls removed; dynamic values replaced with static values; and control flow simplified (Wang et al., 13 Jul 2025). Experiments involved seven detectors, five independent tests per configuration, and the metrics Accuracy, Precision, Recall, and F1 (Wang et al., 13 Jul 2025).
The main finding is that modifying any single key factor improved recall for all seven detectors, and six of seven also achieved higher F1 (Wang et al., 13 Jul 2025). The largest gains were a 35% recall increase and a 13.8% F1 increase. The most dramatic example is LineVul under the setting where external function calls are removed: recall rises from 0.020 to 0.370, an increase of 0.35, and F1 rises from 0.575 to 0.713, an increase of 0.138 (Wang et al., 13 Jul 2025). The paper states that external function calls and dynamic variable values had a more substantial impact than indirect assignment and control flow simplification (Wang et al., 13 Jul 2025).
These results underpin VulTegra’s critique of relying only on CWE-based classification. The argument is not that CWE is useless, but that it is too coarse to capture the code properties that influence learnability and detection sensitivity (Wang et al., 13 Jul 2025). The framework therefore proposes that future work should consider both vulnerability types and inherent code features.
Within the broader research landscape, VulTegra is complementary to systems that address vulnerability identification in other modalities. For example, “VulRTex: A Reasoning-Guided Approach to Identify Vulnerabilities from Rich-Text Issue Report” (Jiang et al., 4 Sep 2025) addresses vulnerability-related issue-report identification and CWE-ID prediction from rich-text issue reports rather than source code. The juxtaposition is instructive: VulTegra interrogates the practical reliability of code-based deep-learning detectors, whereas VulRTex focuses on reasoning-guided triage over issue-report evidence bundles (Jiang et al., 4 Sep 2025). This suggests that practical vulnerability discovery and assessment may require multiple evaluation and evidence paradigms rather than a single benchmark-centered view.
Taken together, VulTegra’s reported results support three general conclusions. Current state-of-the-art detectors remain far from stable and reliable across their declared vulnerability scope; their real-world effectiveness is limited even when benchmark numbers appear strong; and their ability to scale to newly emerged vulnerabilities is poor (Wang et al., 13 Jul 2025). Its broader significance lies in reframing vulnerability detector evaluation as a multidimensional problem involving taxonomy, code structure, deployment realism, temporal shift, and latent code-level factors.