VulGuard: Unified JIT Vulnerability Prediction

• VulGuard is an integrated toolchain for just-in-time vulnerability prediction that automates mining, labeling, and evaluation of Git commits.
• It combines parallelized extraction methods with SZZ-family algorithms to compute expert features across multiple programming languages.
• It standardizes model evaluation under realistic and idealized settings, enhancing reproducibility and operational CI/CD integration.

VulGuard is an automated, end-to-end toolchain for Just-In-Time vulnerability prediction (JIT-VP) that mines, processes, and analyzes commit histories from Git repositories and trains and evaluates state-of-the-art JIT vulnerability prediction models within one framework. It combines repository-scale data extraction, vulnerability labeling via SZZ-family algorithms, feature and graph generation, standardized evaluation, and CI/CD-oriented inference. In the reported implementation, it is delivered as a Python package with a command-line interface, supports C/C++, Java, JavaScript, and Python, and is demonstrated on FFmpeg and the Linux kernel (Nguyen et al., 22 Jul 2025).

1. Problem setting and scope

VulGuard is motivated by several recurrent limitations in JIT-VP research. JIT-VP studies often build bespoke, repository-specific pipelines, which leads to inconsistent evaluation and limited comparability across models and datasets. Mining large repositories at commit granularity is computationally intensive, particularly because git blame and git [diff](https://www.emergentmind.com/topics/differential-transformer-diff) operations become bottlenecks. Data curation is also nontrivial: vulnerability-fixing commits must be identified and then traced back to vulnerability-inducing commits, a process that is time-consuming and noisy. Existing work frequently reuses JIT defect prediction pipelines, which are not tailored for vulnerabilities. VulGuard is positioned as a unified, extensible response to these reproducibility, scalability, and deployment problems (Nguyen et al., 22 Jul 2025).

Within that scope, the system supports repository-scale mining with parallelized git diff and git blame; fine-grained commit artifacts such as messages, code changes, and blame metadata; optional code property graphs; integrated implementations of eight JIT-VP and JIT-DP baselines; and a realistic evaluation setting that includes vulnerability-related and unrelated commits. The paper also emphasizes operational use: VulGuard can be embedded into CI/CD so that trained models can score new commits and provide just-in-time risk signals.

2. End-to-end architecture

The architecture is organized as a pipeline with four layers: data ingestion, extraction, model execution, and evaluation/reporting. The workflow begins by pointing VulGuard to a local Git repository. The extraction module then clones or opens the repository, filters out merge, whitespace-only, and comment-only commits, and retains only source files in the project’s primary language, specifically .c/.h, .cpp, .java, .js, or .py. Fine-grained changes are obtained with git diff, and line-level authorship is recovered with git blame; both are parallelized because independent calls can be executed concurrently (Nguyen et al., 22 Jul 2025).

Commit messages are extracted verbatim. Software engineering metrics reflecting developer activity, code quality, and temporal patterns are computed following prior JIT literature such as Kamei et al. and VCCFinder, with the feature set configurable inside VulGuard. Output is serialized as .jsonl. For graph-based approaches, VulGuard can optionally generate code property graphs using Joern.

Commit annotation is a distinct stage. Vulnerability-fixing commits are first identified with regex filters based on Zhou et al., optionally augmented by a user-provided list of known patch commits. VulGuard then traces from fixing commits to vulnerability-inducing commits through V-SZZ by default, while B-SZZ, AG-SZZ, and MA-SZZ are also available. After labeling, the data is split in time order to avoid leakage and to simulate continuous development; the default split is 75/5/20% for train/validation/test. The model layer formats expert features, commit messages, diff snippets, or property graphs according to the requirements of each integrated model. The evaluation layer then computes threshold-dependent metrics, threshold-independent metrics, and effort-aware metrics, and persists predictions and metrics to disk.

3. Data model, labeling, and corpus scale

VulGuard’s positive class is the vulnerability-introducing commit, while the negative class is all other commits. This makes the labeling problem stricter than patch classification: the objective is not merely to find fixes, but to identify the original commits that introduced vulnerable lines. The paper recommends complementing regex-based identification of fixing commits with a curated list, because regex filters can introduce noise. V-SZZ is preferred for vulnerability contexts, but the availability of multiple SZZ variants makes the labeling stage explicitly configurable (Nguyen et al., 22 Jul 2025).

The reported corpus consists of FFmpeg and Linux kernel master branches, with commits collected as of 2024-09-24. After filtering, the total is 1,081,882 commits across both projects. Extraction used 50 concurrent processes and required approximately 1 hour for FFmpeg and approximately 12 hours for Linux. The resulting datasets are highly imbalanced, especially for Linux, where neutral commits dominate.

Repository	Train (VIC / VFC / VNC)	Val / Test (VIC / VFC / VNC)
FFmpeg	3,826 / 2,519 / 39,823	Val: 255 / 330 / 3,242; Test: 1,020 / 1,903 / 37,778
Linux kernel	3,461 / 1,735 / 796,965	Val: 231 / 616 / 35,086; Test: 922 / 1,691 / 157,039

Across both repositories and all splits, the totals are 9,715 VIC, 8,996 VFC, and 1,069,933 VNC. This class distribution is central to the paper’s framing: performance in controlled settings can differ sharply from performance under realistic imbalance, and VulGuard is designed to make that difference explicit rather than hide it behind patch-centric sampling.

4. Integrated model ecosystem and evaluation harness

VulGuard integrates eight baselines behind a uniform data and execution interface: VCCFinder, CodeJIT, LR-JIT, LAPredict, TLEL, DeepJIT, SimCom, and JITFine. These models span classical machine learning, graph-based learning, deep learning, and ensemble learning. VCCFinder consumes expert features and commit messages. CodeJIT uses code property graphs and is code-centric. LR-JIT, TLEL, and LAPredict are expert-feature baselines. DeepJIT consumes commit messages and code changes. SimCom integrates expert features, commit messages, and code changes. JITFine combines semantic features with expert features and commit changes/messages. VulGuard standardizes feature and graph formatting so that these heterogeneous models can be trained and compared under a consistent CLI/API (Nguyen et al., 22 Jul 2025).

The evaluation harness computes threshold-dependent metrics including Accuracy, Precision, Recall, F1, and Matthews Correlation Coefficient; threshold-independent metrics including ROC-AUC and PR-AUC; and effort-aware metrics including Recall@20, Effort@20, and P-opt. The paper provides explicit formulas for Precision, Recall, Accuracy, and MCC, while effort-aware metrics are described conceptually as functions of the percentage of code inspected rather than through formal equations. This evaluation design is methodologically significant because it enables comparison across models with different input modalities while preserving a common test protocol.

A notable feature is the presence of two evaluation settings. The idealized setting includes only vulnerability-inducing commits and their corresponding fixing commits. The realistic setting includes unrelated neutral commits as well. The latter is the more operationally relevant benchmark, because real development streams contain overwhelmingly many non-security commits. VulGuard’s contribution is not merely that it hosts models, but that it enforces a common, time-ordered, leakage-aware, and imbalance-aware evaluation regime.

5. Empirical performance on FFmpeg and Linux

In the idealized setting, VulGuard reports strong results for several integrated models. On FFmpeg, JITFine achieves the best PR-AUC at 0.959, the best MCC at 0.864, the best F1 at 0.909, and the best ROC-AUC at 0.980. SimCom, DeepJIT, and VCCFinder also perform strongly, but remain below JITFine on the reported best values. On Linux, the picture is slightly more mixed: SimCom achieves the best PR-AUC at 0.892, while JITFine achieves the best MCC at 0.716 and the best F1 at 0.818; ROC-AUC is reported as 0.915 for JITFine and 0.913 for SimCom (Nguyen et al., 22 Jul 2025).

The realistic setting produces a marked collapse in performance across all models. On FFmpeg, SimCom becomes the strongest model with PR-AUC 0.134, MCC 0.226, F1 0.231, and ROC-AUC 0.809. On Linux, the best PR-AUC is only 0.031 for SimCom; the best MCC is 0.081 for TLEL; the best F1 is 0.039 for LR; and the best ROC-AUC is 0.787 for TLEL. DeepJIT and JITFine, which are strong in the idealized setting, drop sharply in the realistic Linux scenario.

This performance gap is one of VulGuard’s central empirical findings. The paper explicitly states that all models degrade markedly when moving from idealized to realistic settings, particularly on PR-AUC, MCC, and F1. The implication is methodological rather than merely numerical: benchmark design strongly shapes conclusions about model quality. VulGuard therefore functions as a corrective to evaluation regimes that exclude neutral commits and thereby overstate practical effectiveness.

6. Reproducibility, CI/CD use, and limitations

VulGuard is released as a public GitHub project with examples and manual at https://github.com/AI4Code-HUST/VulGuard, and the paper describes it as a unified Python package with CLI support (Nguyen et al., 22 Jul 2025). The intended operational workflow includes mining a local repository, training a selected model, evaluating it, and then using inference on new commits inside CI/CD. Teams can set a probability threshold and use inference outputs to block merges or require security review, although the paper does not prescribe specific thresholds or policies.

From an implementation perspective, VulGuard expects Linux-based systems, recommends GPU acceleration for deep models, and stores datasets and intermediate artifacts in .jsonl and model-specific formats. Inputs include the local Git repository, the project’s primary language, and optionally a custom list of known fixing commits. Outputs include commit records with messages, diffs, blame metadata, derived features, labels via V-SZZ or another SZZ variant, predictions, metrics, and saved model artifacts.

The limitations identified in the paper are substantial. Regex-based fixing-commit identification can include false positives, and SZZ-family tracing can yield different labels depending on the selected variant. Class imbalance remains severe, especially for Linux, and materially depresses PR-AUC, MCC, and F1 in realistic settings. Generalizability is not established beyond FFmpeg and Linux, and cross-project generalization is not assessed. The effort-aware metrics are implemented, but the exact unit of “percentage of code inspected” is not formalized. The paper also does not report statistical significance tests or ablations. These constraints mean that VulGuard standardizes evaluation more than it resolves the underlying epistemic uncertainty in vulnerability labeling.

7. Broader reinterpretations of the VulGuard concept

Although the paper titled “VulGuard” refers specifically to the JIT-VP tool, adjacent work uses the name as a broader architectural target. A technical synthesis of “VulRTex” describes it as suitable for a “VulGuard-style OSS vulnerability guard,” emphasizing automated identification of vulnerability-related issue reports under severe imbalance, rich-text reasoning over screenshots and code, CWE-ID prediction, and continuous monitoring of GitHub issue reports (Jiang et al., 4 Sep 2025). A separate synthesis of “VibeGuard” presents a “VulGuard” system focused on guarding AI-generated code before publication, centered on a pre-publish security gate for artifact hygiene, packaging-configuration drift, source-map exposure, hardcoded secrets, and supply-chain risk (Xie, 1 Apr 2026). A further synthesis maps “JavelinGuard” onto a system like “VulGuard” as an LLM guardrail for malicious-intent detection, including prompt injection, jailbreaks, and vulnerability-oriented prompts, with low-latency encoder-based architectures (Datta et al., 9 Jun 2025).

These uses are not identical to the original JIT-VP framework. Rather, they treat VulGuard as a reusable deployment concept spanning commit-risk prediction, issue-report triage, artifact-boundary inspection, and LLM interaction security. This suggests that the term has begun to function as a broader systems label for operational vulnerability guarding, even though its primary published referent remains the unified JIT-VP evaluation tool introduced in 2025 (Nguyen et al., 22 Jul 2025).