Just-in-Time Vulnerability Prediction

• Just-in-time Vulnerability Prediction is a commit-level binary classification task that predicts whether a commit introduces vulnerabilities before deployment.
• It integrates expert-crafted features with code-centric representations like AST-difference and graph-based models to capture structural changes effectively.
• Realistic dataset construction and evaluation reveal that while curated benchmarks achieve high scores, real-world imbalanced commit streams remain a significant challenge.

Searching arXiv for recent and foundational papers on just-in-time vulnerability prediction and closely related JIT defect prediction methodology. Just-in-time vulnerability prediction (JIT-VP) denotes commit-time prediction of whether a software change is vulnerability-inducing, so that risky changes can be triaged before deployment or close to merge rather than after release. Recent work formalizes the task as binary classification over commit information, $f(\mathcal{I}(c)) \in \{0,1\}$, where the positive class is a vulnerability-introducing commit and the negative class is a safe commit; other formulations model the transition from $S_{pre}$ to $S_{post}$ for a changed function, or a commit-triggered vulnerable-versus-benign function inside repository context (Nguyen et al., 14 Jul 2025, Aladics et al., 2023, Yildiz et al., 5 Mar 2025).

1. Problem formulations and task boundaries

The dominant formulation of JIT-VP is commit-level binary classification. In this setting, the instance is a commit, the prediction is made at commit time, and the objective is to identify vulnerability-introducing commits among incoming changes. One recent formalization defines the task directly as learning a mapping from commit information $\mathcal{I}(c)$ to a vulnerable/safe label (Nguyen et al., 14 Jul 2025). A closely related code-centric formulation uses the term dangerous commit for a commit whose changed code either directly contains vulnerable code or interacts with unchanged code so that the resulting merged system becomes vulnerable, and treats vulnerability-fixing commits that are not themselves vulnerability-contributing commits as safe (Nguyen et al., 2023).

The literature also contains narrower formulations. One line treats JIT-VP as prediction over the before and after states of changed functions, with the target defined as whether the transition from $S_{pre}$ to $S_{post}$ introduces vulnerability (Aladics et al., 2023). Another line, exemplified by the JitVul benchmark, is not standard stream-style commit classification: it defines $JitDetect:(\mathcal{R}, f)\rightarrow \{vul,ben\}$, where $\mathcal{R}$ is a repository snapshot and $f$ is a target function modified in a commit, then evaluates whether the model can distinguish the vulnerable function version from its patched benign counterpart (Yildiz et al., 5 Mar 2025).

Taken together, these works suggest three recurring task boundaries. First, commit-level JIT-VP asks whether a commit is vulnerability-inducing in the full commit stream. Second, change-conditioned function-level JIT-VP asks whether a particular changed function introduces vulnerability. Third, commit-triggered repository-context detection asks whether a changed function is vulnerable or benign when analyzed inside repository context. The distinctions are methodologically important because label construction, representation design, and evaluation difficulty change substantially across these formulations (Aladics et al., 2023, Yildiz et al., 5 Mar 2025).

2. Dataset construction and retrospective labeling

JIT-VP datasets are built retrospectively because vulnerability introduction is usually inferred only after a later fix, advisory, or disclosure. One representative JIT-VP dataset construction pipeline starts from a vulnerability fixing commit (VFC) database, applies SZZUnleashed to obtain candidate vulnerability-introducing commits, then ranks those candidates by a relevance score and selects the top 2 best fit VICs for each VFC. Using a manually curated Java VFC source dataset linked to the National Vulnerability Database, this process yielded 564 VFC entries across 198 open-source projects and a function-level prediction dataset of 5934 vulnerable methods and 29,670 non-vulnerable methods, for 35,604 methods in total (Aladics et al., 2023).

A different benchmarking strategy begins from curated CVEs rather than from a previously assembled VFC database. JitVul selects 879 CVE entries spanning 91 CWEs, extracts the vulnerable function version $f_{vul}$ and patched benign function version $S_{pre}$0, then approximates the vulnerability-introducing commit by traversing the target function’s history backward until the vulnerable version was last introduced. The resulting benchmark contains 1,758 pairwise data samples, balanced as 879 vulnerable and 879 benign, and is explicitly organized around paired vulnerable-versus-fixed evaluation rather than around full-stream commit prediction (Yildiz et al., 5 Mar 2025).

The strongest challenge to earlier evaluation practice comes from realistic dataset construction that keeps not only vulnerability-introducing commits and vulnerability-fixing commits, but also vulnerability-neutral commits (VNCs). A large-scale dataset of this kind was constructed from FFmpeg and the Linux kernel, yielding 1,081,882 commits in total: 9,715 VIC, 8,996 VFC, and 1,069,933 VNC. Labels were built by identifying VFCs from curated vulnerability sources, tracing VICs with V-SZZ, and, for Linux, supplementing that process with developer-informed Fixes: references in commit messages; all remaining filtered commits were labeled VNC (Nguyen et al., 14 Jul 2025).

The same realistic perspective is operationalized in VulGuard, which automates patch-commit identification, SZZ-style tracing, and commit extraction. VulGuard supports regex-assisted identification of vulnerability-fixing commits, optional manual patch lists in JSONL, and multiple SZZ variants including V-SZZ, B-SZZ, AG-SZZ, and MA-SZZ (Nguyen et al., 22 Jul 2025). Across all of these dataset-construction lines, label noise remains a central limitation: SZZ-derived tracing is approximate, vulnerability-fixing commits are not always explicit, and unlabeled commits are treated as neutral based on current knowledge rather than on proof of safety (Nguyen et al., 14 Jul 2025, Nguyen et al., 22 Jul 2025).

3. Representations and model families

A large portion of JIT-VP inherits the expert-feature tradition from JIT defect prediction. In that lineage, commit risk is represented through process and change metrics such as NS, ND, NF, Entropy, LA, LD, LT, FIX, NDEV, AGE, NUC, EXP, REXP, and SEXP (Nguyen et al., 14 Jul 2025). Related defect-prediction studies show that these features are useful but fragile: the Kamei-style datasets contain naming inconsistencies such as NM for ND, PD for AGE, and NPT for NUC; LA and LD can mix normalized and raw values when LT = 0; log transformation can introduce log(0) problems; and the feature space lacks direct code context (Ng et al., 2021). This critique is directly relevant to JIT-VP because vulnerability prediction inherits the same preprocessing and feature-semantic risks when it reuses defect-oriented tabular pipelines.

A second family replaces commit metrics with structural code-change representations. The Code Change Tree approach represents a change by computing root-to-leaf paths for the before and after ASTs, removing paths shared by both versions, reconstructing trees from the remaining differences, flattening them, and embedding them with Doc2Vec. On its Java vulnerability-introducing dataset, the average F1-scores were 32.92 for metrics, 40.64 for a simple full-AST before/after representation, and 42.34 for Code Change Tree; average representation size dropped from 174 nodes to 51 nodes (Aladics et al., 2023). The methodological point is that a vulnerability-introducing change is about the structured difference between two program states, not merely about aggregate churn.

A third family is explicitly code-centric and semantics-aware. CodeJIT represents a commit by a Code Transformation Graph (CTG) constructed from old and new relational code graphs with two relation types, structure and dependency, and annotates nodes and edges as added, deleted, or unchanged. The model uses a relational GNN to classify the resulting graph as dangerous or safe (Nguyen et al., 2023). Empirically, this representation yielded strong results on 20,274 commits from 506 real-world C/C++ projects: in the development-process setting, Precision 0.78, Recall 0.70, F1 0.74, Accuracy 0.85; in the cross-project setting, Precision 0.90, Recall 0.80, F1 0.84, Accuracy 0.89 (Nguyen et al., 2023). A crucial ablation showed that keeping related unchanged code substantially matters: on the cross-project setting, the full CTG reached Precision 0.95, Recall 0.70, F1 0.80, Accuracy 0.87, whereas a changed-only graph fell to 0.79, 0.61, 0.69, 0.79 (Nguyen et al., 2023).

Adjacent JIT defect work expands the design space further. SimCom++ combines a Random Forest over 14 expert-crafted features with a textCNN-based semantic model over commit messages and structured code diffs, using GMF early fusion and average late fusion; its average performance was AUC-ROC 0.781, AUC-PR 0.523, F1 0.528, above either component alone (Zhou et al., 2024). IRJIT takes a different direction: it indexes added-line code changes in Elasticsearch, queries new commits with more_like_this under BM25, and classifies by KNN majority vote over retrieved historical changes, yielding a method that is online and explainable at both commit and line level and is reported as up to 23 times faster than JITLine (Sahar et al., 2022). These are defect-prediction results rather than vulnerability results, but they indicate two transferable modeling patterns for JIT-VP: fusion of expert and semantic views, and retrieval-based online prediction.

4. Time, online learning, and granularity

Temporal design is a core methodological issue in JIT prediction because training data are older commits and prediction targets are future commits. A chronology study on JIT defect prediction tested Short-Period Models (SPM) trained on a single recent six-month period and Long-Period Models (LPM) trained on all periods up to time $S_{pre}$1, then evaluated them on later periods. Its central result was that predictive power did not materially deteriorate with older data, and that models using all available data could outperform models using only recent data; at the same time, feature-family importance scores fluctuated over time, especially for “Expertise of the Developer” and “Size,” motivating constant retraining and a recency-weighted sampling rule $S_{pre}$2 (Jahanshahi et al., 2021). For JIT-VP, this is direct methodological evidence about chronology, recency, drift, and retraining, even though the labels in that study were defect-inducing rather than vulnerability-inducing.

Online learning extends this temporal perspective to streaming deployment. In HITL O-JIT-SDP, incoming commits are first tested, then later trained upon in a prequential process, with predicted-positive commits routed to human inspection and predicted-negative commits labeled after a waiting window through bug-fixing evidence. The study recommends $S_{pre}$3 days and $S_{pre}$4 days, uses a 10-fold distributed bootstrap-validation framework with the Wilcoxon signed-rank test for continuous comparison, and reports that human-in-the-loop feedback improves both evaluation validity and predictive performance across 10 projects (Liu et al., 2023). This is defect prediction, but the design transfers naturally to JIT-VP because vulnerability labels are even more delayed and often require security-review feedback.

Granularity is another persistent issue. Commit-level JIT prediction is operationally attractive, but it can remain too coarse for inspection. A fine-grained JIT defect study showed that bug-inducing commits modify, on average, 7.97 files in Trac, 9.25 files in Qt, and 7.19 files in OpenStack, and argued that commit-level alarms still leave substantial manual search burden because only part of a multi-file commit may actually be defect-inducing (Ng et al., 2021). JitVul reaches a related conclusion from the opposite direction: even when the target function is already localized and paired with its fix, distinguishing the vulnerable version from the patched version remains difficult, and static dependency augmentation does not consistently help, whereas adaptive ReAct-style context acquisition yields better pairwise discrimination (Yildiz et al., 5 Mar 2025). This suggests that practical JIT-VP may need commit-level triage combined with finer-grained localization or repository-context reasoning.

5. Evaluation realism, benchmark design, and recurrent misconceptions

The most consequential recent development in JIT-VP evaluation is the shift from idealized datasets to realistic commit streams. Under the idealized setting, training and testing include only VICs and VFCs, which yields apparently strong results: on FFmpeg, the average across eight models was PR-AUC 0.833, MCC 0.651, F1 0.738, ROC-AUC 0.884; on Linux, PR-AUC 0.805, MCC 0.568, F1 0.701, ROC-AUC 0.853 (Nguyen et al., 14 Jul 2025). When those same models are trained on idealized data but tested on realistic data that also include VNCs, average PR-AUC drops from 0.833 to 0.076 on FFmpeg and from 0.805 to 0.017 on Linux. Even when models are trained directly on realistic data, average PR-AUC remains only 0.091 on FFmpeg and 0.016 on Linux, and standard remedies such as random under-sampling, random over-sampling, SMOTE, OSS, and focal loss are reported as ineffective overall (Nguyen et al., 14 Jul 2025).

This result corrects a frequent misunderstanding: high scores on curated VIC-versus-VFC benchmarks do not imply deployment readiness. The realistic setting changes the task from separating vulnerability introducers from vulnerability fixers to separating vulnerability introducers from the full universe of safe commits, where VNCs account for more than 90% of all commits in both benchmark projects (Nguyen et al., 14 Jul 2025). A related metric misconception appears in the JitVul benchmark: the best F1 on GPT-4o-mini was 66.05 for Dependency-Augmented LLM with CoT, but the best pairwise accuracy $S_{pre}$5 was 20.17 for ReAct with few-shot prompting; the benchmark therefore states that a higher F1 score does not necessarily indicate better vulnerability characterization (Yildiz et al., 5 Mar 2025).

Benchmark design is also changing in tooling. VulGuard standardizes Accuracy, Precision, Recall, F1-score, MCC, ROC-AUC, PR-AUC, and effort-aware measures Recall@20, Effort@20, and P-opt, using chronological 75/5/20 splitting by default (Nguyen et al., 22 Jul 2025). In adjacent polyglot defect prediction, a multi-project training regime improved average cross-project AUC from 62.05% to 66.18%, which suggests that pooled training can help when project-specific labels are sparse (Li et al., 2024). That result is not vulnerability-specific, but it is relevant because JIT-VP datasets are typically label-starved and cross-project transfer is often unavoidable.

6. Tooling, operationalization, and open research directions

The main systems contribution in the current literature is VulGuard, an end-to-end framework that unifies commit mining, feature extraction, vulnerability-fix identification, SZZ-based annotation, graph construction, model training, evaluation, and inference. Its Extraction Module covers commit collection, feature extraction, commit annotation, and data splitting; its Evaluation Module integrates eight models—VCCFinder, CodeJIT, LR, LAPredict, TLEL, DeepJIT, SimCom, and JITFine—under one interface (Nguyen et al., 22 Jul 2025). VulGuard supports C/C++, Java, JavaScript, and Python, filters merge, whitespace-only, and comment-only commits, serializes data in JSONL, and is distributed as both a Python library and a CLI. With 50 concurrent processes, extraction took approximately 1 hour for FFmpeg and 12 hours for Linux (Nguyen et al., 22 Jul 2025).

This tooling perspective aligns with the broader methodological shift in the field. Realistic evaluation has shown that current JIT-VP is not yet mature enough for confident deployment on large, imbalanced commit streams (Nguyen et al., 14 Jul 2025). At the same time, the model-design literature has shown that there is no single dominant representation: tabular expert features remain strong baselines, AST-difference encodings improve over metrics for function-level change classification, graph-based semantic change models can achieve strong cross-project results, and online retrieval or human-in-the-loop updates offer deployment-oriented alternatives (Aladics et al., 2023, Nguyen et al., 2023, Sahar et al., 2022, Liu et al., 2023).

Several open problems recur across the literature. Label fidelity remains difficult because VIC identification depends on VFC tracing, SZZ variants, and often manual supplementation (Nguyen et al., 14 Jul 2025, Nguyen et al., 22 Jul 2025). Temporal validity remains essential because random splits or non-chronological protocols can misrepresent deployment conditions (Nguyen et al., 14 Jul 2025, Jahanshahi et al., 2021). Security-specific semantics are still underrepresented in many commit-level models; even strong tabular and code-centric systems remain largely language-, project-, or representation-specific (Nguyen et al., 2023, Ng et al., 2021). Granularity remains unresolved, because commit-level warnings are actionable for triage but often insufficient for localization (Ng et al., 2021, Yildiz et al., 5 Mar 2025). Finally, polyglot and cross-project generalization are increasingly important as systems span multiple languages and within-project vulnerability labels remain sparse (Li et al., 2024).

JIT-VP is therefore best understood as a family of temporally ordered, retrospectively labeled, commit-triggered security prediction problems rather than as a single settled benchmark. The field contains stable methodological cores—commit-level instances, historical traceability, chronology-aware evaluation, and class-imbalance sensitivity—but the current evidence also shows that representation choice, dataset realism, and deployment assumptions dominate conclusions about model quality (Nguyen et al., 14 Jul 2025, Nguyen et al., 22 Jul 2025).