---
title: Just-in-Time Vulnerability Prediction
url: https://www.emergentmind.com/topics/just-in-time-vulnerability-prediction-jit-vp
type: topic
---

# Just-in-Time Vulnerability Prediction

Searching arXiv for recent and foundational papers on just-in-time vulnerability prediction and closely related JIT defect prediction methodology.
Just-in-time vulnerability prediction (JIT-VP) denotes commit-time prediction of whether a software change is vulnerability-inducing, so that risky changes can be triaged before deployment or close to merge rather than after release. Recent work formalizes the task as binary classification over commit information, \(f(\mathcal{I}(c)) \in \{0,1\}\), where the positive class is a vulnerability-introducing commit and the negative class is a safe commit; other formulations model the transition from \(S_{pre}\) to \(S_{post}\) for a changed function, or a commit-triggered vulnerable-versus-benign function inside repository context [2507.10729][2303.16591][2503.03586].

## 1. Problem formulations and task boundaries

The dominant formulation of JIT-VP is **commit-level binary classification**. In this setting, the instance is a commit, the prediction is made at commit time, and the objective is to identify vulnerability-introducing commits among incoming changes. One recent formalization defines the task directly as learning a mapping from commit information \(\mathcal{I}(c)\) to a vulnerable/safe label [2507.10729]. A closely related code-centric formulation uses the term **dangerous commit** for a commit whose changed code either directly contains vulnerable code or interacts with unchanged code so that the resulting merged system becomes vulnerable, and treats vulnerability-fixing commits that are not themselves vulnerability-contributing commits as safe [2304.08396].

The literature also contains narrower formulations. One line treats JIT-VP as prediction over the **before** and **after** states of changed functions, with the target defined as whether the transition from \(S_{pre}\) to \(S_{post}\) introduces vulnerability [2303.16591]. Another line, exemplified by the JitVul benchmark, is not standard stream-style commit classification: it defines \(JitDetect:(\mathcal{R}, f)\rightarrow \{vul,ben\}\), where \(\mathcal{R}\) is a repository snapshot and \(f\) is a target function modified in a commit, then evaluates whether the model can distinguish the vulnerable function version from its patched benign counterpart [2503.03586].

Taken together, these works suggest three recurring task boundaries. First, **commit-level JIT-VP** asks whether a commit is vulnerability-inducing in the full commit stream. Second, **change-conditioned function-level JIT-VP** asks whether a particular changed function introduces vulnerability. Third, **commit-triggered repository-context detection** asks whether a changed function is vulnerable or benign when analyzed inside repository context. The distinctions are methodologically important because label construction, representation design, and evaluation difficulty change substantially across these formulations [2303.16591][2503.03586].

## 2. Dataset construction and retrospective labeling

JIT-VP datasets are built retrospectively because vulnerability introduction is usually inferred only after a later fix, advisory, or disclosure. One representative JIT-VP dataset construction pipeline starts from a **vulnerability fixing commit (VFC)** database, applies **SZZUnleashed** to obtain candidate vulnerability-introducing commits, then ranks those candidates by a relevance score and selects the **top 2 best fit VICs** for each VFC. Using a manually curated Java VFC source dataset linked to the National Vulnerability Database, this process yielded **564 VFC entries** across **198 open-source projects** and a function-level prediction dataset of **5934 vulnerable methods** and **29,670 non-vulnerable methods**, for **35,604 methods** in total [2303.16591].

A different benchmarking strategy begins from curated CVEs rather than from a previously assembled VFC database. JitVul selects **879 CVE entries** spanning **91 CWEs**, extracts the vulnerable function version \(f_{vul}\) and patched benign function version \(f_{ben}\), then approximates the vulnerability-introducing commit by traversing the target function’s history backward until the vulnerable version was last introduced. The resulting benchmark contains **1,758 pairwise data samples**, balanced as **879 vulnerable** and **879 benign**, and is explicitly organized around paired vulnerable-versus-fixed evaluation rather than around full-stream commit prediction [2503.03586].

The strongest challenge to earlier evaluation practice comes from realistic dataset construction that keeps not only vulnerability-introducing commits and vulnerability-fixing commits, but also **vulnerability-neutral commits (VNCs)**. A large-scale dataset of this kind was constructed from **FFmpeg** and the **Linux kernel**, yielding **1,081,882 commits** in total: **9,715 VIC**, **8,996 VFC**, and **1,069,933 VNC**. Labels were built by identifying VFCs from curated vulnerability sources, tracing VICs with **V-SZZ**, and, for Linux, supplementing that process with developer-informed `Fixes:` references in commit messages; all remaining filtered commits were labeled VNC [2507.10729].

The same realistic perspective is operationalized in **VulGuard**, which automates patch-commit identification, SZZ-style tracing, and commit extraction. VulGuard supports regex-assisted identification of vulnerability-fixing commits, optional manual patch lists in JSONL, and multiple SZZ variants including **V-SZZ**, **B-SZZ**, **AG-SZZ**, and **MA-SZZ** [2507.16685]. Across all of these dataset-construction lines, label noise remains a central limitation: SZZ-derived tracing is approximate, vulnerability-fixing commits are not always explicit, and unlabeled commits are treated as neutral based on current knowledge rather than on proof of safety [2507.10729][2507.16685].

## 3. Representations and model families

A large portion of JIT-VP inherits the **expert-feature** tradition from JIT defect prediction. In that lineage, commit risk is represented through process and change metrics such as **NS**, **ND**, **NF**, **Entropy**, **LA**, **LD**, **LT**, **FIX**, **NDEV**, **AGE**, **NUC**, **EXP**, **REXP**, and **SEXP** [2507.10729]. Related defect-prediction studies show that these features are useful but fragile: the Kamei-style datasets contain naming inconsistencies such as `NM` for `ND`, `PD` for `AGE`, and `NPT` for `NUC`; `LA` and `LD` can mix normalized and raw values when `LT = 0`; log transformation can introduce `log(0)` problems; and the feature space lacks direct code context [2109.13634]. This critique is directly relevant to JIT-VP because vulnerability prediction inherits the same preprocessing and feature-semantic risks when it reuses defect-oriented tabular pipelines.

A second family replaces commit metrics with **structural code-change representations**. The **Code Change Tree** approach represents a change by computing root-to-leaf paths for the before and after ASTs, removing paths shared by both versions, reconstructing trees from the remaining differences, flattening them, and embedding them with **Doc2Vec**. On its Java vulnerability-introducing dataset, the average F1-scores were **32.92** for metrics, **40.64** for a simple full-AST before/after representation, and **42.34** for Code Change Tree; average representation size dropped from **174 nodes** to **51 nodes** [2303.16591]. The methodological point is that a vulnerability-introducing change is about the structured difference between two program states, not merely about aggregate churn.

A third family is explicitly **code-centric and semantics-aware**. CodeJIT represents a commit by a **Code Transformation Graph (CTG)** constructed from old and new relational code graphs with two relation types, **structure** and **dependency**, and annotates nodes and edges as **added**, **deleted**, or **unchanged**. The model uses a relational GNN to classify the resulting graph as dangerous or safe [2304.08396]. Empirically, this representation yielded strong results on **20,274 commits** from **506 real-world C/C++ projects**: in the development-process setting, **Precision 0.78**, **Recall 0.70**, **F1 0.74**, **Accuracy 0.85**; in the cross-project setting, **Precision 0.90**, **Recall 0.80**, **F1 0.84**, **Accuracy 0.89** [2304.08396]. A crucial ablation showed that keeping **related unchanged code** substantially matters: on the cross-project setting, the full CTG reached **Precision 0.95**, **Recall 0.70**, **F1 0.80**, **Accuracy 0.87**, whereas a changed-only graph fell to **0.79**, **0.61**, **0.69**, **0.79** [2304.08396].

Adjacent JIT defect work expands the design space further. **SimCom++** combines a Random Forest over 14 expert-crafted features with a textCNN-based semantic model over commit messages and structured code diffs, using **GMF** early fusion and average late fusion; its average performance was **AUC-ROC 0.781**, **AUC-PR 0.523**, **F1 0.528**, above either component alone [2403.11079]. **IRJIT** takes a different direction: it indexes added-line code changes in Elasticsearch, queries new commits with **more_like_this** under **BM25**, and classifies by **KNN** majority vote over retrieved historical changes, yielding a method that is online and explainable at both commit and line level and is reported as up to **23 times faster** than JITLine [2210.02435]. These are defect-prediction results rather than vulnerability results, but they indicate two transferable modeling patterns for JIT-VP: fusion of expert and semantic views, and retrieval-based online prediction.

## 4. Time, online learning, and granularity

Temporal design is a core methodological issue in JIT prediction because training data are older commits and prediction targets are future commits. A chronology study on JIT defect prediction tested **Short-Period Models (SPM)** trained on a single recent six-month period and **Long-Period Models (LPM)** trained on all periods up to time \(i\), then evaluated them on later periods. Its central result was that predictive power did not materially deteriorate with older data, and that models using **all available data** could outperform models using only recent data; at the same time, feature-family importance scores fluctuated over time, especially for **“Expertise of the Developer”** and **“Size,”** motivating constant retraining and a recency-weighted sampling rule \(\left(1-\frac{i-k}{i}\right)\times 100\%\) [2103.03506]. For JIT-VP, this is direct methodological evidence about chronology, recency, drift, and retraining, even though the labels in that study were defect-inducing rather than vulnerability-inducing.

Online learning extends this temporal perspective to streaming deployment. In **HITL O-JIT-SDP**, incoming commits are first tested, then later trained upon in a prequential process, with predicted-positive commits routed to human inspection and predicted-negative commits labeled after a waiting window through bug-fixing evidence. The study recommends **\(W_{QA}=7\)** days and **\(W_{BFC}=15\)** days, uses a **10-fold distributed bootstrap-validation** framework with the **Wilcoxon signed-rank test** for continuous comparison, and reports that human-in-the-loop feedback improves both evaluation validity and predictive performance across 10 projects [2308.13707]. This is defect prediction, but the design transfers naturally to JIT-VP because vulnerability labels are even more delayed and often require security-review feedback.

Granularity is another persistent issue. Commit-level JIT prediction is operationally attractive, but it can remain too coarse for inspection. A fine-grained JIT defect study showed that bug-inducing commits modify, on average, **7.97 files** in Trac, **9.25 files** in Qt, and **7.19 files** in OpenStack, and argued that commit-level alarms still leave substantial manual search burden because only part of a multi-file commit may actually be defect-inducing [2110.00579]. JitVul reaches a related conclusion from the opposite direction: even when the target function is already localized and paired with its fix, distinguishing the vulnerable version from the patched version remains difficult, and static dependency augmentation does not consistently help, whereas adaptive **ReAct**-style context acquisition yields better pairwise discrimination [2503.03586]. This suggests that practical JIT-VP may need commit-level triage combined with finer-grained localization or repository-context reasoning.

## 5. Evaluation realism, benchmark design, and recurrent misconceptions

The most consequential recent development in JIT-VP evaluation is the shift from **idealized** datasets to **realistic** commit streams. Under the idealized setting, training and testing include only **VICs** and **VFCs**, which yields apparently strong results: on FFmpeg, the average across eight models was **PR-AUC 0.833**, **MCC 0.651**, **F1 0.738**, **ROC-AUC 0.884**; on Linux, **PR-AUC 0.805**, **MCC 0.568**, **F1 0.701**, **ROC-AUC 0.853** [2507.10729]. When those same models are trained on idealized data but tested on realistic data that also include **VNCs**, average PR-AUC drops from **0.833 to 0.076** on FFmpeg and from **0.805 to 0.017** on Linux. Even when models are trained directly on realistic data, average PR-AUC remains only **0.091** on FFmpeg and **0.016** on Linux, and standard remedies such as random under-sampling, random over-sampling, SMOTE, OSS, and focal loss are reported as ineffective overall [2507.10729].

This result corrects a frequent misunderstanding: high scores on curated VIC-versus-VFC benchmarks do not imply deployment readiness. The realistic setting changes the task from separating vulnerability introducers from vulnerability fixers to separating vulnerability introducers from the full universe of safe commits, where **VNCs account for more than 90% of all commits** in both benchmark projects [2507.10729]. A related metric misconception appears in the JitVul benchmark: the best **F1** on GPT-4o-mini was **66.05** for Dependency-Augmented LLM with CoT, but the best **pairwise accuracy** \(pAcc\) was **20.17** for ReAct with few-shot prompting; the benchmark therefore states that a higher F1 score does not necessarily indicate better vulnerability characterization [2503.03586].

Benchmark design is also changing in tooling. **VulGuard** standardizes **Accuracy**, **Precision**, **Recall**, **F1-score**, **MCC**, **ROC-AUC**, **PR-AUC**, and effort-aware measures **Recall@20**, **Effort@20**, and **P-opt**, using chronological **75/5/20** splitting by default [2507.16685]. In adjacent polyglot defect prediction, a multi-project training regime improved average cross-project AUC from **62.05%** to **66.18%**, which suggests that pooled training can help when project-specific labels are sparse [2407.10906]. That result is not vulnerability-specific, but it is relevant because JIT-VP datasets are typically label-starved and cross-project transfer is often unavoidable.

## 6. Tooling, operationalization, and open research directions

The main systems contribution in the current literature is **VulGuard**, an end-to-end framework that unifies commit mining, feature extraction, vulnerability-fix identification, SZZ-based annotation, graph construction, model training, evaluation, and inference. Its **Extraction Module** covers commit collection, feature extraction, commit annotation, and data splitting; its **Evaluation Module** integrates eight models—**VCCFinder**, **CodeJIT**, **LR**, **LAPredict**, **TLEL**, **DeepJIT**, **SimCom**, and **JITFine**—under one interface [2507.16685]. VulGuard supports **C/C++**, **Java**, **JavaScript**, and **Python**, filters merge, whitespace-only, and comment-only commits, serializes data in JSONL, and is distributed as both a Python library and a CLI. With **50 concurrent processes**, extraction took approximately **1 hour** for FFmpeg and **12 hours** for Linux [2507.16685].

This tooling perspective aligns with the broader methodological shift in the field. Realistic evaluation has shown that current JIT-VP is not yet mature enough for confident deployment on large, imbalanced commit streams [2507.10729]. At the same time, the model-design literature has shown that there is no single dominant representation: tabular expert features remain strong baselines, AST-difference encodings improve over metrics for function-level change classification, graph-based semantic change models can achieve strong cross-project results, and online retrieval or human-in-the-loop updates offer deployment-oriented alternatives [2303.16591][2304.08396][2210.02435][2308.13707].

Several open problems recur across the literature. **Label fidelity** remains difficult because VIC identification depends on VFC tracing, SZZ variants, and often manual supplementation [2507.10729][2507.16685]. **Temporal validity** remains essential because random splits or non-chronological protocols can misrepresent deployment conditions [2507.10729][2103.03506]. **Security-specific semantics** are still underrepresented in many commit-level models; even strong tabular and code-centric systems remain largely language-, project-, or representation-specific [2304.08396][2109.13634]. **Granularity** remains unresolved, because commit-level warnings are actionable for triage but often insufficient for localization [2110.00579][2503.03586]. Finally, **polyglot and cross-project generalization** are increasingly important as systems span multiple languages and within-project vulnerability labels remain sparse [2407.10906].

JIT-VP is therefore best understood as a family of temporally ordered, retrospectively labeled, commit-triggered security prediction problems rather than as a single settled benchmark. The field contains stable methodological cores—commit-level instances, historical traceability, chronology-aware evaluation, and class-imbalance sensitivity—but the current evidence also shows that representation choice, dataset realism, and deployment assumptions dominate conclusions about model quality [2507.10729][2507.16685].

Source: https://www.emergentmind.com/topics/just-in-time-vulnerability-prediction-jit-vp