---
title: JIT-Defects4J Dataset
url: https://www.emergentmind.com/topics/jit-defects4j-dataset
type: topic
---

# JIT-Defects4J Dataset

Searching arXiv for recent papers on JIT-Defects4J and related Defects4J/JIT defect prediction work.
The **JIT-Defects4J dataset** is a just-in-time defect prediction corpus derived from the Defects4J ecosystem and used for commit-level and line-level analysis of bug-inducing changes in Java projects. In the account currently available from the literature, it is described most concretely in work on refactoring-aware relabeling, where it serves as the core benchmark for just-in-time defect prediction under commit tangling and refactoring effects [2507.19714]. Its construction links manually identified bug-fixing lines to earlier bug-inducing lines and commits through `git blame`, while also exposing classical code-change metrics and semantic artifacts such as commit messages and diffs [2507.19714]. The dataset occupies a distinctive position between traditional SZZ-based JIT corpora, which emphasize process metrics, and Defects4J-style benchmarks, which emphasize curated bug/fix pairs and executable tests. This suggests that JIT-Defects4J functions as a bridge between defect prediction, defect localization, and program-repair-oriented analysis.

## 1. Provenance and conceptual position

JIT-Defects4J is rooted in the Defects4J project family rather than in the earlier Kamei-style just-in-time datasets. The base projects are Apache Java projects derived from the Defects4J ecosystem, and the dataset construction is explicitly described as a two-stage process involving **LLTC4J** and then **JIT-Defects4J** [2507.19714]. LLTC4J, attributed to Herbold et al., focuses on bug-fixing commits only and manually labels changed lines in those commits as “bug-fixing line” versus other non-bug-fixing changes in the same commit. JIT-Defects4J, attributed to Ni et al., extends that material into a JIT defect prediction dataset by tracing each bug-fixing line backward with `git blame` to identify the earlier bug-inducing line and its introducing commit [2507.19714].

This lineage distinguishes JIT-Defects4J from classical JIT datasets such as the Kamei dataset and the Trac dataset. Kamei-style corpora are commit-level, SZZ-based, and centered on hand-engineered process metrics, with well-known issues around preprocessing, label noise, and limited semantic traceability [2109.13634]. The Trac dataset similarly retains commit hashes and supports semantic reconstruction, but it is not based on Defects4J and does not use Defects4J’s curated bug corpus [2110.00579]. By contrast, JIT-Defects4J is explicitly tied to the Defects4J ecosystem and thereby inherits a closer connection to bug-fix semantics and repository-grounded source evolution [2507.19714].

The relationship to Defects4J itself is also consequential. Defects4J provides manually isolated buggy and fixed versions with test suites, and later work has shown that Defects4J patches are typically small, localized, and addition-heavy [1801.06393]. A plausible implication is that JIT-Defects4J inherits some of these structural biases while reframing them into a temporal prediction setting.

## 2. Construction pipeline and data model

The dataset construction described for JIT-Defects4J begins with manually labeled bug-fixing lines from LLTC4J and then applies `git blame` to locate the last revision that introduced each such line [2507.19714]. If a bug-fixing line is denoted \(s^{fix}\), the backward trace returns a bug-inducing commit and bug-inducing line, written in the paper as:
\[
\texttt{git blame} \Rightarrow (c^{ind}, s^{ind})
\]
The earlier commit \(c^{ind}\) is labeled buggy, and the earlier line \(s^{ind}\) is labeled as a bug-inducing line. All other commits are treated as clean [2507.19714].

The resulting dataset is therefore **commit-level and line-level**. The prediction unit for JIT defect prediction is the commit, with a binary label of buggy versus clean, but each commit is also connected to a set of changed lines, some of which carry line-level bug-inducing labels [2507.19714]. This dual granularity is central to later effort-aware evaluation, because model outputs at commit level can be assessed against the discovery of bug-inducing lines under a lines-of-code inspection budget [2507.19714].

The same paper states that JIT-Defects4J exposes two major classes of representation. First, it includes **14 classical code change metrics** computed with **CommitGuru**. Second, it includes **semantic inputs**—specifically commit messages and code diffs—extracted using **PyDriller** [2507.19714]. This means that JIT-Defects4J can support both metrics-based and semantics-based JIT defect prediction models.

This construction differs from Kamei-style SZZ datasets in an important respect. Traditional datasets often expose only aggregated metrics and sometimes omit commit hashes, which limits semantic reconstruction [2110.00579]. JIT-Defects4J, as summarized in the available account, explicitly supports richer downstream analyses because the inducing changes are tied to concrete lines and commits [2507.19714].

## 3. Composition, granularity, and class distribution

The most concrete published description reports that JIT-Defects4J contains **21 projects**, **27,319 commits** in total, **2,397 buggy commits**, **119,368 lines** of changed code with line-level labels, and **12,594 bug-inducing lines** [2507.19714]. The same source characterizes the resulting imbalance as moderate: buggy commits range from **2.7% to 19.3%** per project, and bug-inducing lines range from **3.9% to 22.6%** per project [2507.19714].

The projects listed are:

- ant-ivy
- commons-bcel
- commons-beanutils
- commons-codec
- commons-collections
- commons-compress
- commons-configuration
- commons-dbcp
- commons-digester
- commons-io
- commons-jcs
- commons-lang
- commons-math
- commons-net
- commons-scxml
- commons-validator
- commons-vfs
- giraph
- gora
- opennlp
- parquet-mr

Selected per-project examples are also given. For `commons-math`, the dataset includes **4,026 commits**, **362 buggy commits**, **16,960 labeled lines**, and **3,046 bug-inducing lines**. For `ant-ivy`, it includes **1,771 commits**, **350 buggy commits**, **14,853 labeled lines**, and **1,650 bug-inducing lines** [2507.19714].

| Aspect | Reported value |
|---|---:|
| Projects | 21 |
| Total commits | 27,319 |
| Buggy commits | 2,397 |
| Labeled changed lines | 119,368 |
| Bug-inducing lines | 12,594 |

Because the dataset is commit-level for prediction but line-level for supervision and effort-aware evaluation, it supports a finer notion of inspection effort than many classical JIT benchmarks. This aligns with earlier arguments that coarse commit-level labels overestimate practical inspection burden when a defective commit touches many files or lines that are not themselves defective [2110.00579].

## 4. Feature space and supported modeling regimes

The literature describes JIT-Defects4J as supporting both **classical metric-based** and **semantic** JIT defect prediction. The 14 classical change metrics reported are the Kamei-style set grouped as diffusion, size, purpose, history, and experience features [2507.19714]. They are:

- \(NS\): number of modified subsystems
- \(ND\): number of modified directories
- \(NF\): number of modified files
- Entropy: distribution of modified code across files
- \(LA\): lines of code added
- \(LD\): lines of code deleted
- \(LT\): lines of code in a file before the change
- \(FIX\): whether the commit is a defect fix
- \(NDEV\): number of developers that changed the modified files
- \(AGE\): average time interval between previous change and current change
- \(NUC\): number of unique changes to modified files
- \(EXP\): developer’s overall experience
- \(REXP\): recent experience
- \(SEXP\): subsystem experience

These metrics are standard in JIT defect prediction research, but previous work has emphasized that they are sensitive to preprocessing choices and that smaller subsets such as \(LT\), \(PD/AGE\), and \(SEXP\) can be especially informative in some corpora [2109.13634]. This suggests that JIT-Defects4J can serve not only as a benchmark for modeling but also as a benchmark for studying feature stability and transferability.

The dataset also supports semantic modeling through commit messages and diffs [2507.19714]. This enables the evaluation of models such as DeepJIT, JITLine, and JIT-Fine, which consume tokenized change content rather than relying only on process metrics [2507.19714]. In related JIT literature, semantic models have been motivated partly by the inadequacy of coarse process features for distinguishing changes with similar metric profiles but different semantic correctness [2109.13634].

A further extension arises from refactoring-aware representations. The CAT-based study defines **66 refactoring-aware metrics (RAMs)**, combining line-level, class-level, and method-level counts of refactoring, propagation, move, and edit categories, and then concatenates them with the original 14 metrics to obtain an 80-dimensional feature vector [2507.19714]. For semantic models, CAT also reorders diff lines so that categories judged more informative for defect risk are prioritized under a 512-token input limit [2507.19714].

## 5. Label quality, refactoring, and CAT-based relabeling

A central controversy surrounding JIT-Defects4J concerns **label correctness in the presence of refactoring and propagation**. The original construction traces bug-fixing lines backward with `git blame`, but the refactoring-aware analysis argues that this can incorrectly blame pure refactoring or refactoring-propagation lines as bug-inducing [2507.19714]. The premise is concise: **refactoring \(\neq\) bug-inducing** [2507.19714].

To address this, the paper introduces **Code chAnge Tactics (CAT)**, which classifies changed statements into 18 fine-grained categories including `<Add/Del>`, `<Add/Del>_Move`, `<Add/Del>_Refactoring`, `<Add/Del>_Propagation`, `<Add/Del>_Edit`, and their composites [2507.19714]. CAT uses `git diff`, **RefactoringMiner 2.0**, call-dependency analysis for propagation detection, and statement matching using `JGit.diff.EditList` plus TF-based cosine similarity with threshold **0.8** [2507.19714]. The aim is to distinguish pure refactoring or propagation from genuinely bug-inducing edits.

Applied to JIT-Defects4J, CAT yields several quantitative corrections. The study reports that:

- **1,676 out of 12,594** bug-inducing lines are actually pure refactor/propagate lines
- **145** commits originally labeled buggy should be relabeled clean
- **210** new buggy commits are discovered earlier in history
- **35** of those 210 are excluded because they exceed a **10,000-line change filter**
- In total, **355 commits** in the dataset have corrected labels
- This is summarized as a **13.7%** improvement in labeling accuracy at commit level [2507.19714]

The same paper also reports that **41.3%** of all commits contain refactoring and that **10.8%** of added lines are involved in refactoring, with **0.7%** involving propagation and **1.4%** being tangled refactor+edit lines [2507.19714]. A plausible implication is that any JIT benchmark derived from real repository histories will be highly sensitive to whether it separates behavior-preserving structural change from semantically meaningful edits.

This criticism is reinforced by broader literature on Defects4J and JIT datasets. Defects4J fixes are curated and isolated from unrelated edits [1801.06393], whereas JIT datasets necessarily confront tangled real commits. Earlier JIT work on Trac similarly emphasized that commit-level labels conceal substantial internal heterogeneity and inspection effort [2110.00579].

## 6. Evaluation protocols and empirical usage

The available JIT-Defects4J literature describes its use as the main benchmark for six JIT defect prediction approaches: **Deeper**, **JIT-DIL**, **LApredict**, **DeepJIT**, **JITLine**, and **JIT-Fine** [2507.19714]. The reported train/test protocol follows Ni et al. and uses **80%** of commits for training and **20%** for testing within each project [2507.19714].

The evaluation includes standard classification metrics and effort-aware measures. The paper defines:
\[
Precision = \frac{TP}{TP+FP}
\]
\[
Recall = \frac{TP}{TP+FN}
\]
\[
F1 = \frac{2 \times Precision \times Recall}{Precision + Recall}
\]
For effort-aware evaluation, commits are ranked by predicted defectiveness and inspected under a lines-of-code budget. The reported measures are:

\[
R@20\%E = \frac{\# \text{ of true bug-inducing lines in inspected LOC}}{\text{total \# of bug-inducing lines}}
\]

\[
E@20\%R = \frac{\text{LOC inspected when first 20\% of bug-inducing lines are found}}{\text{total LOC}}
\]

\[
P_{opt} = 1 - \frac{Area(\text{Optimal}) - Area(M)}{Area(\text{Optimal}) - Area(\text{Worst})}
\]

These measures exploit the line-level labels that distinguish JIT-Defects4J from purely commit-level corpora [2507.19714].

The refactoring-aware study reports that moving from the original dataset labels to the augmented labels affects semantic models most strongly. For **JITLine-S**, F1 rises from **24.6** to **29.2**, an **18.6%** increase. For **JIT-Fine-S**, F1 rises from **19.3** to **26.5**, a **37.3%** increase; \(R@20\%E\) rises from **36.4** to **63.5**, a **74.4%** increase; and \(E@20\%R\) drops from **9.0** to **2.2** [2507.19714]. The same study reports further gains when CAT-derived representations are added, including up to **43.2%** recall increase and **32.5%** F1 increase for certain metrics-based models [2507.19714].

Beyond prediction, JIT-Defects4J is implicated in work on defect localization. The CodeFlowLM abstract reports exploratory analysis of just-in-time defect localization and notes that false positives may partly reflect “potential dataset mislabeling in JIT-Defects4J” [2512.00231]. However, the detailed source material available for that paper does not provide factual dataset-level elaboration beyond the abstract. It is therefore only possible to say that the dataset has already surfaced in discussions of label reliability for localization-oriented prompting and reasoning [2512.00231].

## 7. Relationship to adjacent benchmarks and methodological cautions

JIT-Defects4J is methodologically adjacent to, but distinct from, several other benchmark traditions. Compared with classical JIT datasets, it offers finer traceability to concrete code lines and richer semantic inputs [2507.19714]. Compared with Defects4J proper, it shifts the unit of analysis from isolated buggy/fixed version pairs to historical introducing commits and line-level blame links. Compared with the Trac dataset, it remains Java-specific and Defects4J-centric rather than issue-tracker-centric [2110.00579].

Several cautions follow from the surrounding literature.

First, **Defects4J-derived evaluation may be contaminated for LLM-based methods**. Work on the GitHub Recent Bugs dataset found likely overlap between Defects4J artifacts and StarCoder’s training data, with **59%** of Defects4J v1.0 bugs marked as compromised under a 90% overlap criterion [2310.13229]. JIT-Defects4J is not named there, but because it is derived from the same repositories, a plausible implication is that LLM-based evaluation on JIT-Defects4J may also be vulnerable to contamination [2310.13229].

Second, **Defects4J itself has reproducibility and specification limitations**. A later study of Defects4J 2.0 reports that **180 of 835 defects (21.6%)** are not workable for rigorous APR evaluation and that **59 (7.1%)** more have obviously under-specified tests [2604.26674]. JIT-Defects4J is a prediction dataset rather than an APR benchmark, but because it inherits from Defects4J projects, this suggests that downstream tasks involving test-driven validation or repair should treat per-bug test adequacy and execution stability carefully.

Third, **Defects4J faults and patches are structurally biased**. Patch dissection shows that the median Defects4J patch size is **4** lines, **92.41%** of patches modify only one file, and the most common repair patterns are Conditional Block, Expression Fix, and Wraps-with [1801.06393]. A plausible implication is that JIT-Defects4J may overrepresent defect introductions whose eventual fixes are small and localized, although the JIT setting reintroduces tangling and historical context absent from curated Defects4J patches.

Finally, JIT-Defects4J also sits within a broader push toward **fine-grained JIT prediction**. Earlier research argued that commit-level JIT labels leave developers with substantial inspection effort and advocated file-level or line-level analyses enabled by commit-linked datasets [2110.00579]. In that sense, JIT-Defects4J operationalizes a direction that earlier JIT work had primarily motivated rather than fully realized.

## 8. Research uses and prospective extensions

Within the documented literature, JIT-Defects4J serves three main purposes. It is a benchmark for **commit-level just-in-time defect prediction**, a source of **line-level effort-aware evaluation**, and a substrate for studying **label quality under refactoring and tangled changes** [2507.19714]. Because it combines process metrics with semantic inputs, it supports comparative study of metrics-only, semantics-only, and hybrid models [2507.19714].

Several extensions are suggested by neighboring research. Graph-based JIT prediction has shown that developer–file contribution graphs can substantially improve defect prediction on other JIT corpora [2110.05371]. This suggests that a graph-enriched variant of JIT-Defects4J, if equipped with developer identity normalization and full file histories, could support contribution-graph or GNN-style formulations. Similarly, fault-taxonomy work on Defects4J has produced control-flow and data-flow class labels for **488 faults** across seven projects [2502.02299]. Where JIT-Defects4J instances align with those Defects4J bugs, these labels could provide an additional layer of semantic supervision for predicting not merely whether a change is buggy, but what kind of fault it introduces.

The current state of the evidence therefore presents JIT-Defects4J as a **high-value but methodologically delicate** benchmark: richer than traditional JIT datasets in granularity and semantics, yet sensitive to blame-based mislabeling, refactoring tangling, benchmark contamination, and inherited Defects4J biases [2507.19714]. Its significance lies less in being a definitive ground truth than in making those tensions measurable within a single dataset.

Source: https://www.emergentmind.com/topics/jit-defects4j-dataset