JIT-Defects4J Dataset

• JIT-Defects4J is a defect prediction dataset derived from the Defects4J ecosystem, linking bug-fixing commits to earlier bug-inducing changes via git blame.
• It features both 14 classical code change metrics and semantic inputs like commit messages and diffs, supporting hybrid prediction models.
• The dataset bridges traditional SZZ-based corpora and curated benchmarks, offering fine-grained, effort-aware evaluation with corrected labels using refactoring-aware methods.

Searching arXiv for papers on JIT-Defects4J and related Defects4J/JIT defect prediction work. The JIT-Defects4J dataset is a just-in-time defect prediction corpus derived from the Defects4J ecosystem and used for commit-level and line-level analysis of bug-inducing changes in Java projects. In the account currently available from the literature, it is described most concretely in work on refactoring-aware relabeling, where it serves as the core benchmark for just-in-time defect prediction under commit tangling and refactoring effects (Niu et al., 25 Jul 2025). Its construction links manually identified bug-fixing lines to earlier bug-inducing lines and commits through git blame, while also exposing classical code-change metrics and semantic artifacts such as commit messages and diffs (Niu et al., 25 Jul 2025). The dataset occupies a distinctive position between traditional SZZ-based JIT corpora, which emphasize process metrics, and Defects4J-style benchmarks, which emphasize curated bug/fix pairs and executable tests. This suggests that JIT-Defects4J functions as a bridge between defect prediction, defect localization, and program-repair-oriented analysis.

1. Provenance and conceptual position

JIT-Defects4J is rooted in the Defects4J project family rather than in the earlier Kamei-style just-in-time datasets. The base projects are Apache Java projects derived from the Defects4J ecosystem, and the dataset construction is explicitly described as a two-stage process involving LLTC4J and then JIT-Defects4J (Niu et al., 25 Jul 2025). LLTC4J, attributed to Herbold et al., focuses on bug-fixing commits only and manually labels changed lines in those commits as “bug-fixing line” versus other non-bug-fixing changes in the same commit. JIT-Defects4J, attributed to Ni et al., extends that material into a JIT defect prediction dataset by tracing each bug-fixing line backward with git blame to identify the earlier bug-inducing line and its introducing commit (Niu et al., 25 Jul 2025).

This lineage distinguishes JIT-Defects4J from classical JIT datasets such as the Kamei dataset and the Trac dataset. Kamei-style corpora are commit-level, SZZ-based, and centered on hand-engineered process metrics, with well-known issues around preprocessing, label noise, and limited semantic traceability (Ng et al., 2021). The Trac dataset similarly retains commit hashes and supports semantic reconstruction, but it is not based on Defects4J and does not use Defects4J’s curated bug corpus (Ng et al., 2021). By contrast, JIT-Defects4J is explicitly tied to the Defects4J ecosystem and thereby inherits a closer connection to bug-fix semantics and repository-grounded source evolution (Niu et al., 25 Jul 2025).

The relationship to Defects4J itself is also consequential. Defects4J provides manually isolated buggy and fixed versions with test suites, and later work has shown that Defects4J patches are typically small, localized, and addition-heavy (Sobreira et al., 2018). A plausible implication is that JIT-Defects4J inherits some of these structural biases while reframing them into a temporal prediction setting.

2. Construction pipeline and data model

The dataset construction described for JIT-Defects4J begins with manually labeled bug-fixing lines from LLTC4J and then applies git blame to locate the last revision that introduced each such line (Niu et al., 25 Jul 2025). If a bug-fixing line is denoted $s^{fix}$, the backward trace returns a bug-inducing commit and bug-inducing line, written in the paper as: $\texttt{git blame} \Rightarrow (c^{ind}, s^{ind})$ The earlier commit $c^{ind}$ is labeled buggy, and the earlier line $s^{ind}$ is labeled as a bug-inducing line. All other commits are treated as clean (Niu et al., 25 Jul 2025).

The resulting dataset is therefore commit-level and line-level. The prediction unit for JIT defect prediction is the commit, with a binary label of buggy versus clean, but each commit is also connected to a set of changed lines, some of which carry line-level bug-inducing labels (Niu et al., 25 Jul 2025). This dual granularity is central to later effort-aware evaluation, because model outputs at commit level can be assessed against the discovery of bug-inducing lines under a lines-of-code inspection budget (Niu et al., 25 Jul 2025).

The same paper states that JIT-Defects4J exposes two major classes of representation. First, it includes 14 classical code change metrics computed with CommitGuru. Second, it includes semantic inputs—specifically commit messages and code diffs—extracted using PyDriller (Niu et al., 25 Jul 2025). This means that JIT-Defects4J can support both metrics-based and semantics-based JIT defect prediction models.

This construction differs from Kamei-style SZZ datasets in an important respect. Traditional datasets often expose only aggregated metrics and sometimes omit commit hashes, which limits semantic reconstruction (Ng et al., 2021). JIT-Defects4J, as summarized in the available account, explicitly supports richer downstream analyses because the inducing changes are tied to concrete lines and commits (Niu et al., 25 Jul 2025).

3. Composition, granularity, and class distribution

The most concrete published description reports that JIT-Defects4J contains 21 projects, 27,319 commits in total, 2,397 buggy commits, 119,368 lines of changed code with line-level labels, and 12,594 bug-inducing lines (Niu et al., 25 Jul 2025). The same source characterizes the resulting imbalance as moderate: buggy commits range from 2.7% to 19.3% per project, and bug-inducing lines range from 3.9% to 22.6% per project (Niu et al., 25 Jul 2025).

The projects listed are:

• ant-ivy
• commons-bcel
• commons-beanutils
• commons-codec
• commons-collections
• commons-compress
• commons-configuration
• commons-dbcp
• commons-digester
• commons-io
• commons-jcs
• commons-lang
• commons-math
• commons-net
• commons-scxml
• commons-validator
• commons-vfs
• giraph
• gora
• opennlp
• parquet-mr

Selected per-project examples are also given. For commons-math, the dataset includes 4,026 commits, 362 buggy commits, 16,960 labeled lines, and 3,046 bug-inducing lines. For ant-ivy, it includes 1,771 commits, 350 buggy commits, 14,853 labeled lines, and 1,650 bug-inducing lines (Niu et al., 25 Jul 2025).

Aspect	Reported value
Projects	21
Total commits	27,319
Buggy commits	2,397
Labeled changed lines	119,368
Bug-inducing lines	12,594

Because the dataset is commit-level for prediction but line-level for supervision and effort-aware evaluation, it supports a finer notion of inspection effort than many classical JIT benchmarks. This aligns with earlier arguments that coarse commit-level labels overestimate practical inspection burden when a defective commit touches many files or lines that are not themselves defective (Ng et al., 2021).

4. Feature space and supported modeling regimes

The literature describes JIT-Defects4J as supporting both classical metric-based and semantic JIT defect prediction. The 14 classical change metrics reported are the Kamei-style set grouped as diffusion, size, purpose, history, and experience features (Niu et al., 25 Jul 2025). They are:

• $NS$: number of modified subsystems
• $ND$: number of modified directories
• $NF$: number of modified files
• Entropy: distribution of modified code across files
• $LA$: lines of code added
• $LD$: lines of code deleted
• $LT$: lines of code in a file before the change
• $\texttt{git blame} \Rightarrow (c^{ind}, s^{ind})$0: whether the commit is a defect fix
• $\texttt{git blame} \Rightarrow (c^{ind}, s^{ind})$1: number of developers that changed the modified files
• $\texttt{git blame} \Rightarrow (c^{ind}, s^{ind})$2: average time interval between previous change and current change
• $\texttt{git blame} \Rightarrow (c^{ind}, s^{ind})$3: number of unique changes to modified files
• $\texttt{git blame} \Rightarrow (c^{ind}, s^{ind})$4: developer’s overall experience
• $\texttt{git blame} \Rightarrow (c^{ind}, s^{ind})$5: recent experience
• $\texttt{git blame} \Rightarrow (c^{ind}, s^{ind})$6: subsystem experience

These metrics are standard in JIT defect prediction research, but previous work has emphasized that they are sensitive to preprocessing choices and that smaller subsets such as $\texttt{git blame} \Rightarrow (c^{ind}, s^{ind})$7, $\texttt{git blame} \Rightarrow (c^{ind}, s^{ind})$8, and $\texttt{git blame} \Rightarrow (c^{ind}, s^{ind})$9 can be especially informative in some corpora (Ng et al., 2021). This suggests that JIT-Defects4J can serve not only as a benchmark for modeling but also as a benchmark for studying feature stability and transferability.

The dataset also supports semantic modeling through commit messages and diffs (Niu et al., 25 Jul 2025). This enables the evaluation of models such as DeepJIT, JITLine, and JIT-Fine, which consume tokenized change content rather than relying only on process metrics (Niu et al., 25 Jul 2025). In related JIT literature, semantic models have been motivated partly by the inadequacy of coarse process features for distinguishing changes with similar metric profiles but different semantic correctness (Ng et al., 2021).

A further extension arises from refactoring-aware representations. The CAT-based study defines 66 refactoring-aware metrics (RAMs), combining line-level, class-level, and method-level counts of refactoring, propagation, move, and edit categories, and then concatenates them with the original 14 metrics to obtain an 80-dimensional feature vector (Niu et al., 25 Jul 2025). For semantic models, CAT also reorders diff lines so that categories judged more informative for defect risk are prioritized under a 512-token input limit (Niu et al., 25 Jul 2025).

5. Label quality, refactoring, and CAT-based relabeling

A central controversy surrounding JIT-Defects4J concerns label correctness in the presence of refactoring and propagation. The original construction traces bug-fixing lines backward with git blame, but the refactoring-aware analysis argues that this can incorrectly blame pure refactoring or refactoring-propagation lines as bug-inducing (Niu et al., 25 Jul 2025). The premise is concise: refactoring $c^{ind}$0 bug-inducing (Niu et al., 25 Jul 2025).

To address this, the paper introduces Code chAnge Tactics (CAT), which classifies changed statements into 18 fine-grained categories including <Add/Del>, <Add/Del>_Move, <Add/Del>_Refactoring, <Add/Del>_Propagation, <Add/Del>_Edit, and their composites (Niu et al., 25 Jul 2025). CAT uses git diff, RefactoringMiner 2.0, call-dependency analysis for propagation detection, and statement matching using JGit.diff.EditList plus TF-based cosine similarity with threshold 0.8 (Niu et al., 25 Jul 2025). The aim is to distinguish pure refactoring or propagation from genuinely bug-inducing edits.

Applied to JIT-Defects4J, CAT yields several quantitative corrections. The study reports that:

• 1,676 out of 12,594 bug-inducing lines are actually pure refactor/propagate lines
• 145 commits originally labeled buggy should be relabeled clean
• 210 new buggy commits are discovered earlier in history
• 35 of those 210 are excluded because they exceed a 10,000-line change filter
• In total, 355 commits in the dataset have corrected labels
• This is summarized as a 13.7% improvement in labeling accuracy at commit level (Niu et al., 25 Jul 2025)

The same paper also reports that 41.3% of all commits contain refactoring and that 10.8% of added lines are involved in refactoring, with 0.7% involving propagation and 1.4% being tangled refactor+edit lines (Niu et al., 25 Jul 2025). A plausible implication is that any JIT benchmark derived from real repository histories will be highly sensitive to whether it separates behavior-preserving structural change from semantically meaningful edits.

This criticism is reinforced by broader literature on Defects4J and JIT datasets. Defects4J fixes are curated and isolated from unrelated edits (Sobreira et al., 2018), whereas JIT datasets necessarily confront tangled real commits. Earlier JIT work on Trac similarly emphasized that commit-level labels conceal substantial internal heterogeneity and inspection effort (Ng et al., 2021).

6. Evaluation protocols and empirical usage

The available JIT-Defects4J literature describes its use as the main benchmark for six JIT defect prediction approaches: Deeper, JIT-DIL, LApredict, DeepJIT, JITLine, and JIT-Fine (Niu et al., 25 Jul 2025). The reported train/test protocol follows Ni et al. and uses 80% of commits for training and 20% for testing within each project (Niu et al., 25 Jul 2025).

The evaluation includes standard classification metrics and effort-aware measures. The paper defines: $c^{ind}$1

$c^{ind}$2

$c^{ind}$3

For effort-aware evaluation, commits are ranked by predicted defectiveness and inspected under a lines-of-code budget. The reported measures are:

$c^{ind}$4

$c^{ind}$5

$c^{ind}$6

These measures exploit the line-level labels that distinguish JIT-Defects4J from purely commit-level corpora (Niu et al., 25 Jul 2025).

The refactoring-aware study reports that moving from the original dataset labels to the augmented labels affects semantic models most strongly. For JITLine-S, F1 rises from 24.6 to 29.2, an 18.6% increase. For JIT-Fine-S, F1 rises from 19.3 to 26.5, a 37.3% increase; $c^{ind}$7 rises from 36.4 to 63.5, a 74.4% increase; and $c^{ind}$8 drops from 9.0 to 2.2 (Niu et al., 25 Jul 2025). The same study reports further gains when CAT-derived representations are added, including up to 43.2% recall increase and 32.5% F1 increase for certain metrics-based models (Niu et al., 25 Jul 2025).

Beyond prediction, JIT-Defects4J is implicated in work on defect localization. The CodeFlowLM abstract reports exploratory analysis of just-in-time defect localization and notes that false positives may partly reflect “potential dataset mislabeling in JIT-Defects4J” (Monteiro et al., 28 Nov 2025). However, the detailed source material available for that paper does not provide factual dataset-level elaboration beyond the abstract. It is therefore only possible to say that the dataset has already surfaced in discussions of label reliability for localization-oriented prompting and reasoning (Monteiro et al., 28 Nov 2025).

7. Relationship to adjacent benchmarks and methodological cautions

JIT-Defects4J is methodologically adjacent to, but distinct from, several other benchmark traditions. Compared with classical JIT datasets, it offers finer traceability to concrete code lines and richer semantic inputs (Niu et al., 25 Jul 2025). Compared with Defects4J proper, it shifts the unit of analysis from isolated buggy/fixed version pairs to historical introducing commits and line-level blame links. Compared with the Trac dataset, it remains Java-specific and Defects4J-centric rather than issue-tracker-centric (Ng et al., 2021).

Several cautions follow from the surrounding literature.

First, Defects4J-derived evaluation may be contaminated for LLM-based methods. Work on the GitHub Recent Bugs dataset found likely overlap between Defects4J artifacts and StarCoder’s training data, with 59% of Defects4J v1.0 bugs marked as compromised under a 90% overlap criterion (Lee et al., 2023). JIT-Defects4J is not named there, but because it is derived from the same repositories, a plausible implication is that LLM-based evaluation on JIT-Defects4J may also be vulnerable to contamination (Lee et al., 2023).

Second, Defects4J itself has reproducibility and specification limitations. A later study of Defects4J 2.0 reports that 180 of 835 defects (21.6%) are not workable for rigorous APR evaluation and that 59 (7.1%) more have obviously under-specified tests (Krafczyk et al., 29 Apr 2026). JIT-Defects4J is a prediction dataset rather than an APR benchmark, but because it inherits from Defects4J projects, this suggests that downstream tasks involving test-driven validation or repair should treat per-bug test adequacy and execution stability carefully.

Third, Defects4J faults and patches are structurally biased. Patch dissection shows that the median Defects4J patch size is 4 lines, 92.41% of patches modify only one file, and the most common repair patterns are Conditional Block, Expression Fix, and Wraps-with (Sobreira et al., 2018). A plausible implication is that JIT-Defects4J may overrepresent defect introductions whose eventual fixes are small and localized, although the JIT setting reintroduces tangling and historical context absent from curated Defects4J patches.

Finally, JIT-Defects4J also sits within a broader push toward fine-grained JIT prediction. Earlier research argued that commit-level JIT labels leave developers with substantial inspection effort and advocated file-level or line-level analyses enabled by commit-linked datasets (Ng et al., 2021). In that sense, JIT-Defects4J operationalizes a direction that earlier JIT work had primarily motivated rather than fully realized.

8. Research uses and prospective extensions

Within the documented literature, JIT-Defects4J serves three main purposes. It is a benchmark for commit-level just-in-time defect prediction, a source of line-level effort-aware evaluation, and a substrate for studying label quality under refactoring and tangled changes (Niu et al., 25 Jul 2025). Because it combines process metrics with semantic inputs, it supports comparative study of metrics-only, semantics-only, and hybrid models (Niu et al., 25 Jul 2025).

Several extensions are suggested by neighboring research. Graph-based JIT prediction has shown that developer–file contribution graphs can substantially improve defect prediction on other JIT corpora (Bryan et al., 2021). This suggests that a graph-enriched variant of JIT-Defects4J, if equipped with developer identity normalization and full file histories, could support contribution-graph or GNN-style formulations. Similarly, fault-taxonomy work on Defects4J has produced control-flow and data-flow class labels for 488 faults across seven projects (Spuy et al., 4 Feb 2025). Where JIT-Defects4J instances align with those Defects4J bugs, these labels could provide an additional layer of semantic supervision for predicting not merely whether a change is buggy, but what kind of fault it introduces.

The current state of the evidence therefore presents JIT-Defects4J as a high-value but methodologically delicate benchmark: richer than traditional JIT datasets in granularity and semantics, yet sensitive to blame-based mislabeling, refactoring tangling, benchmark contamination, and inherited Defects4J biases (Niu et al., 25 Jul 2025). Its significance lies less in being a definitive ground truth than in making those tensions measurable within a single dataset.