JITDATA and CCIBENCH: CCI Benchmarking

• The paper introduces JITDATA and CCIBENCH as benchmark datasets that pair atomic code edits with comment annotations for evaluating inconsistency detection.
• It details a structured diff encoding approach with activity labeling to decompose code changes into atomic operations, enhancing semantic fidelity.
• CARL-CCI, built on a CodeT5+ backbone with a hybrid loss, achieves state-of-the-art improvements in F1 scores for detecting comment inconsistencies.

Ensuring semantic fidelity between software code and its comments is pivotal for program comprehension and maintainability. Automated comment inconsistency detection, or CCI detection, aims to identify when code modifications are not appropriately reflected in the associated comments. Two benchmark datasets—JITDATA and CCIBENCH—enable rigorous, large-scale evaluation of CCI detection algorithms. These datasets are now foundational in the empirical assessment of structured, activity-labeled code diff analysis for comment inconsistency detection, most notably as implemented in the CARL-CCI system leveraging the CodeT5+ backbone (Nguyen et al., 22 Dec 2025).

1. Datasets: JITDATA and CCIBENCH

JITDATA and CCIBENCH are publicly available benchmarks targeting the evaluation of just-in-time comment inconsistency detection in evolving software repositories. These datasets comprise annotated code snapshots paired with associated comments, each labeled for semantic consistency post-edit.

• JITDATA: Contains 32,988 training samples, 3,756 validation samples, and 3,944 test samples.
• CCIBENCH: Comprises 18,162 training samples, 2,068 validation samples, and 2,130 test samples.

Each sample records an atomic code change—including add, delete, keep, replace operations—tagged alongside the original and updated comments. This design ensures precise benchmarking of how well a model can detect when comment text fails to track code behavior across diverse modification histories.

2. Structured Code Diff Encoding and Activity Labeling

Traditional approaches to code-comment inconsistency detection often flatten code changes into linear token streams, obscuring the structural semantics of code evolution. JITDATA and CCIBENCH enable and require richer representations:

• Each edit is decomposed into atomic operations with special tokens:
  • <Add> ... <EndAdd>, <Del> ... <EndDel>, <Keep> ... <EndKeep>, <ReplaceOld> old-tokens <ReplaceNew> new-tokens <EndReplace>
• The complete diff is partitioned into:
  • $\mathcal{S}_{old}$: deleted and replaced-old tokens
  • $\mathcal{S}_{new}$: added and replaced-new tokens
  • $\mathcal{S}_{unchanged}$: all keep tokens

Activity labeling assigns semantic identity to each edit operation, substantially improving the model’s capacity to capture the correlation between code modifications and associated comment staleness.

3. CARL-CCI System Architecture and Training Protocols

CARL-CCI leverages the CodeT5p-220M (“CodeT5+”) backbone with 12 Transformer encoder layers, frozen decoder, and a lightweight binary classification head to detect comment inconsistency (Nguyen et al., 22 Dec 2025).

• Input Formatting:

x = [CLS] S_old [SEP] S_new [SEP] comment C [SEP] full_tagged_diff S_diff [EOS]

• Model Modifications: The decoder is omitted. A dense layer with sigmoid activation is attached to the encoder output for binary classification.
• Training Regimen:
  • Batch size: 32
  • Learning rate: $1\times10^{-5}$ (AdamW)
  • Contrastive temperature $\tau = 0.08$
  • Joint objective: binary cross-entropy ($L_{BCE}$) + label-aware contrastive loss ($L_{contrast}$)
  • Up to 20 epochs with early stopping on dev F1 score

4. Loss Functions and Mathematical Formulations

CARL-CCI optimizes a hybrid loss to leverage both binary and contrastive supervision.

Binary Cross-Entropy:

$$L_{BCE}(\mathcal{B}) = -\frac{1}{N} \sum_{i\in\mathcal{B}} [ y_i \log p_i + (1-y_i)\log(1-p_i) ]$$

Supervised Contrastive Terms:

• Positive pairs: $$L_{InfoNCE} = -\frac{1}{|\mathcal{B}_C|} \sum_{i \in \mathcal{B}_C} \log p_{ii}$$
• Negative push: $$L_{neg} = -\frac{1}{|\mathcal{B}_I|} \sum_{j \in \mathcal{B}_I} \log (1 - p_{jj})$$
• Joint contrastive loss: $L_{contrast} = L_{InfoNCE} + \alpha L_{neg}$
• Aggregate objective: $L_{total} = L_{BCE} + \lambda L_{contrast}$ where $\alpha \geq 0$ and $\lambda \geq 0$.

5. Empirical Performance and Component Contribution

Extensive comparative analysis demonstrates that structured activity-labeling yields the highest single ablation gain for CCI detection. Key results include:

Model	JITDATA F1 (%)	CCIBENCH F1 (%)
DeepJIT (2020)	81.96	87.15
C4RLLama (2025)	85.24	88.67
CCISolver (2025)	86.19	89.54
CodeLlama/Qwen2.5/DS	81.79–85.59	84.48–88.51
CARL-CCI (Ours)	90.89	93.38

Ablation Analysis:

Ablation Condition	JITDATA F1 (%)	Δ	CCIBENCH F1 (%)	Δ
w/o AL & CL	87.46	–3.43	89.47	–3.91
w/o AL (only CL)	87.79	–3.10	90.23	–3.15
w/o CL (only AL)	90.74	–0.15	93.06	–0.32
Full (AL + CL)	90.89		93.38

The activity-labeled decomposition of code diffs contributes approximately 3–4 percentage points (pp) improvement, whereas supervised contrastive learning adds up to 0.5 pp.

6. Significance and Future Extensions

JITDATA and CCIBENCH have established themselves as reference datasets for structured CCI detection. By facilitating granular, activity-aware training and evaluation, these benchmarks allow for precise measurement of models’ ability to generalize across code evolution patterns. The empirical superiority of the CARL-CCI approach—with CodeT5+ backbone—suggests that structured diff encoding and activity labeling are essential for robust inconsistency detection. A plausible implication is that further hybrid objectives or richer context modeling may yield additional gains.

Applications of JITDATA and CCIBENCH extend to repository auditing, automatic documentation health-checking, and preventative code review signaling. Their adoption foregrounds an emerging consensus: only structured, temporally-aware models leveraging explicit change activities are reliably scalable for industrial-grade comment consistency automation.