Corrected CodeShovel Oracle

• Corrected CodeShovel Oracle is an expert-validated benchmark system that refines method-level software history by addressing false positives, false negatives, and granularity issues.
• It employs a union-of-candidates from five automated tools followed by dual-expert review to confirm only genuine semantic modifications.
• Empirical results show improved precision and recall with significant adjustments in commit inclusion, enhancing the robustness of automated software evolution analysis.

A corrected CodeShovel oracle is an expert-validated, method-level ground truth for evaluating software history generation tools, produced through a combined automated and manual pipeline. This oracle addresses systematic inaccuracies in previous oracles—most notably, the manual CodeShovel and CodeTracker oracles—by employing a union-and-verify methodology across multiple automated detectors and rigorous human adjudication. The corrected oracle substantially refines the commit-level history of source code methods and supports robust, unbiased performance evaluation in research on automated software evolution analysis (Islam et al., 19 Jul 2025).

1. Rationale and Motivating Problems

The original CodeShovel oracle (Grund et al., ICSE 2021) was designed for benchmarking method-level code history tools but suffered from three substantial flaws. First, it included false positives: commits not actually modifying the method, such as merge-commit artifacts. Second, it missed genuine changes, yielding false negatives, commonly during method renames or non-standard edits. Third, it exhibited granularity blind spots, overlooking semantically meaningful modifications such as annotation, JavaDoc, or formatting-based changes. These flaws led to biased evaluation metrics—overstating precision when spurious commits were included and underestimating recall when genuine changes were omitted. Improving the oracle thus became necessary to restore measurement fidelity for tool evaluation (Islam et al., 19 Jul 2025).

2. Construction Methodology

The corrected CodeShovel oracle is built through a three-phase, semi-automated pipeline:

1. Automated Candidate Collection: For each target method, five detection tools are applied—
   • CodeShovel library API
   • CodeTracker library API
   • IntelliJ “Show History for Method” via GUI automation
   • GitFuncName: git log <commit> –no-merges -L :<func>:<file>
   • GitLineRange: git log <commit> –no-merges -L <start>,<end>:<file>
2. Union Aggregation and Sorting:
   • The union of all reported candidate commits is computed for each method: $$C_\text{auto}(m) = \bigcup_{t \in T} \text{Commits}_t(m)$$.
   • Candidates are sorted in reverse chronological order by commit timestamp.
3. Expert Validation and Completion:
   • Two experts independently review each candidate commit, examining diffs against its parent(s).
   • A commit is retained if both experts agree that it introduces a semantic change (signature, body, JavaDoc, annotation, and name/structural changes).
   • Experts then manually search for missing changes outside the automated union, adding any overlooked relevant commits.
   • Resolution is reached by live discussion to reconcile disagreements, establishing the final set $H(m)$ (Islam et al., 19 Jul 2025).

Formal Description and Algorithm

Algorithm 1 (verbatim from (Islam et al., 19 Jul 2025)):

$H(m)$8

This formalizes the inclusion criterion: a commit $c$ modifies method $m$ if and only if both expert reviewers confirm a semantic code change in $m$.

3. Ground Truth Criteria and Evaluation Integration

The corrected oracle, $H(m)$, for each method $m$ consists of the union of all automated tool candidates confirmed by both experts, plus any additional expert-discovered relevant commits: $$H(m) = \{\,c \in C_\text{auto}(m) \mid f(c, m) = 1\,\} \cup C_\text{add}(m)$$ where $f(c, m) = 1$ iff both experts validate $c$ as a semantic change.

The evaluation of competing tools integrates the corrected oracle via standard information retrieval metrics at the commit level:

• $H(m)$0 (true positives)
• $H(m)$1 (false positives)
• $H(m)$2 (false negatives)
• Precision $H(m)$3
• Recall $H(m)$4
• $H(m)$5

These are averaged over all methods per evaluation run. The evaluation formulas themselves remain unchanged from prior work; only the reference sets $H(m)$6 are replaced, leading to automatic recalibration of all metric counts (Islam et al., 19 Jul 2025).

4. Quantitative Improvements and Example Corrections

Applying the corrected CodeShovel oracle to the canonical 200-method benchmark set yielded substantial updates:

Oracle	#Methods Changed (Excl.)	#Added Commits (Excl.)	#Removed Commits
CodeShovel	40	59	186
CodeTracker	43	134	67

Illustrative cases:

• fireErrors (checkstyle): The original oracle omitted a method introduction under the alias displayErrors. The corrected oracle included the missing rename commit and earlier ancestor, shifting the method’s introduction two steps backward.
• createPattern overload: Spurious inclusion of commits associated with unrelated method overloads, corrected by expert de-duplication.
• runChild (junit4): Merge-commit artifacts retained in prior oracles were dropped after diff analysis confirmed no actual method modification occurred.

This process resulted in both increased recall (more true changes present in $H(m)$7) and increased precision (fewer spurious entries), improving benchmark robustness (Islam et al., 19 Jul 2025).

5. Limitations and Recommendations for Reproduction

Residual limitations include potential subtleties in detecting deep semantic code changes (e.g., structural refactorings), which could elude even dual-expert review. The process relies primarily on textual git diffs; significant changes in method signatures may complicate reliable linkages between old and new versions. Automated AST (Abstract Syntax Tree) matching is suggested as a future enhancement.

For constructing analogous oracles in other languages or codebases, the process generalizes as follows:

1. Select target methods and retrieve their code spans.
2. Apply a diverse set of automated method history tools.
3. Aggregate and chronologically sort all candidate change commits.
4. At least two domain experts independently validate each candidate by diff inspection.
5. Adjudicate disagreements collaboratively and supplement with any manually discovered missing commits.
6. Finalize and package the resulting oracle set for downstream evaluation (Islam et al., 19 Jul 2025).

6. Impact on the Field and Future Directions

The corrected CodeShovel oracle has become the new reference ground truth for method-level history benchmarks in automated software evolution analysis. Its adoption enables more accurate and fair measurement of tool precision and recall, directly addressing biases introduced by previous oracles. All evaluations in the development of HistoryFinder and related tools now rely on this oracle, with demonstrable empirical advantages over prior datasets.

Expanding expert validation (e.g., via larger validation panels) and incorporating AST-level structural analysis are identified directions for further reducing internal validity threats and capturing complex refactoring cases. Extending this methodology to languages beyond Java is feasible, provided tooling and expert resources are available (Islam et al., 19 Jul 2025).

In summary, the corrected CodeShovel oracle embodies a rigorous, reproducible standard for ground truth construction in code change history research, significantly enhancing the reliability of comparative evaluations in the field.