DiFF Benchmark: AST Differencing Evaluation

• DiFF Benchmark is a fine-grained evaluation suite that measures AST differencing accuracy at the commit level in both bug-fix and refactoring scenarios.
• It constructs a reliable ground-truth through aggregated outputs from six tools and expert consensus, ensuring semantic precision in node mappings.
• The benchmark employs diverse metrics such as precision, recall, F1 score, and runtime to reveal performance strengths and limitations of various AST diff algorithms.

The DiFF Benchmark is a fine-grained, commit-level evaluation suite for measuring the accuracy of Abstract Syntax Tree (AST) differencing tools in real-world software evolution scenarios. Developed to address the lack of standardized, high-granularity test sets for semantic code differencing, DiFF provides a ground-truth oracle of AST node mappings across both bug-fix and large-scale refactoring commits, enabling rigorous comparison and analysis of current and next-generation AST diff algorithms (Alikhanifard et al., 2024).

1. Design and Ground-Truth Construction

The DiFF Benchmark is composed of two complementary datasets:

• Bug-fix Commits: 800 bug-fixing commits from Defects4J, spanning 17 Java projects. These represent small, local changes that challenge tools minimally.
• Refactoring Commits: 188 refactoring-heavy commits curated from the Refactoring Oracle across 187 projects, each affecting no more than two files, selected to stress-test diff tools on dispersed, semantically-motivated code transformations.

Ground-truth AST-node mappings were constructed through a process of tool output aggregation and expert consensus:

• Six tools (RefactoringMiner 3.0; GumTree 3.0 [greedy/simple]; GumTree 2.1.0; IJM; MTDiff) were run on every commit.
• Side-by-side visual inspection and assessment were conducted using six criteria: statement continuity, developer intent, visual clarity, control-flow consistency, identifier-rename consistency, and comment anchoring.
• Mappings with unanimous tool agreement were accepted automatically; disputed cases (<3%) were resolved by group consensus, producing a fine-grained, human-validated oracle.
• Each commit/file pair records: matchedElements (type, method, field, enum-constant signature matches) and mappings (all mapped statements/expressions with AST type and code span).
• Inter-file moves are captured as grouped mappings by source/destination file.

2. Supported Evaluation Metrics

DiFF enables multidimensional quantitative analysis through the following metrics:

• Precision (P): $P = \frac{TP}{TP + FP}$, where $TP$ is the count of correctly predicted mappings and $FP$ the extra (incorrect) ones.
• Recall (R): $R = \frac{TP}{TP + FN}$, with $FN$ the missed mappings in a tool's output.
• F$_1$ Score: Harmonic mean of precision and recall.
• Perfect-diff Rate: Percentage of commits with all mappings correct and none missing.
• Semantic Incompatibility Count: Number of type-correct but semantically-incorrect matches.
• Runtime per Commit: Median runtime per tool/commit, giving practical performance baselines.

Unchanged subtrees under unmodified declarations were excluded to prevent metric inflation.

3. Benchmarking Results and Comparative Analysis

Key findings and comparative data from the benchmark include:

Metric	RefactoringMiner 3.0	GumTree 3.0 (simple)	GumTree 2.1.0	IJM	MTDiff
Multi-mapping F$_1$ (refactoring)	~99%	17%	<15%	n/a	<15%
Semantic Incompatibles (all commits)	0	214	504	85	1,288
Method Mapping (P/R, refactoring)	93.5/75.1%	93.9/75.9%	n/a	99.4/82.7	n/a
Field Mapping (P/R, refactoring)	98.8/98.5%	52.7/97.3%	n/a	98.2/44.5	n/a
Statement Mapping (F$_1$, refactoring)	99.5%	78.9%	n/a	76.3%	80.5%
Inter-file Moves (P/R)	99.6/99.6%	0/0%	0/0%	0/0%	0/0%
Perfect-diff Rate (Defects4J/refactoring)	89.3/81.9%	n/a	n/a	75.6/13.8%	n/a
Runtime (Defects4J/refactoring)	75/247ms	8/22ms	14/246ms	13/60ms	118/325ms

RefactoringMiner 3.0 is the only tool supporting multi-mappings and cross-file moves, consistently reporting F$_1$ ≈ 99% in sophisticated refactoring scenarios. Other tools show significant performance drops for such cases, underscoring the importance of semantic awareness, refactoring integration, and robust identifier handling.

4. Methodological Features and Innovations

The DiFF Benchmark is distinctive in several methodological aspects:

• Multi-mapping: Only RefactoringMiner supports mappings such as one-to-many and many-to-one, essential for documenting code fragment extraction, merging, or duplication elimination.
• Semantic Awareness: Explicit avoidance of matching AST nodes with incompatible semantic roles, eliminating high rates of semantically incorrect matches.
• Refactoring Awareness: Leverages detected refactoring instances (up to 60 types), call sites, and method signatures to drive accurate statement and field movement mappings within and across files.
• Inter-file Support: Captures moves across file boundaries, highly relevant for practical refactorings but unsupported by most legacy tools.
• Granularity: Supports mapping at the method, field, type/enum, and sub-statement/expression level.
• Execution Efficiency: While RefactoringMiner is not always the fastest (especially on large refactorings), its runtime remains in the same order of magnitude as competitive tools.

5. Insights, Limitations, and Recommendations

• Bug-fix vs. Refactoring Difficulty: All tools achieve >98% F$_1$ on Defects4J (bug-fix) commits, demonstrating that small-scale editing is a “low barrier.” Robust evaluation must include refactoring-intensive, multi-file commits to expose substantive differences between algorithms.
• Greedy Matching Pitfalls: Tools with greedy matching strategies (e.g., GumTree greedy, MTDiff) obtain more matches but also produce more false positives and semantic errors, typically underperforming “simple” configurations.
• Language and Refactoring Features: Incorporating language-specific cues and structural awareness (call sites, field references) yields dramatic improvements for complex refactorings.
• Tool Limitations: No tool except RefactoringMiner 3.0 supports Java test-class refactorings, certain rare multi-mapping patterns, or robust sub-string/near-duplicate detection.
• Future Directions: Extending the benchmark beyond Java (e.g., C#, Python) requires integrating with additional refactoring engines. Development of dedicated test-diff benchmarks and enhanced multi-mapping for exotic code patterns is warranted.

6. Significance and Community Impact

The DiFF Benchmark establishes a gold standard for systematic, reproducible, and semantically precise evaluation of AST differencing tools under real-world software evolution pressures. Its dual focus on both localized bug-fix edits and large-scale refactorings, combined with rich, expert-validated ground truth, allows deep inspection of algorithmic strengths and weaknesses.

Key contributions include:

• A standardized, challenging corpus representing both easy and hard scenarios.
• Rich evaluation metrics spanning precision, recall, F$TP$0, semantic violations, and perfect-diff rates.
• Direct plug-in infrastructure for tool authors to benchmark new algorithms or extensions on a recognized, community-maintained testbed.

DiFF thus serves as a practical, extensible reference for future research on program differencing and automated code change comprehension, with direct relevance to the advancement of program understanding, automated refactoring, and intelligent developer tooling (Alikhanifard et al., 2024).