AtomicCommitBench: Can Coding Agents Reconstruct Commit Histories from Squashed Patches?

Abstract: Coding agents often finish a session by returning one squashed patch that mixes feature implementation, bug fixes, refactorings, tests, and configuration edits. While the final code may be correct, collapsing unrelated edits into one patch removes the history structure needed for review, selective revert, and later maintenance. We study retrospective commit-history reconstruction: given a completed squashed change, an agent groups its hunks into commits and materializes a replayable commit sequence. We formalize the task as hunk-to-commit partitioning with a replay requirement and build AtomicCommitBench, containing 800 real consecutive-commit episodes from 10 Python projects. Because multiple decompositions may be reasonable, we evaluate outputs using complementary metrics: PPAR for replay validity, ARI for reference-based grouping quality, and TCR for failure containment on scoreable modified-test episodes. Natural retrospective reconstruction proves substantially harder than replay checking or synthetic tangling. Although nearly all non-random methods achieve replay validity (PPAR >= 0.988), grouping quality ranges from 0.03 to 0.46 ARI. Matched synthetic composites are much easier than real same-author squashed diffs (+0.333 ARI). In our evaluation, the GPT-5.4 setup driven by Codex CLI (0.46 ARI) and the GLM-5 setup driven by Claude Code (0.43 ARI) outperform MiniMax (0.31) and Kimi (0.29). Qualitative analysis identifies same-file lumping and support-hunk drift as recurring failure modes. Dependency-Aware Commit Evidence (DACE) improves the lower-scoring setups by 0.05 to 0.08 ARI, indicating that dependency cues and hunk-role information help agents avoid locality-driven grouping errors. AtomicCommitBench enables evaluation of the commit histories produced by coding agents alongside the final code.

• The paper introduces AtomicCommitBench, a benchmark that formalizes hunk-to-commit partitioning for reconstructing atomic commit histories from squashed patches.
• It employs a three-pronged evaluation using replay validity (PPAR), grouping quality (ARI), and failure containment (TCR) on 800 multi-commit episodes from mature Python repositories.
• Empirical findings reveal that advanced agent configurations, such as GPT-5.4 and GLM-5, outperform baseline heuristics while highlighting challenges in handling complex, tangled diffs.

Retrospective Commit-History Reconstruction with AtomicCommitBench

Motivation and Task Formalization

AtomicCommitBench introduces a formal methodology for assessing the retrospective organization of commit histories in software development. Coding agents frequently deliver squashed patches encapsulating varied change intents—feature additions, bug fixes, refactorings, test modifications, and configuration edits—often obscuring the semantic intent and modularity necessary for robust maintenance, reviewability, and automated tooling. The paper formalizes the challenge as hunk-to-commit partitioning, requiring agents to reconstruct a sequence of atomic commits from a squashed patch, given only the pre-change repository state. The task is shaped as a partitioning problem over diff hunks, with strong replay requirements: predicted groupings must be sequentially applicable without conflicts, materializing the intended history.

Figure 1: Overview of retrospective commit-history reconstruction, separating structural replay validity, reference-based grouping quality, and selective-revert failure containment.

The benchmark is constructed from 800 multi-commit episodes sampled from 10 mature Python repositories, with each episode representing consecutive, same-author commits that are squashed for evaluation. The paper distinguishes its measurement approach from classical correctness-by-final-tree; instead, it explicitly quantifies the quality of history structuring, leveraging observed developer decompositions as a reference while acknowledging multiple plausible groupings.

Benchmark Design and Metrics

AtomicCommitBench employs a three-pronged evaluation protocol:

1. Structural Replay Validity (PPAR): Measures whether the reconstructed sequence can be applied sequentially without conflicts.
2. Reference-Based Grouping Quality (ARI): Evaluates alignment between predicted hunk groupings and human-maintained commit decompositions using Adjusted Rand Index.
3. Selective-Revert Failure Containment (TCR): Probes whether individual predicted commits localize behavioral changes, using patch-based test breakage analyses in episodes modifying executable tests.

The dataset is stratified into Easy, Medium, and Hard tiers based on commit count and diff size, providing a granularity-sensitive substrate for diagnostic analysis. Baseline heuristics (file-based split, hunk-split, dependency splits, random partitions) and four model-agent configurations—GPT-5.4, GLM-5, MiniMax, and Kimi—are evaluated. The DACE evidence-augmented runs introduce dependency and hunk-role profiling, supplying explicit cues for grouping decisions, but leave final organization to the agent.

Empirical Findings

Reconstruction Difficulty and Baseline Saturation

Replay validity nearly saturates across non-random methods (PPAR $\geq 0.988$), affirming triviality in producing executable patch sequences. However, grouping quality exhibits considerable variance (ARI $[0.03, 0.46]$), especially when synthetic composites (cross-episode tangling) are contrasted with real squashed diffs (ARI gap $+0.333$), highlighting the insufficiency of synthetic evaluation for capturing intra-session boundary ambiguity.

Figure 2: Main grouping results on 800 natural retrospective episodes, showing ARI stratification across baselines, model-agent setups, and DACE-augmented runs.

Figure 3: Untangling baseline (B5) ARI gap between matched synthetic composites and real squashed diffs, demonstrating synthetic cases are substantially easier for automated grouping.

Model-Agent Performance and History Quality

Model-agent setups outperform strong file-based heuristics, indicating recovery of meaningful organizational structure beyond path-local heuristics. GPT-5.4 (ARI=0.46) and GLM-5 (ARI=0.43) establish a high-quality band, while MiniMax (ARI=0.31) and Kimi (ARI=0.29) are closer to baseline. TCR, computed on scoreable episodes, reinforces the same setup ranking, quantifying failure containment and its relation to modularity and atomicity. The ARI gap widens on harder episodes (larger, more tangled diffs), and setup rankings are robust across alternative reference decompositions, strict output coverage, and pretraining cutoff splits.

Figure 4: A representative reconstruction case illustrating necessity of cross-file hunk grouping and same-file splitting for maintainable histories.

Error Mechanisms and Tool Augmentation

Qualitative trace analysis uncovers recurrent failure modes: excessive locality leads to same-file lumping and misattribution of support hunks. DACE augmentation supplies explicit dependency and hunk-role evidence, substantially improving ARI for locality-dominated setups (MiniMax: +0.075 ARI, Kimi: +0.049 ARI), but yielding minimal change for GPT-5.4 and GLM-5, suggesting these agents already internalize such cues. Successful DACE usage is characterized by anti-lumping and distributed intent connection, especially where path-based heuristics fail.

Practical and Theoretical Implications

AtomicCommitBench advances process-oriented evaluation of coding agents, shifting measurement from final patch correctness to the quality of artifact histories. Commit structuring is empirically shown to impact downstream tasks—defect prediction, code review, selective revert, automated retrieval—and is increasingly consumed by AI systems for maintenance and research. The benchmark establishes reproducible metrics and comparative diagnostics, supporting future development of prompt engineering, tool-assisted grouping, and advanced agentic coding systems. The evaluation protocol is extensible to other agentic artifacts with replayable validity and reference-based metrics.

Practically, these results demonstrate that current leading agents can construct draft histories aligned with human organization, but substantial gaps persist in harder, more ambiguous episodes. Theoretically, correctly decomposed histories reinforce modularity for downstream code search and bug localization, positioning history organization as a key property for agentic coding pipelines.

Speculation on Future AI Developments

Anticipated future advances include tighter integration of dependency reasoning, more granular sub-hunk grouping, and hybrid approaches to commit untangling. Improved evidence aggregation and explicit agentic support tools (similar to DACE) could further reduce locality-driven errors. As the ecosystem matures, commit structuring will likely be paired with message generation, semantic tagging, and context-aware retrieval, strengthening AI’s ability to both produce and consume maintainable histories. Benchmarking protocols similar to AtomicCommitBench may evolve to accommodate other process-level artifacts, supporting rigorous comparative evaluation across agentic coding workflows.

Conclusion

AtomicCommitBench provides a rigorous framework for evaluating the retrospective organization of commit histories produced by coding agents. Replay validity is trivial, but grouping quality—and its alignment to human decompositions—remains challenging. Strong agentic setups surpass file-based heuristics and demonstrate measurable progress toward structured, maintainable histories, with qualitative and quantitative analyses clarifying the mechanisms of error and improvement. Explicit evidence aids locality-driven setups, highlighting the role of dependency and hunk-role reasoning. The benchmark and its methodology are set to inform future work on agentic history structuring and process-level artifact evaluation in software engineering and AI-driven programming systems.