EvoClaw Benchmark: Long-Horizon Code Evolution

• EvoClaw is a benchmark that measures AI agents' ability to continuously evolve real-world codebases while tracking technical debt and error propagation.
• It employs Milestone DAGs generated via the DeepCommit pipeline to structure semantically coherent development goals and enforce quality assurance.
• The benchmark reveals challenges in maintaining cumulative precision and preventing regressions in long-horizon, dependency-driven software evolution.

EvoClaw is a repository-evolution benchmark designed to evaluate AI coding agents on long-horizon, dependency-driven software development. In contrast to benchmarks assessing agents on isolated, one-off coding tasks, EvoClaw requires agents to continuously evolve a real-world codebase across a stream of semantically coherent development goals (“milestones”), maintaining system integrity and limiting error accumulation over extended software evolution. This design highlights dimensions of technical debt, error propagation, and maintenance rarely addressed in classical agent evaluation paradigms (Deng et al., 13 Mar 2026).

1. Milestone DAGs and the DeepCommit Pipeline

Central to EvoClaw is the Milestone Directed Acyclic Graph (DAG), wherein each node represents a “milestone”—a cluster of atomic commits grouped by semantic cohesiveness and structural dependency—and each edge encodes prerequisite relationships between these clusters. The construction of Milestone DAGs hinges on DeepCommit, a pipeline consisting of several automated and agentic phases:

• Commit-level DAG construction: Using static analysis of git-blame line dependencies, file co-change, and symbol-level diffs, the pipeline infers the directed structure of changes.
• Seed discovery and milestone consolidation: Milestone “seeds” are chosen via topological (e.g., out-degree, descendant count) and semantic (commit message) heuristics, after which neighboring commits are consolidated using criteria such as file overlap, temporal proximity, and semantic similarity.
• Dependency inference and validation: Proposed edges between milestones derive from shared files, symbol references, and temporal ordering. These candidate dependencies are validated by a LLM agent.
• Granularity refinement: Milestones deemed oversized (lines of code, LOC, above mean + 2 standard deviations) are split; undersized milestones (below mean – std and under 100 LOC) are merged, aiming for a coefficient of variation (CV) below 1.0 in LOC distribution.

A fully automated quality assurance layer checks for 100% commit coverage, ensures the DAG is acyclic and dependencies are consistent, and verifies executability via Docker-based test environments, achieving a test collection success rate of 87.1% and environment error rates below 0.10% (Deng et al., 13 Mar 2026).

2. Benchmark Design, Dataset, and Task Protocol

EvoClaw spans 7 high-impact open-source repositories, each featuring 98 graded milestones (plus 3 for context) and 124 inter-milestone dependencies, distributed across Go, Rust, Java, TypeScript, and Python. Milestones reflect real-world development goals, including feature additions, bug fixes, refactorings, enhancements, and chores. Empirically, each milestone modifies an average of 27.4 files, contains a patch of 570 LOC, is associated with 17.1 F2P (Fail-to-Pass) tests, and is monitored by 6,218 P2P (Pass-to-Pass) regression tests. Each milestone’s System Requirements Specification (SRS) is specified, with a mean length of 1,348 words.

The core task protocol enforces a dependency-driven task stream: a milestone unlocks only when all its prerequisites are resolved. Agents operate in a persistent environment, with each patch (whether correct or not) accumulating modifications and regressions. After every milestone, the system snapshots the state, reruns all relevant test suites—including filtering flaky tests via 3–5 executions—and classifies test outcomes as implementing new functionality (F2P) or preserving prior behavior (P2P). This protocol models long-term software evolution, system integrity, and error accumulation (Deng et al., 13 Mar 2026).

3. Metrics, Measurement, and Error Tracking

EvoClaw introduces a detailed evaluation suite focused on both per-milestone accuracy and cumulative error management.

Let $m$ denote a milestone:

• $N_{\text{required},m}$: number of F2P tests
• $N_{\text{fixed},m}$: number of F2P tests passed by the agent
• $N_{\text{broken},m}$: number of P2P tests broken by the agent
• $\epsilon = 1$ (smoothing)

The following metrics are computed:

• Recall$_m$ = $N_{\text{fixed},m} / N_{\text{required},m}$
• Precision$_m$ = $$(N_{\text{fixed},m} + \epsilon) / (N_{\text{fixed},m} + N_{\text{broken},m} + \epsilon)$$
• Score$_m$ = $N_{\text{required},m}$0
• Overall Score = $N_{\text{required},m}$1
• Resolve Rate: fraction of milestones on which all F2P and P2P tests pass

Error chains are tracked for each test transitioning from pass to fail, with propagation classified as P0 Root Cause (initial failure), P1 Inherited (downstream dependency), PX Missing (skipped test), or PH Healed (subsequent fix). A reviewer agent attributes root causes, finding Logic Errors (∼57%) most prevalent, in addition to Omission and Extraneous changes. This scheme enables measurement of error accumulation: regular regression violates system integrity, while regressing test curves (fitted as $N_{\text{required},m}$2) distinguish linear feature growth (Recall) from rapid saturation (Precision) (Deng et al., 13 Mar 2026).

4. Agent Frameworks and Experimental Outcomes

EvoClaw evaluates 12 frontier LLM agents via four distinct agentic frameworks, all in terminal-based interactive mode:

Framework	Example Models
Claude Code	Claude Opus 4.5/4.6, Claude Sonnet 4.5/4.6
Codex CLI	GPT 5.2, GPT 5.2-Codex, GPT 5.3-Codex
Gemini CLI	Gemini 3 Pro, Gemini 3.1 Pro, Gemini 3 Flash
OpenHands	MiniMax M2.5, Kimi K2.5, GPT 5.3-Codex, Gemini 3 Flash, Claude Opus 4.6

A striking performance drop is observed between independent and continuous task settings. While independent settings (stateless, each milestone reset) yield top overall Score above 80%, continuous (dependency- and stateful) settings yield best Scores of 38.03% (OpenHands+Claude Opus 4.6) and Resolve Rates of at most 13.37% (Gemini 3 Pro). Per-repository drops are substantial: e.g., Claude Sonnet 4.6 achieves 93.2% on scikit-learn in independent settings but only 21.1% under continuous evolution.

Behavioral analysis reveals “precision saturation”—agents repeatedly break pre-existing tests as they add new features. Failure modes include logic error dominance, cascading inherited errors, and behavioral extremes: “blind thrashing” (high edit churn, few verifications) and excessive verification (little editing, many test runs). Optimal regimes balance moderate editing with disciplined verification (Deng et al., 13 Mar 2026).

5. Continuous Software Evolution Dynamics

EvoClaw’s results expose critical limitations in agentic code evolution. State-of-the-art agents maintain near-linear Recall as tasks accrue, but cumulative Precision saturates, indicating that while agents can continue adding new features, they increasingly fail to avoid regressions. This “snowball effect” of unresolved regressions ultimately stalls further development. The contrast between stateless and stateful settings underscores agents’ current inability to robustly handle temporal dependencies, system-wide side effects, and technical debt.

Error chain analysis identifies early failures as disproportionately impactful: inherited propagation (P1) rates climb in later milestones, and logic errors are both frequent and persistent. System integrity is enforced strictly by requiring that all previously passing P2P tests remain passing (regression prevention) and that new F2P tests transition from fail to pass (new functionality). These constraints, representing real-world software evolution demands, are not satisfied by most current models under continuous evaluation (Deng et al., 13 Mar 2026).

6. Insights, Implications, and Future Directions

EvoClaw demonstrates a fundamental gap in long-horizon software evolution: agents excel at isolated coding objectives but fail to sustain system quality over persistent, dependency-driven timelines. Proactive repository exploration—such as leveraging read/search tools—and disciplined, on-the-fly test verification are strong predictors of agentic success. However, the overall inability to prevent regression and manage accumulating technical debt indicates a need for new agent designs.

Future directions highlighted include constructing long-horizon maintenance objectives that incorporate regression risk into agent reward functions, enhancing regression-aware code synthesis (treating passing tests as hard constraints), improving context management (e.g., through patch reversion or branching), and integrating structural program analysis (e.g., type systems, formal proofs) to capture regression risk beyond what can be caught by tests alone. Expansions to the benchmark—incorporating semantic, intent-driven DAGs and annotations from both humans and LLMs—are suggested to address the full complexity of software process dependencies (Deng et al., 13 Mar 2026).

EvoClaw and the DeepCommit pipeline are publicly released, providing code, data, and leaderboards, and constitute a rigorous platform for next-generation research on autonomous software evolution agents (Deng et al., 13 Mar 2026).