EvoClaw: Benchmark for Continuous Software Evolution

• EvoClaw is a benchmark for continuous software evolution that models persistent software maintenance through interdependent, temporally structured milestones.
• It uses DeepCommit to reconstruct verifiable Milestone DAGs from commit histories, ensuring dependency-aware and executable evolution sequences.
• The benchmark reveals that while AI agents can implement new features, they struggle with regression prevention, context retention, and long-term maintenance.

Searching arXiv for the cited EvoClaw benchmark paper and closely related context. EvoClaw is a benchmark for evaluating AI agents on continuous software evolution rather than isolated, stateless coding tasks. Introduced by Deng et al. as “EvoClaw: Evaluating AI Agents on Continuous Software Evolution” (Deng et al., 13 Mar 2026), it operationalizes software maintenance as a persistent, temporally dependent process in which earlier code changes constrain later work, regressions can accumulate, and system integrity must be preserved over time. Within the broader discussion of agentic systems in software, EvoClaw functions as an empirical stress test for long-horizon development: it contrasts sharply with benchmarks centered on one-off issue resolution and provides evidence that frontier agents retain substantial feature-implementation capability yet struggle with regression prevention, context retention, and technical-debt control under continuous evolution (Cao, 4 Jun 2026).

1. Benchmark definition and conceptual scope

EvoClaw is explicitly a benchmark and evaluation framework, not an agent architecture or development methodology. Its purpose is to measure how AI coding agents perform when they must continuously evolve and maintain software across a sequence of dependent development milestones, rather than solve independent tasks on reset code snapshots (Deng et al., 13 Mar 2026). The central premise is that real software work is temporally structured: implementation decisions propagate forward, dependency constraints accumulate, and local shortcuts can create future maintenance burdens.

This design differentiates EvoClaw from function-completion benchmarks, issue-resolution benchmarks such as SWE-bench and SWE-bench Verified, and release-level or project-level code generation settings that do not encode persistent, dependency-constrained histories (Deng et al., 13 Mar 2026). EvoClaw targets what its authors describe as dimensions largely missing from current evaluations: sustaining system integrity, limiting error accumulation, and handling long-term maintenance under dependency chains.

A milestone in EvoClaw is defined as a semantically cohesive development goal, more meaningful than a single commit and more granular than a full release. Milestones are organized into a Directed Acyclic Graph, or Milestone DAG, whose edges encode inter-milestone dependencies and whose topology determines which tasks may proceed in parallel and which require prerequisites (Deng et al., 13 Mar 2026). This structure allows EvoClaw to represent real repository evolution as executable sequences rather than as static snapshots.

A plausible implication is that EvoClaw shifts the unit of analysis from “can an agent solve this task now?” to “can an agent preserve correctness while modifying a codebase whose present state is partly the result of its own prior actions?” That shift is foundational to its research significance.

2. DeepCommit and the construction of executable evolution sequences

EvoClaw is built on an agentic pipeline called DeepCommit, which reconstructs verifiable Milestone DAGs from noisy commit histories (Deng et al., 13 Mar 2026). DeepCommit transforms linear version-control traces into structured, testable software-evolution itineraries. This reconstruction is necessary because raw commit logs do not directly provide semantically coherent, dependency-aware evaluation units.

The pipeline proceeds in three phases. First, commit history preprocessing selects relevant ranges using a branch-out/first-parent strategy, filters non-source-file changes and other noise, and extracts static signals such as commit-level dependencies, symbol-level changes, and file-level co-change statistics. Second, Milestone DAG construction identifies seed commits, consolidates related commits into milestones, infers dependency edges, and refines milestone granularity. Third, runtime environment resolution reconstructs repository states in topological order, prepares executable environments, and collects test signals for evaluation (Deng et al., 13 Mar 2026).

Granularity refinement is governed by a balancing strategy using the coefficient of variation,

$CV = \text{std}/\text{mean},$

with target $CV < 1.0$. Oversized milestones are decomposed and underspecified ones are merged according to thresholds reported in the paper (Deng et al., 13 Mar 2026). Dependency edges may be strong or weak, depending on whether missing upstream work causes downstream build or test failures or only graceful degradation.

The resulting benchmark artifacts are executable. For each milestone, DeepCommit defines START and END states, reconstructs repository snapshots, prepares Dockerized environments, and collects stable transitions in test outcomes. Quality-assurance properties reported for the reconstructed graphs include 100% commit coverage, confirmed DAG acyclicity, an 87.1% test collection rate, and environment-induced errors that are negligible at $\le 0.10\%$ (Deng et al., 13 Mar 2026).

This construction matters because EvoClaw’s claims rest not simply on repository mining but on verifiable execution. The benchmark is therefore not merely historical replay; it is a reproducible testbed for persistent agent behavior.

3. Evaluation protocol, metrics, and formalism

EvoClaw evaluates agents under two contrasting regimes: Continuous Task Evaluation and Independent Task Evaluation (Deng et al., 13 Mar 2026). In the continuous regime, milestones are streamed according to the Milestone DAG scheduler, a persistent codebase is evolved across milestones, and each submission is snapshotted and evaluated in isolation while the agent continues working. In the independent regime, each milestone is instead presented on its canonical snapshot, removing the effects of prior agent-authored modifications.

Acceptance is tied to two types of test signals. Fail-to-Pass (F2P) tests verify newly introduced functionality, while Pass-to-Pass (P2P) tests enforce regression preservation (Deng et al., 13 Mar 2026). The benchmark therefore measures both feature implementation and maintenance reliability.

EvoClaw separates these dimensions through Recall and Precision, then combines them in a harmonic-mean Score. The paper provides the following formulas:

$$\text{Recall}_m = \frac{N_{\text{fixed}, m}}{N_{\text{requried}, m}}$$

$$\text{Precision}_m = \frac{N_{\text{fixed}, m} + \epsilon}{N_{\text{fixed}, m} + N_{\text{broken}, m} + \epsilon}$$

where $N_{\text{fixed},m}$ counts F2P tests fixed, $N_{\text{broken},m}$ counts P2P tests that regress, and $\epsilon = 1$ is a smoothing factor. The milestone score is

$$\text{Score}_m = \frac{2 \cdot \text{Precision}_m \cdot \text{Recall}_m}{\text{Precision}_m + \text{Recall}_m}$$

and the final reported benchmark metric is

$$\text{Score} = \frac{1}{|M|} \sum_{m \in M} \text{Score}_m.$$

The benchmark also reports Resolve Rate, defined as the fraction of milestones for which all associated F2P and P2P tests pass (Deng et al., 13 Mar 2026). Evolution dynamics are analyzed using a saturation model,

$CV < 1.0$0

to compare continuous and independent trajectories.

The underlying agentic setting is naturally related to the iterative loop formalized in the broader agentic-systems literature:

$CV < 1.0$1

where an LLM $CV < 1.0$2 chooses actions conditioned on state $CV < 1.0$3 and memory $CV < 1.0$4 (Cao, 4 Jun 2026). EvoClaw stresses this loop under persistence and error accumulation rather than under single-turn execution.

4. Dataset composition and experimental setup

EvoClaw draws from seven high-impact open-source repositories across five languages: Go, Rust, Java, TypeScript, and Python (Deng et al., 13 Mar 2026). The repositories listed in the paper are zeromicro/go-zero, apache/dubbo, BurntSushi/ripgrep, nushell/nushell, element-hq/element-web, navidrome/navidrome, and scikit-learn/scikit-learn. Release ranges were selected to capture non-linear development and rich dependency structure.

The benchmark contains 98 human-verified graded milestones, plus 3 non-graded milestones for continuity, and 124 inter-milestone dependencies (Deng et al., 13 Mar 2026). The average milestone modifies 27.4 files, the mean SRS length is 1,348 words, and gold patch sizes range from less than 100 to more than 1,500 LOC, with an average of 570 LOC. Repositories range from 9 to 23 milestones.

The evaluation reported in the EvoClaw paper covers 12 frontier models across 4 agent frameworks (Deng et al., 13 Mar 2026). The frameworks and model families are as follows.

Framework	Reported models
Claude Code (v2.1.50)	Claude Sonnet 4.5, Claude Opus 4.5, Claude Sonnet 4.6, Claude Opus 4.6
Codex CLI (v0.105.0)	GPT 5.2, GPT 5.2-Codex, GPT 5.3-Codex
Gemini CLI (v0.29.5)	Gemini 3 Pro, Gemini 3.1 Pro, Gemini 3 Flash
OpenHands (v1.2.1)	Claude Opus 4.6, GPT 5.3-Codex, Gemini 3 Flash, Kimi K2.5, MiniMax M2.5

The runtime environment is standardized via sandboxed Docker containers, with isolated evaluation containers per snapshot and verified builds and test collection in both START and END states (Deng et al., 13 Mar 2026). Agents have full shell access and use standard development operations such as reading, searching, editing code, building, and running tests. OpenHands is reported to allow up to 3,000 iterations per milestone (Deng et al., 13 Mar 2026).

This setup is notable because the benchmark does not reduce agent performance to raw model capability alone. It explicitly exposes the interaction among model, framework, memory/compaction behavior, persistent environment, and verification loop.

5. Empirical findings and the continuous-evolution performance cliff

The central empirical finding of EvoClaw is a large degradation from isolated to continuous settings. The paper reports that overall performance scores drop from greater than $CV < 1.0$5 on isolated tasks to at most $CV < 1.0$6 in continuous settings, exposing what Deng et al. describe as agents’ “profound struggle with long-term maintenance and error propagation” (Deng et al., 13 Mar 2026). In the synthetic summary reused by the software-paradigm analysis, this contrast is condensed as 82% for isolated tasks, at most 38% for continuous evolution, and a reported drop of $CV < 1.0$7 (Cao, 4 Jun 2026).

The benchmark’s best continuous Score is 38.03% for Claude Opus 4.6 in OpenHands, with Claude Opus 4.6 in Claude Code reaching 36.29% (Deng et al., 13 Mar 2026). Resolve Rates are much lower; the paper reports examples such as Gemini 3 Pro at 13.37% (Deng et al., 13 Mar 2026). These figures indicate that partial task progress often remains possible even when full milestone resolution becomes rare.

The decomposition into Recall and Precision is crucial. EvoClaw reports that Recall grows near-linearly across milestones, indicating that agents continue to implement new features, whereas Precision saturates rapidly, indicating a failure to preserve existing behavior as the codebase evolves (Deng et al., 13 Mar 2026). In other words, the bottleneck is not simply inability to write new code, but inability to write new code without damaging prior functionality.

Several additional regularities reinforce this conclusion. Larger gold patch LOC monotonically lowers scores; later execution order and deeper DAG layers reduce both Score and Resolve Rate; and continuous trajectories fit saturation ceilings that independent trajectories do not exhibit over the observed window (Deng et al., 13 Mar 2026). The paper also reports that Logic Error accounts for approximately 57% of root causes in error-chain analysis, with inherited failures and missing test execution events increasingly dominating later milestones (Deng et al., 13 Mar 2026).

These findings support the interpretation advanced in “The End of Software Engineering,” which juxtaposes EvoClaw with SWE-bench Verified and LangChain’s multi-agent coordination study. In that framing, isolated-task benchmarks demonstrate strong bounded competence, multi-agent orchestration demonstrates organizational productivity gains, and EvoClaw reveals a distinct axis on which contemporary systems remain fragile: temporal continuity and cumulative maintenance (Cao, 4 Jun 2026).

6. Failure modes, comparative significance, and implications for agentic engineering

EvoClaw highlights four persistent challenges: context drift, error propagation, technical debt awareness, and verification fidelity (Cao, 4 Jun 2026). These are not incidental benchmark artifacts but recurrent long-horizon failure modes.

Context drift refers to the loss of coherent understanding as codebases and histories exceed effective context windows. Agents progressively lose track of global invariants and dependencies (Cao, 4 Jun 2026).

Error propagation refers to the snowball effect by which small early mistakes contaminate later work. EvoClaw’s continuous setting makes this visible because downstream milestones operate on the agent’s modified codebase rather than on clean snapshots (Deng et al., 13 Mar 2026).

Technical debt awareness concerns the tendency to optimize for immediate task completion rather than maintainability. The benchmark shows that passing immediate tests can coexist with latent architectural degradation (Cao, 4 Jun 2026).

Verification fidelity concerns insufficient test coverage or narrow test scopes that allow semantically incorrect changes to survive until novel inputs or later milestones expose them (Cao, 4 Jun 2026).

The case studies reported in the benchmark illustrate these phenomena concretely. In apache/dubbo, a responsibility boundary is misplaced, producing superficial test success while violating resolver-chain contracts. In nushell, shotgun fixes and API signature degradation satisfy immediate conditions while introducing architectural debt and harming future evolution (Deng et al., 13 Mar 2026). These examples suggest that EvoClaw measures not only pass/fail competence but also alignment with architectural intent.

Within the broader paradigm argument of agentic software, EvoClaw is used as evidence that long-term maintenance remains a gating problem for Agent-as-a-Service and for the emergence of Agentic Engineering as a distinct discipline (Cao, 4 Jun 2026). The same work connects this difficulty to a general complexity scaling claim,

$CV < 1.0$8

interpreting sustained maintenance as intrinsically hard because interacting dependencies generate exponentially many possible interaction paths (Cao, 4 Jun 2026). This suggests that benchmarks like EvoClaw are not merely exposing current model weakness but probing a structurally demanding regime of software work.

A common misconception is to treat strong results on isolated coding benchmarks as evidence that autonomous software maintenance is nearly solved. EvoClaw contradicts that interpretation. It indicates that feature implementation ability and long-horizon maintenance reliability are separable capabilities, and that the latter currently lags far behind the former (Deng et al., 13 Mar 2026).

7. Reproducibility, limitations, and research outlook

EvoClaw is designed as a reproducible benchmark. The paper reports public code, data, and a leaderboard: code at the benchmark repository, data at a public dataset release, and a dedicated leaderboard site (Deng et al., 13 Mar 2026). Evaluation uses Dockerized environments, isolated testbeds, language-appropriate test collection, and flaky-test filtering. A single full evaluation is reported to cost approximately \$500 with frontier models such as Claude Opus 4.5 (Deng et al., 13 Mar 2026).

The benchmark’s limitations are also explicit. Its data cover seven repositories, five languages, and 98 graded milestones, which is sufficient to reveal strong trends but remains limited in representativeness (Deng et al., 13 Mar 2026). Some process-level changes, such as build-system modernization, CI or release engineering, and cross-module intent-defined work, are preferentially filtered when they lack executable F2P signals or form weakly connected subgraphs. DeepCommit’s topological construction can fragment semantically unified work, as illustrated by the scikit-learn case study comparing human and machine-derived DAGs (Deng et al., 13 Mar 2026). Test coverage and flakiness remain residual concerns despite the reported collection and pass-to-fail statistics.

Future work identified in the paper includes expanding to more domains and languages, increasing DAG length and breadth, improving modeling of process-level non-functional edges, integrating real-world live environments, and leveraging stronger LLMs to harvest richer itineraries (Deng et al., 13 Mar 2026). The benchmark therefore functions as both an evaluation instrument and a research agenda: it exposes where current agents fail and indicates which capabilities—shared memory, robust verification, recovery mechanisms, orchestration, and maintenance-aware design—require further development (Cao, 4 Jun 2026).

In sum, EvoClaw occupies a distinctive position in the contemporary evaluation landscape. It captures the “second half” of software work—maintenance and evolution—through temporally dependent, verifiable milestone sequences mined from real repositories (Deng et al., 13 Mar 2026). Its principal contribution is not merely a lower score for current agents, but a redefinition of what meaningful coding-agent evaluation must include if the goal is autonomous long-running software development rather than episodic code generation.