EvoAgentBench: LLM Agent Self-Evolution Benchmark

• EvoAgentBench is a benchmark that assesses long-horizon LLM agent self-evolution by converting execution traces into reusable procedural abilities.
• It uniquely evaluates experience encoding, routing, and uptake to diagnose whether prior operational procedures can be effectively transferred to new tasks.
• Designed across domains like web research, algorithmic reasoning, software engineering, and knowledge work, it ensures robust training support and precise transfer analysis.

EvoAgentBench is a benchmark for agent self-evolution via Ability-guided transfer. It is designed for long-horizon LLM agents whose useful experience is not merely stored information but reusable procedures for searching, debugging, verification, and execution control. In contrast to standard agent benchmarks, which usually measure single-episode task solving, and memory benchmarks, which emphasize information retention, EvoAgentBench isolates whether prior execution traces can be distilled into procedural knowledge and then transferred to later, related tasks. Its evaluation target is therefore not generic benchmark accuracy but the mechanics of experience encoding, routing, and uptake across four agentic domains: web research, algorithmic reasoning, software engineering, and knowledge work (Gao et al., 6 Jul 2026).

1. Problem formulation and benchmark objective

EvoAgentBench is motivated by a specific gap in LLM-agent evaluation. Standard agent benchmarks mostly test whether an agent can solve a held-out task in one episode. Memory benchmarks, by contrast, focus on recall, retrieval, personalization, or long-context retention. The benchmark argues that self-evolution depends on a different phenomenon: whether prior execution traces can be converted into reusable procedural ability and then applied to new tasks.

The central evaluative question is whether prior agent experience becomes reusable procedural ability, and whether that ability improves future performance when it is routed to the correct test task. The benchmark therefore separates three components of transfer. Experience encoding asks whether the method extracted the correct reusable procedure. Routing asks whether the procedure was retrieved and applied to the appropriate future task. Uptake asks whether the base agent actually benefited from the transferred content.

This framing distinguishes EvoAgentBench from both conventional held-out evaluation and generic memory testing. It does not ask whether the model remembers facts or prior conversations. It asks whether the model can reuse operations such as a search strategy, a debugging pattern, a verification workflow, or a repository repair procedure. A common misconception is to treat it as a standard train/test benchmark with a procedural theme; the benchmark is explicitly constructed so that every test task is ability-supported by training-side evidence, making transfer rather than random generalization the object of measurement (Gao et al., 6 Jul 2026).

2. Trace-grounded Abilities and the extraction pipeline

The benchmark is organized around the notion of an Ability, defined as a trace-grounded reusable operation with explicit applicability conditions. An Ability is neither a topic label nor a dataset category. It is an operational unit extracted from actual agent behavior.

Its construction pipeline has three stages: Trace collection, Ability extraction and canonicalization, and Ability graph construction and constrained splitting. For each task $x$, EvoAgentBench collects multi-backbone traces under no-skill conditions so that the observed procedures arise from raw agent behavior rather than injected assistance. The trace pool is defined as

$\mathcal{T}(x)=\{(\tau_{x,b}, o_{x,b}, \nu_{x,b}) : b\in\mathcal{B}_{\mathrm{trace}\}.$

The construction backbones are Kimi-K2.5, GLM-5.1, and DeepSeek-V3.2.

Given task query $q_x$, ground truth $y_x^\star$, and trace pool $\mathcal{T}(x)$, the extractor produces raw Ability cards as

$$\mathcal{A}_{\mathrm{raw}(x)} = E_\phi\bigl(q_x, y_x^\star, \mathcal{T}(x)\bigr).$$

Each raw Ability card has the form

$$a=(\gamma_a,\pi_a,\mathcal{E}_a,\partial_a,\rho_a),$$

where $\gamma_a$ is the trigger condition, $\pi_a$ is the reusable procedure, $\mathcal{E}_a$ is the evidence linking the card to trajectories, outputs, verifier results, and target, $\mathcal{T}(x)=\{(\tau_{x,b}, o_{x,b}, \nu_{x,b}) : b\in\mathcal{B}_{\mathrm{trace}\}.$0 is the applicability boundary, and $\mathcal{T}(x)=\{(\tau_{x,b}, o_{x,b}, \nu_{x,b}) : b\in\mathcal{B}_{\mathrm{trace}\}.$1 is the role.

The role system has three categories. Method denotes the primary task-solving procedure. Guard denotes a correction against recurring invalid behavior. Workflow denotes an execution-control procedure, such as validation or verification. The applicability boundary is a central design choice: it prevents overly broad abstractions from being treated as valid transfer units. The benchmark explicitly rejects vague advice such as “search better” as insufficiently operationally specific. Extraction compares successful and failed attempts across multiple backbones in order to identify recurring procedures and failure modes rather than model-specific quirks (Gao et al., 6 Jul 2026).

3. Canonicalization and the Ability Graph

Raw Ability cards are task-local and noisy, so EvoAgentBench canonicalizes them into shared operational units. For each domain, raw cards are first blocked by embedding similarity, and candidate pairs are selected if cosine similarity exceeds a domain threshold $\mathcal{T}(x)=\{(\tau_{x,b}, o_{x,b}, \nu_{x,b}) : b\in\mathcal{B}_{\mathrm{trace}\}.$2. The benchmark emphasizes that similarity is only a pre-filter, not a merge rule.

A pair is merged only if it satisfies operational equivalence: the same role type, compatible triggers, equivalent reusable procedures, the same success mechanism or correction target, and compatible applicability boundaries. Adjudication combines three independent LLM adjudicators with expert review. If all three adjudicators agree, the decision is accepted automatically; otherwise, a domain expert reviews it. Outcomes are merge or cannot-link.

Merges are not transitively closed by default. The benchmark explicitly states that $\mathcal{T}(x)=\{(\tau_{x,b}, o_{x,b}, \nu_{x,b}) : b\in\mathcal{B}_{\mathrm{trace}\}.$3 and $\mathcal{T}(x)=\{(\tau_{x,b}, o_{x,b}, \nu_{x,b}) : b\in\mathcal{B}_{\mathrm{trace}\}.$4 does not imply that all three belong to one Ability. Connected components are checked for internal consistency, and incompatible components are split by experts. The resulting canonical Ability unit is

$\mathcal{T}(x)=\{(\tau_{x,b}, o_{x,b}, \nu_{x,b}) : b\in\mathcal{B}_{\mathrm{trace}\}.$5

where $\mathcal{T}(x)=\{(\tau_{x,b}, o_{x,b}, \nu_{x,b}) : b\in\mathcal{B}_{\mathrm{trace}\}.$6 is the member raw-card set, $\mathcal{T}(x)=\{(\tau_{x,b}, o_{x,b}, \nu_{x,b}) : b\in\mathcal{B}_{\mathrm{trace}\}.$7 the canonical trigger, $\mathcal{T}(x)=\{(\tau_{x,b}, o_{x,b}, \nu_{x,b}) : b\in\mathcal{B}_{\mathrm{trace}\}.$8 the canonical procedure, $\mathcal{T}(x)=\{(\tau_{x,b}, o_{x,b}, \nu_{x,b}) : b\in\mathcal{B}_{\mathrm{trace}\}.$9 aggregated evidence, $q_x$0 the applicability boundary, $q_x$1 the role, and $q_x$2 the supporting task set.

These canonical units are then mapped back to tasks. For domain $q_x$3, with canonical Ability set $q_x$4, the abilities of task $q_x$5 are

$q_x$6

Only edge-eligible abilities can induce transfer edges: a unit must be supported by at least two tasks and must remain operationally specific after merging. If $q_x$7 denotes the edge-eligible abilities of task $q_x$8, the domain graph is

$q_x$9

with $y_x^\star$0 iff

$y_x^\star$1

The resulting Ability Graph defines task relatedness through shared transferable procedures rather than topic overlap or benchmark metadata. This graph is the benchmark’s main structural object and is the basis for its split protocol and transfer analysis (Gao et al., 6 Jul 2026).

4. Domains, source tasks, and constrained split construction

EvoAgentBench covers four long-horizon agentic domains. Web research is sourced from BrowseComp-Plus and features abilities such as search strategy, evidence verification, and information triangulation. Algorithmic reasoning is sourced from LiveCodeBench and includes algorithm selection, complexity-aware reasoning, and debugging anti-pattern correction. Software engineering is sourced from SWE-Bench Verified and includes repository-level patching, regression fixing, and test-driven repair workflows. Knowledge work is sourced from GDPVal and emphasizes artifact construction, answer validation, and structured deliverable workflows (Gao et al., 6 Jul 2026).

The benchmark begins from 2,605 source tasks: 830 from BrowseComp-Plus, 500 from SWE-Bench Verified, 1,055 from LiveCodeBench V6, and 220 from the GDPVal gold subset. Trace-grounded extraction yields 7,326 raw Ability cards from 2,516 tasks that produced at least one extractable card. Canonicalization merges these into 170 canonical Ability units. The full retained Ability graph contains 1,108 tasks and 56 Ability communities.

The main evaluation split consists of 528 training tasks and 267 test tasks. Its defining constraint is verified training-side Ability support for every test task. In the paper’s notation, a test task $y_x^\star$2 is allowed only if

$y_x^\star$3

In plain terms, every test task must share at least one canonical Ability family with training tasks, every test-side Ability family must also appear in training, and no test task is unsupported. The reported result is zero unsupported test tasks. Average train-side support per test task is capped at 10, with Supp@10 = 8.34 overall.

This split design is central to the benchmark’s interpretation. A positive or negative transfer result cannot be dismissed as a consequence of arbitrary data partitioning, because the protocol guarantees that each test task has nontrivial procedural support on the training side. A plausible implication is that remaining failures can be localized more cleanly to extraction quality, routing quality, or agent uptake rather than to unsupported evaluation instances (Gao et al., 6 Jul 2026).

5. Evaluation protocol, methods, and transfer metric

EvoAgentBench evaluates methods across two scaffolds and three backbones, yielding six scaffold–backbone settings. The scaffolds are OpenClaw, a full-featured modular framework with 26 built-in tools, and Nanobot, a lightweight framework with 7 core tools. The backbones are Qwen3.5-27B, Qwen3.5-397B, and Gemma-4-31B.

The benchmark compares five method classes. Vanilla is the no-evolution baseline. Memento retrieves the closest retained training case. ReasoningBank retrieves distilled reasoning-memory items. GEPA evolves a prompt on training data and broadcasts it. Anchor Skill is a diagnostic reference that uses curator-side ability labels and curated skill routing. Anchor is not deployable because those labels are unavailable to automatic methods, but its content is built only from training-side evidence.

The principal metric is average transfer gain $y_x^\star$4, defined as improvement relative to Vanilla on the test split, where $y_x^\star$5 is the no-evolution baseline score and $y_x^\star$6 is the score after applying method $y_x^\star$7. The benchmark’s interpretation of “positive gain across settings” is correspondingly strict: a setting is positive only if transfer gain is above zero relative to Vanilla. This criterion is evaluated across domains, scaffolds, and backbones rather than on a single aggregate average.

The protocol therefore measures not only whether a method improves performance, but whether it does so robustly under changes in backbone and agent substrate. This is important because the benchmark’s stated goal is diagnosis of self-evolution mechanisms, not merely ranking a set of retrieval or prompting heuristics by one pooled accuracy number (Gao et al., 6 Jul 2026).

6. Empirical findings, failure modes, and position within the broader evolution-benchmark literature

The main empirical result is that curated Ability content transfers reliably across model families, but no current automatic method sustains positive gain in all settings. Anchor Skill improves in every cell, which shows that the extracted Ability content itself is genuinely transferable when delivered correctly. By contrast, all automatic methods exhibit at least one negative cell. Memento can be strongly negative in some settings, indicating that surface-similarity retrieval can mismatch badly. ReasoningBank is usually positive on average but still has several negative cells. GEPA has the strongest automatic average in some settings, yet it also regresses in certain domains and scaffold–backbone combinations (Gao et al., 6 Jul 2026).

These findings support the benchmark’s claim that the bottleneck is method-side extraction and routing rather than the existence of transferable procedural content. Because tasks, tools, timeouts, and scoring are fixed across methods, the gap between Anchor and automatic systems points to failures in extracting the correct reusable procedure, indexing or routing it to the correct future task, or getting the test-time agent to apply it effectively. The benchmark also reports that cost and accuracy are decoupled, and that some methods increase turn count without improving performance.

EvoAgentBench occupies a specific niche within a broader family of “evolution” benchmarks, and conflating these settings is a recurrent source of confusion. In SWE-EVO, the target is long-horizon software evolution over time: an agent receives a pre-release codebase plus a release-note-style specification and must evolve the system to match the next tagged version while preserving behavior (Thai et al., 20 Dec 2025). In EvoClaw, the target is continuous software evolution under persistent state, technical debt, and milestone dependencies, with sharp performance degradation from isolated to continuous evaluation (Deng et al., 13 Mar 2026). In ProEvolve, the focus is programmable evolution of tool-using environments through graph transformations such as completion, saturation, and deprecation, producing environment trajectories rather than ability-transfer splits (Li et al., 6 Mar 2026). In Evo-Memory, the emphasis is self-evolving memory over task streams, where the agent retrieves, integrates, and updates memory continuously during deployment (Wei et al., 25 Nov 2025).

Against this background, EvoAgentBench is narrower and more diagnostic. It is not principally a benchmark of codebase evolution, environment drift, or persistent transcript memory. It is a benchmark of whether past trajectories become operationally reusable procedures, whether those procedures can be routed to appropriate future tasks, and whether the agent can exploit them at test time. This suggests that its principal contribution is methodological: it converts self-evolution from a coarse claim of post hoc improvement into a structured evaluation of procedural reuse, with explicit support constraints and graph-defined transfer relations (Gao et al., 6 Jul 2026).