---
title: 'EvoAgentBench: LLM Agent Self-Evolution Benchmark'
url: https://www.emergentmind.com/topics/evoagentbench
type: topic
---

# EvoAgentBench: LLM Agent Self-Evolution Benchmark

EvoAgentBench is a benchmark for agent self-evolution via Ability-guided transfer. It is designed for long-horizon LLM agents whose useful experience is not merely stored information but reusable procedures for searching, debugging, verification, and execution control. In contrast to standard agent benchmarks, which usually measure single-episode task solving, and memory benchmarks, which emphasize information retention, EvoAgentBench isolates whether prior execution traces can be distilled into procedural knowledge and then transferred to later, related tasks. Its evaluation target is therefore not generic benchmark accuracy but the mechanics of experience encoding, routing, and uptake across four agentic domains: web research, algorithmic reasoning, software engineering, and knowledge work [2607.05202].

## 1. Problem formulation and benchmark objective

EvoAgentBench is motivated by a specific gap in LLM-agent evaluation. Standard agent benchmarks mostly test whether an agent can solve a held-out task in one episode. Memory benchmarks, by contrast, focus on recall, retrieval, personalization, or long-context retention. The benchmark argues that self-evolution depends on a different phenomenon: whether prior execution traces can be converted into reusable procedural ability and then applied to new tasks.

The central evaluative question is whether prior agent experience becomes reusable procedural ability, and whether that ability improves future performance when it is routed to the correct test task. The benchmark therefore separates three components of transfer. **Experience encoding** asks whether the method extracted the correct reusable procedure. **Routing** asks whether the procedure was retrieved and applied to the appropriate future task. **Uptake** asks whether the base agent actually benefited from the transferred content.

This framing distinguishes EvoAgentBench from both conventional held-out evaluation and generic memory testing. It does not ask whether the model remembers facts or prior conversations. It asks whether the model can reuse operations such as a search strategy, a debugging pattern, a verification workflow, or a repository repair procedure. A common misconception is to treat it as a standard train/test benchmark with a procedural theme; the benchmark is explicitly constructed so that every test task is ability-supported by training-side evidence, making transfer rather than random generalization the object of measurement [2607.05202].

## 2. Trace-grounded Abilities and the extraction pipeline

The benchmark is organized around the notion of an **Ability**, defined as a trace-grounded reusable operation with explicit applicability conditions. An Ability is neither a topic label nor a dataset category. It is an operational unit extracted from actual agent behavior.

Its construction pipeline has three stages: **Trace collection**, **Ability extraction and canonicalization**, and **Ability graph construction and constrained splitting**. For each task \(x\), EvoAgentBench collects multi-backbone traces under **no-skill** conditions so that the observed procedures arise from raw agent behavior rather than injected assistance. The trace pool is defined as

\[
\mathcal{T}(x)=\{(\tau_{x,b}, o_{x,b}, \nu_{x,b}) : b\in\mathcal{B}_{\mathrm{trace}\}. 
\]

The construction backbones are **Kimi-K2.5**, **GLM-5.1**, and **DeepSeek-V3.2**.

Given task query \(q_x\), ground truth \(y_x^\star\), and trace pool \(\mathcal{T}(x)\), the extractor produces raw Ability cards as

\[
\mathcal{A}_{\mathrm{raw}(x)} = E_\phi\bigl(q_x, y_x^\star, \mathcal{T}(x)\bigr).
\]

Each raw Ability card has the form

\[
a=(\gamma_a,\pi_a,\mathcal{E}_a,\partial_a,\rho_a),
\]

where \(\gamma_a\) is the trigger condition, \(\pi_a\) is the reusable procedure, \(\mathcal{E}_a\) is the evidence linking the card to trajectories, outputs, verifier results, and target, \(\partial_a\) is the applicability boundary, and \(\rho_a\) is the role.

The role system has three categories. **Method** denotes the primary task-solving procedure. **Guard** denotes a correction against recurring invalid behavior. **Workflow** denotes an execution-control procedure, such as validation or verification. The applicability boundary is a central design choice: it prevents overly broad abstractions from being treated as valid transfer units. The benchmark explicitly rejects vague advice such as “search better” as insufficiently operationally specific. Extraction compares successful and failed attempts across multiple backbones in order to identify recurring procedures and failure modes rather than model-specific quirks [2607.05202].

## 3. Canonicalization and the Ability Graph

Raw Ability cards are task-local and noisy, so EvoAgentBench canonicalizes them into shared operational units. For each domain, raw cards are first blocked by embedding similarity, and candidate pairs are selected if cosine similarity exceeds a domain threshold \(\theta_d\). The benchmark emphasizes that similarity is only a pre-filter, not a merge rule.

A pair is merged only if it satisfies operational equivalence: the same role type, compatible triggers, equivalent reusable procedures, the same success mechanism or correction target, and compatible applicability boundaries. Adjudication combines **three independent LLM adjudicators** with expert review. If all three adjudicators agree, the decision is accepted automatically; otherwise, a domain expert reviews it. Outcomes are **merge** or **cannot-link**.

Merges are not transitively closed by default. The benchmark explicitly states that \(a_i \sim a_j\) and \(a_j \sim a_k\) does not imply that all three belong to one Ability. Connected components are checked for internal consistency, and incompatible components are split by experts. The resulting canonical Ability unit is

\[
u=(\mathcal{R}_u,\Gamma_u,\Pi_u,\mathcal{E}_u,\partial_u,\rho_u,\mathcal{X}_u),
\]

where \(\mathcal{R}_u\) is the member raw-card set, \(\Gamma_u\) the canonical trigger, \(\Pi_u\) the canonical procedure, \(\mathcal{E}_u\) aggregated evidence, \(\partial_u\) the applicability boundary, \(\rho_u\) the role, and \(\mathcal{X}_u\) the supporting task set.

These canonical units are then mapped back to tasks. For domain \(d\), with canonical Ability set \(\mathcal{U}_d\), the abilities of task \(x\) are

\[
A(x)=\{u\in\mathcal{U}_d : x\in\mathcal{X}_u\}.
\]

Only **edge-eligible** abilities can induce transfer edges: a unit must be supported by at least two tasks and must remain operationally specific after merging. If \(A^+(x)\subseteq A(x)\) denotes the edge-eligible abilities of task \(x\), the domain graph is

\[
G_d=(V_d,E_d),
\]

with \((x_i,x_j)\in E_d\) iff

\[
A^+(x_i)\cap A^+(x_j)\neq\varnothing.
\]

The resulting **Ability Graph** defines task relatedness through shared transferable procedures rather than topic overlap or benchmark metadata. This graph is the benchmark’s main structural object and is the basis for its split protocol and transfer analysis [2607.05202].

## 4. Domains, source tasks, and constrained split construction

EvoAgentBench covers four long-horizon agentic domains. **Web research** is sourced from **BrowseComp-Plus** and features abilities such as search strategy, evidence verification, and information triangulation. **Algorithmic reasoning** is sourced from **LiveCodeBench** and includes algorithm selection, complexity-aware reasoning, and debugging anti-pattern correction. **Software engineering** is sourced from **SWE-Bench Verified** and includes repository-level patching, regression fixing, and test-driven repair workflows. **Knowledge work** is sourced from **GDPVal** and emphasizes artifact construction, answer validation, and structured deliverable workflows [2607.05202].

The benchmark begins from **2,605 source tasks**: **830** from BrowseComp-Plus, **500** from SWE-Bench Verified, **1,055** from LiveCodeBench V6, and **220** from the GDPVal gold subset. Trace-grounded extraction yields **7,326 raw Ability cards** from **2,516 tasks** that produced at least one extractable card. Canonicalization merges these into **170 canonical Ability units**. The full retained Ability graph contains **1,108 tasks** and **56 Ability communities**.

The main evaluation split consists of **528 training tasks** and **267 test tasks**. Its defining constraint is verified training-side Ability support for every test task. In the paper’s notation, a test task \(t\) is allowed only if

\[
\exists f\in F(t) \text{ such that } |\{t' \in D_{\mathrm{train}} : f \in F(t')\}| > 0.
\]

In plain terms, every test task must share at least one canonical Ability family with training tasks, every test-side Ability family must also appear in training, and no test task is unsupported. The reported result is **zero unsupported test tasks**. Average train-side support per test task is capped at 10, with **Supp@10 = 8.34** overall.

This split design is central to the benchmark’s interpretation. A positive or negative transfer result cannot be dismissed as a consequence of arbitrary data partitioning, because the protocol guarantees that each test task has nontrivial procedural support on the training side. A plausible implication is that remaining failures can be localized more cleanly to extraction quality, routing quality, or agent uptake rather than to unsupported evaluation instances [2607.05202].

## 5. Evaluation protocol, methods, and transfer metric

EvoAgentBench evaluates methods across **two scaffolds** and **three backbones**, yielding **six scaffold–backbone settings**. The scaffolds are **OpenClaw**, a full-featured modular framework with **26 built-in tools**, and **Nanobot**, a lightweight framework with **7 core tools**. The backbones are **Qwen3.5-27B**, **Qwen3.5-397B**, and **Gemma-4-31B**.

The benchmark compares five method classes. **Vanilla** is the no-evolution baseline. **Memento** retrieves the closest retained training case. **ReasoningBank** retrieves distilled reasoning-memory items. **GEPA** evolves a prompt on training data and broadcasts it. **Anchor Skill** is a diagnostic reference that uses curator-side ability labels and curated skill routing. Anchor is not deployable because those labels are unavailable to automatic methods, but its content is built only from training-side evidence.

The principal metric is average transfer gain \(\Delta\), defined as improvement relative to **Vanilla** on the test split, where \(r_0(x)\) is the no-evolution baseline score and \(r_m(x)\) is the score after applying method \(m\). The benchmark’s interpretation of “positive gain across settings” is correspondingly strict: a setting is positive only if transfer gain is above zero relative to Vanilla. This criterion is evaluated across domains, scaffolds, and backbones rather than on a single aggregate average.

The protocol therefore measures not only whether a method improves performance, but whether it does so robustly under changes in backbone and agent substrate. This is important because the benchmark’s stated goal is diagnosis of self-evolution mechanisms, not merely ranking a set of retrieval or prompting heuristics by one pooled accuracy number [2607.05202].

## 6. Empirical findings, failure modes, and position within the broader evolution-benchmark literature

The main empirical result is that curated Ability content transfers reliably across model families, but no current automatic method sustains positive gain in all settings. **Anchor Skill** improves in **every cell**, which shows that the extracted Ability content itself is genuinely transferable when delivered correctly. By contrast, all automatic methods exhibit at least one negative cell. **Memento** can be strongly negative in some settings, indicating that surface-similarity retrieval can mismatch badly. **ReasoningBank** is usually positive on average but still has several negative cells. **GEPA** has the strongest automatic average in some settings, yet it also regresses in certain domains and scaffold–backbone combinations [2607.05202].

These findings support the benchmark’s claim that the bottleneck is method-side extraction and routing rather than the existence of transferable procedural content. Because tasks, tools, timeouts, and scoring are fixed across methods, the gap between Anchor and automatic systems points to failures in extracting the correct reusable procedure, indexing or routing it to the correct future task, or getting the test-time agent to apply it effectively. The benchmark also reports that cost and accuracy are decoupled, and that some methods increase turn count without improving performance.

EvoAgentBench occupies a specific niche within a broader family of “evolution” benchmarks, and conflating these settings is a recurrent source of confusion. In **SWE-EVO**, the target is long-horizon **software evolution over time**: an agent receives a pre-release codebase plus a release-note-style specification and must evolve the system to match the next tagged version while preserving behavior [2512.18470]. In **EvoClaw**, the target is **continuous software evolution** under persistent state, technical debt, and milestone dependencies, with sharp performance degradation from isolated to continuous evaluation [2603.13428]. In **ProEvolve**, the focus is **programmable evolution of tool-using environments** through graph transformations such as completion, saturation, and deprecation, producing environment trajectories rather than ability-transfer splits [2603.05910]. In **Evo-Memory**, the emphasis is **self-evolving memory** over task streams, where the agent retrieves, integrates, and updates memory continuously during deployment [2511.20857].

Against this background, EvoAgentBench is narrower and more diagnostic. It is not principally a benchmark of codebase evolution, environment drift, or persistent transcript memory. It is a benchmark of whether past trajectories become operationally reusable procedures, whether those procedures can be routed to appropriate future tasks, and whether the agent can exploit them at test time. This suggests that its principal contribution is methodological: it converts self-evolution from a coarse claim of post hoc improvement into a structured evaluation of procedural reuse, with explicit support constraints and graph-defined transfer relations [2607.05202].

Source: https://www.emergentmind.com/topics/evoagentbench