EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer

Abstract: Agent self-evolution in long-horizon LLM systems is largely procedural: useful experience is not merely stored information, but reusable procedures for searching, debugging, and verification. Yet current evaluations do not isolate this form of transfer. Agent benchmarks test single-episode task solving; memory benchmarks target information retention rather than procedural reuse. We introduce EvoAgentBench, a benchmark for agent self-evolution via Ability-guided transfer across four agentic domains: web research, algorithmic reasoning, software engineering, and knowledge work. EvoAgentBench extracts trace-grounded Abilities from agent executions, canonicalizes them into operational units, and builds domain-specific Ability Graphs linking tasks that share procedural overlap. By design, every test task is backed by verified training-side Ability support. Across a 528/267 train/test split, two scaffolds, and three backbones, curated Ability content transfers reliably across model families, but no current automatic method sustains positive gain in all settings. EvoAgentBench shifts self-evolution evaluation from aggregate accuracy comparison to fine-grained diagnosis of experience encoding, routing, and uptake. The benchmark is publicly available at https://huggingface.co/datasets/EverMind-AI/EvoAgentBench.

• The paper's main contribution is designing a benchmark based on ability graphs to evaluate procedural self-evolution in LLM-based agents across four domains.
• It details a multi-backbone trace collection and ability extraction pipeline that isolates procedural knowledge transfer from common confounding factors.
• Experimental results indicate that while ideal skill injection yields consistent gains, current automatic evolution methods often suffer negative transfer in certain settings.

EvoAgentBench: A Benchmark for Agent Self-Evolution via Ability Transfer

Motivation and Context

The rapid advancement of LLM-based agents in complex, long-horizon domains has propelled interest in evaluating and improving their capacity for procedural self-evolution—specifically, their ability to convert prior task trajectories into reusable procedures that generalize to new, related tasks. Traditional benchmarks address episodic task-solving or information retention, but do not isolate procedural knowledge transfer. Existing self-evolution evaluations often confound skill quality with confounding factors such as retrieval policy, task order, or scaffold dynamics, and critically, they do not guarantee that test tasks are ability-supported by prior experience. This leads to a fundamental gap: there is no explicit, diagnosable measure of procedural transfer from experience in current agent evaluation regimes.

Benchmark Design and Dataset Construction

EvoAgentBench addresses these limitations by introducing a benchmark that systematically evaluates agent self-evolution via Ability-guided transfer across four long-horizon domains: web research, algorithmic reasoning, software engineering, and knowledge work. The central innovation is the construction of an Ability Graph. In this graph, nodes represent tasks and edges indicate explicit procedural overlap, derived from reusable Abilities that are grounded in agent execution traces (not superficial metadata).

The benchmark construction pipeline is staged as follows:

1. Multi-Backbone Trace Collection: For each task, agent executions are collected from diverse agent backbones (Kimi-K2.5, GLM-5.1, DeepSeek-V3.2), providing a heterogeneous trace pool. This ensures procedural abstraction is robust to model idiosyncrasies.
2. Trace-Grounded Ability Extraction: Leveraging these traces, the pipeline extracts raw Ability cards as operational tuples encoding trigger conditions, procedures, applicability boundaries, roles (Method, Guard, Workflow), and supporting evidence.
3. Ability Canonicalization: Raw cards are clustered based on operational equivalence (not semantics), adjudicated by multiple LLMs and human experts, then assembled into canonical Ability families after consistency checks.
4. Ability Graph Construction and Partition: Canonical Ability units establish edge-eligible linkage among tasks, enabling partitioning into communities that reflect procedural—rather than arbitrary—relatedness.
5. Constrained Split: Test tasks are sampled such that every test task is ability-supported by the train split yet remains instance-disjoint, guaranteeing that any transfer is procedural rather than the result of memorization or data leakage.

   Figure 1: EvoAgentBench construction pipeline, illustrating multi-backbone trace harvesting, ability extraction/canonicalization, and ability-aware instance splitting.

The benchmark ultimately covers 528 training and 267 test tasks, with 170 canonical Ability units linking the dataset. This structure directly supports fine-grained diagnosis of transfer: when a method fails, one can algorithmically determine whether support was missing due to extraction, routing, or uptake, rather than random split artifact.

Evaluation Framework and Experimental Findings

The evaluation protocol ensures that methods differ only in how they encode, retrieve, and apply procedural experience from the training set. Two agent scaffolds (OpenClaw and Nanobot) and three LLM backbones (Qwen3.5-27B, Qwen3.5-397B, Gemma-4-31B) are considered, covering a range of architectures and compute scales.

Four conditions are compared:

• Vanilla: no evolution or transfer mechanism
• Memento: case-based retrieval of closest training trajectories (Zhou et al., 22 Aug 2025)
• ReasoningBank: distilled memory of natural language strategies (Ouyang et al., 29 Sep 2025)
• GEPA: prompt evolution over the training set with global prompt broadcast (Agrawal et al., 25 Jul 2025)
• Anchor Skill: curator-routed injection of Ability-grounded skills via the ability graph (a non-deployable diagnostic reference, as it assumes access to curator-side ability assignments, but content is derived solely from training evidence)

Strong Numerical and Empirical Results

• The Anchor Skill diagnostic always yields positive transfer across all backbones, scaffolds, and domains—demonstrating that procedural experience, when delivered via ability-aware routing, is universally beneficial: e.g., +7.5, +10.5, and +5.8 point gains on Qwen3.5-27B, Qwen3.5-397B, and Gemma-4-31B, respectively.
• No existing automatic method exhibits positive gain in every evaluation cell (scaffold/backbone/domain): all currently deployed self-evolution paradigms show negative transfer in some domains or settings. Even approaches with strong headline results (e.g., GEPA, ReasoningBank) display inconsistent gains and sometimes harm agent performance.
• Catastrophic negative transfer was measured (e.g., Memento with a −36.3 point drop in SWE, Nanobot, Qwen3.5-27B), implicating breakdowns in extraction and routing rather than inherent limits of procedural reuse.

Analysis and Implications

Diagnosability and Methodological Clarity

The ability-aware split enables unique clarity: when an agent fails to improve, one can localize the bottleneck to either (i) inability to extract relevant procedural content from training experience, (ii) failure in routing/applying the relevant ability, or (iii) mismatches in the agent's uptake process. Since all methods, except Anchor, share the same backbone, split, and tool configuration, method-level differences can be precisely attributed to procedural mechanism rather than confounded variables.

Automatic Methods: Extraction and Routing Are Fragile

• Similarity-based retrieval (Memento) is vulnerable to superficial query similarity mismatches, yielding both positive and highly negative deltas.
• Global prompt evolution (GEPA) offers improvement in some average settings, but lacks task specificity, leading to both useful and detrimental effects depending on domain and backbone.
• Strategy bank approaches (ReasoningBank) yield modest average gains but do not deliver robust improvements across the full evaluation grid.
• Methodological implication: Merely extracting, storing, or evolving skills is insufficient; correct routing and applicability filtering of procedural artifacts are crucial for robust transfer. Furthermore, ability-level support must be controlled for transfer gains to be interpretable.

Cost-Accuracy Tradeoff

Analysis of agent step/turn overhead reveals a complex relationship: evolution does not guarantee reduced turn count, and in many cases, poorly matched procedures increase overhead without solving more tasks. Cost and accuracy must be jointly reported; optimizing for accuracy alone masks heterogeneity in agent behavior induced by procedural evolution.

Theoretical and Practical Implications

EvoAgentBench exposes a fundamental gap between what can be achieved by ideal procedural extraction/routing (as revealed by Anchor Skill) and what is realized by current automatic methods. This suggests several lines for future research:

• Algorithmic: Novel self-evolution mechanisms should emphasize robust, context-aware routing and applicability filtering, possibly leveraging richer interaction with the underlying ability graph.
• Representation: Procedural artifacts should explicitly encode applicability conditions and failure modes, moving beyond raw case storage or surface-level similarity.
• Evaluation: Benchmarks should control for ability-level support to ensure transfer is actionable and measurable; multi-setting and multi-backbone evaluation is mandatory for robust progress.

Limitations and Future Directions

While EvoAgentBench spans four text-based agentic domains, it does not yet cover embodied or multimodal agents, nor does it investigate how ability extraction varies with different LLM extractors. Absolute transfer gains are specific to the supported split and agent-scale considered. Extending the paradigm to broader domains, additional modalities, and variable extraction mechanisms are promising avenues.

Conclusion

EvoAgentBench provides an explicit, fine-grained framework for benchmarking agent self-evolution as procedural ability transfer. Empirical results demonstrate that while ideal skill delivery is beneficial, automatic self-evolution methods are currently brittle and prone to domain-, model-, and routing-induced regression. Future progress in AI agent self-improvement will depend on principled ability extraction, precise routing, and controlled evaluation, as operationalized by EvoAgentBench.