AgentEvolver: Towards Efficient Self-Evolving Agent System (2511.10395v1)

Abstract: Autonomous agents powered by LLMs have the potential to significantly enhance human productivity by reasoning, using tools, and executing complex tasks in diverse environments. However, current approaches to developing such agents remain costly and inefficient, as they typically require manually constructed task datasets and reinforcement learning (RL) pipelines with extensive random exploration. These limitations lead to prohibitively high data-construction costs, low exploration efficiency, and poor sample utilization. To address these challenges, we present AgentEvolver, a self-evolving agent system that leverages the semantic understanding and reasoning capabilities of LLMs to drive autonomous agent learning. AgentEvolver introduces three synergistic mechanisms: (i) self-questioning, which enables curiosity-driven task generation in novel environments, reducing dependence on handcrafted datasets; (ii) self-navigating, which improves exploration efficiency through experience reuse and hybrid policy guidance; and (iii) self-attributing, which enhances sample efficiency by assigning differentiated rewards to trajectory states and actions based on their contribution. By integrating these mechanisms into a unified framework, AgentEvolver enables scalable, cost-effective, and continual improvement of agent capabilities. Preliminary experiments indicate that AgentEvolver achieves more efficient exploration, better sample utilization, and faster adaptation compared to traditional RL-based baselines.

• The paper introduces a self-evolving framework that synergizes self-questioning, self-navigating, and self-attributing to enhance agent performance and sample efficiency.
• The paper presents a novel method for autonomous task generation and fine-grained credit assignment, reducing dependence on manual task curation and traditional reward signals.
• The framework achieves significant improvements over standard RL baselines, including over a 50% reduction in steps to convergence and enhanced scalability across diverse environments.

AgentEvolver: A Framework for Efficient Self-Evolving Agent Systems

Introduction and Motivation

AgentEvolver introduces a principled architecture for autonomous agent self-improvement, specifically targeting the high-cost, low-efficiency regime inherent in current LLM-driven agent development. Prior paradigms either rely on workflow-based orchestration—requiring manual task curation and rigid action design—or on RL pipelines which suffer from expensive, inefficient exploration and sample utilization. AgentEvolver addresses these bottlenecks by operationalizing three synergistic mechanisms: self-questioning (for autonomous task generation), self-navigating (for experience-guided, sample-efficient exploration), and self-attributing (for fine-grained credit assignment). The framework integrates these components into an end-to-end pipeline designed for scalable agent evolution without human-engineered rewards or curricula.

Figure 1: Overview of the AgentEvolver framework. The self-evolving process is driven by self-questioning, self-navigating, and self-attributing mechanisms for autonomous data generation, efficient learning, and dense reward assignment.

Problem Formulation and Theoretical Foundation

AgentEvolver formalizes agent learning within an interaction sandbox $$\mathcal{E} = (\mathcal{S}, \mathcal{A}, \mathcal{P})$$—abstracting away any predefined reward or task objectives—where agents must estimate both task distributions and reward functions autonomously. The core optimization problem is thus bifurcated into (1) proxy task generation ($F_\mathrm{task}$) via environment exploration, and (2) proxy reward design ($F_\mathrm{reward}$) based on fine-grained trajectory attribution. Performance is measured with respect to a hidden target task distribution $p_\mathrm{target}(g)$, placing a premium on the agent's ability to generalize from proxy tasks/rewards to true downstream objectives.

Self-Questioning: Autonomous Task Discovery and Data Synthesis

AgentEvolver's self-questioning mechanism synthesizes high-quality training tasks through a curiosity-driven process, significantly reducing dependence on expensive, handcrafted datasets. This is accomplished via:

• Exploration phase guided by environment profiles acting as action priors, enabling LLM-driven stochastic exploration.
• Adaptive task synthesis, leveraging user preferences for customizing trajectory-to-task mapping and complexity scaling.
• A multi-stage task curation pipeline for deduplication, feasibility checking, and hybridization with original distribution samples.

  Figure 2: Self-questioning module pipeline: exploration informed by environment profiles, followed by task synthesis and curation.

This mechanism demonstrates that synthetic data, when properly curated and paired with robust LLM-based judgements, yields agent learning curves near or surpassing those achieved by original target datasets (cf. experimental Table 2). The process enables data diversity and scalability, supporting both challenge-oriented benchmarking and cross-domain generalization.

Self-Navigating: Experience-Guided, Efficient Exploration

The self-navigating mechanism incorporates structured natural-language experiences, distilled from previous trajectories, into both the rollout and optimization process. This consists of:

• Construction of a cold-start experience pool from synthetic tasks.
• Retrieval of contextually-relevant experiences via semantic similarity, injected into trajectories through explicit in-context learning (ICL) prompts.
• Experience-mixed rollouts, calibrated by proportion $\eta$, balancing exploration and exploitation for optimal sample efficiency.
• Experience stripping during optimization, preventing overfitting to explicit prompts, and selective boosting to address off-policy learning instability.

  Figure 3: Self-navigating mechanism: experience acquisition, experience-mixed rollout, and experience incorporation to policy optimization.

Empirical analysis shows that implicit experience learning—via RL training using distilled experiences—substantially outperforms explicit ICL-only methods and vanilla RL. This is enabled by a judicious balance between prior-guided exploitation and exploration, with ablations highlighting the importance of hyperparameter calibration for both the experience ratio and GRPO clipping threshold.

Self-Attributing: Fine-Grained Stepwise Credit Assignment

Traditional outcome-only reward attribution is sub-optimal for long-horizon, multi-step agents due to uniform credit assignment and sparse supervision. AgentEvolver introduces self-attributing by:

• Retrospective step-wise attribution using LLM-based evaluators, yielding GOOD/BAD binary labels per action.
• Quantification and normalization of attribution signals, with trajectory-wise standardization to prevent length bias.
• Dual-channel composite reward construction, fusing normalized step-wise and outcome-based signals.
• Token-level advantage propagation for GRPO-style policy optimization.

Experimental studies demonstrate that dense, attribution-guided credit assignment improves both sample efficiency (over 50% reduction in steps to convergence) and final performance metrics compared to outcome-only baselines. Ablations reveal that both channels are indispensable: process-level signals accelerate and stabilize learning, while outcome-based supervision is required for correct grounding in task objectives. Hyperparameter exploration identifies the optimal range for the attribution weight $\alpha$, balancing fast convergence and long-term robustness.

Figure 4: Learning curves for different alpha values on AppWorld, showing trade-offs between early convergence and final performance.

System Infrastructure and Context Management

AgentEvolver is implemented as a decoupled, scalable infrastructure comprising: a four-stage training orchestration loop (task synthesis, rollout, experience summarization, sample/model optimization); hierarchical rollout execution (service layer, workers, manager); and a modular context management subsystem. The context manager supports four fundamental templates for message organization (causal, reasoning-augmented, sliding-window, and self-managing), enabling flexible adaptation to diverse long-horizon agent benchmarks and maintaining temporal and computational efficiency.

Figure 5: Context-managing templates, illustrating various modes of agent-history organization and memory control strategies.

The environment service offers gym-compatible, Ray-based concurrency for efficient sampling across benchmarks such as AppWorld and BFCL, supporting standardized deployment in real-world and simulated environments.

Experimental Analysis

Comprehensive benchmarking (on AppWorld and BFCL-v3) demonstrates AgentEvolver’s competitive edge over baseline and ablation candidates. Incorporation of all three mechanisms yields marked improvements:

• Qwen2.5-7B: avg@8 +29.4%, best@8 +36.1%
• Qwen2.5-14B: avg@8 +27.8%, best@8 +30.3%

Notably, AgentEvolver achieves these gains utilizing substantially fewer parameters and data compared to larger baseline models, showing superior scaling and sample efficiency.

Figure 6: Performance comparison on benchmarks; AgentEvolver outperforms larger models with fewer parameters.

Each mechanism individually provides strong benefits (+Questioning for task diversity, +Navigating for efficient search, +Attributing for dense optimization), but their combination yields synergistic effects, establishing state-of-the-art agentic performance across domains.

Implications and Future Directions

AgentEvolver provides a systematic blueprint for autonomous agent evolution, facilitating scalable, cost-effective, and theoretically grounded agentic learning in open-ended tasks. Key practical implications include:

• Full pipeline automation for agent training, reducing manual effort and enabling dynamic deployment in novel environments.
• Fine-grained, LLM-driven feedback signals that improve exploration efficiency and sample utilization, vital for real-world, expensive environments.
• Transferable, modular system infrastructure supporting extensibility and integration with different RL, agentic, and context management architectures.

The theoretical implications suggest new directions in agent-environment framing, distributional curriculum synthesis, and reward modeling. More robust task coverage and generalization are achievable through self-synthesized hybrid data, while attribution-based learning opens the door for more interpretable and debuggable agent policies.

Conclusion

AgentEvolver sets a benchmark in the construction of self-improving agentic systems, delivering a tightly linked framework for autonomous task generation, efficient policy optimization, and fine-grained credit assignment. Comprehensive experimental results substantiate the framework's effectiveness, both in terms of scaling, resource utilization, and final performance metrics. Extensions toward challenge-oriented domains, larger LLMs, and unified LLM-level self-evolving loops represent natural next steps.

By emphasizing modularity, sample efficiency, and environment-action abstraction, AgentEvolver positions itself as a versatile foundation for future research in scalable agentic reasoning and continual improvement. The framework provides a roadmap for rigorous, data-driven, and practical agent development in complex, real-world settings.