Agentic Entropy-Balanced Policy Optimization

• AEPO is an agentic reinforcement learning algorithm designed to balance exploration and exploitation through dynamic entropy control in both rollout and policy update stages.
• It employs a dynamic entropy monitoring and adaptive resource allocation strategy to prevent sampling collapse, ensuring diverse exploration across complex tasks.
• By integrating stop-gradient operations and entropy-aware advantage estimation, AEPO stabilizes learning and enhances performance on long-horizon, tool-integrated web challenges.

Agentic Entropy-Balanced Policy Optimization (AEPO) is an agentic reinforcement learning algorithm formulated to address the dual requirements of exploration-driven learning and stability during policy optimization for web agents, particularly in multi-turn, tool-integrated interaction scenarios. AEPO introduces innovations in both rollout generation and policy update mechanisms to strategically balance entropy, prevent training collapse, and maximize both sampling diversity and final performance across challenging, long-horizon benchmarks.

1. Motivation and Entropy-Related Challenges

AEPO arises from the empirical observation that mainstream agentic RL algorithms, which utilize entropy as the guiding signal for exploration, face two critical challenges:

• High-Entropy Rollout Collapse: When consecutive high-entropy tool-call steps accumulate in the rollout phase, sampling budget is excessively concentrated along narrow branches. This "collapse" restricts solution space coverage and hinders diverse exploration.
• Gradient Clipping of High-Entropy Tokens: Standard policy optimization procedures, notably those employing generic gradient clipping, suppress gradients corresponding to high-uncertainty (high-entropy) tokens. This clipping effect impedes learning from exploratory steps and may prematurely anchor the agent to suboptimal regions.

AEPO is explicitly designed to mediate these phenomena by dynamically controlling entropy in both the sampling (rollout) and training (policy update) phases.

2. Dynamic Entropy-Balanced Rollout Mechanism

The first component of AEPO is a dynamic rollout strategy that adaptively manages the allocation of global (complete trajectory) versus branch (partial trajectory event-driven) sampling. Core workflow includes:

• Entropy Pre-Monitoring: Prior to rollout expansion, AEPO estimates the uncertainty at both the root query ($H_\text{root}$) and average tool-call outputs ($H_\text{tool}^{\text{avg}}$).
• Adaptive Budget Allocation: The number of global samples $m$ from total budget $k$ is set by

$$m = k \cdot \sigma\left(\beta \cdot (H_\text{root} - H_\text{tool}^{\text{avg}})\right)$$

where $\sigma(\cdot)$ is the sigmoid, and $\beta$ tunes sensitivity. A large gap allocates more budget to global exploration, promoting trajectory diversity.

• Branch Penalty on High-Entropy Events: During branching, the sampling probability is made dependent on both the current token's entropy gap $\Delta H_t$ and the count $l$ of consecutive high-entropy steps:

$$P_t = (\alpha + \gamma \cdot \Delta H_t) \left(1 - \hat{P}(l)\right)$$

Higher $l$ incurs stronger penalization, thus preventing over-branching.

This flexible allocation framework ensures broad coverage, avoids "sampling collapse," and more evenly distributes exploration across the agent's solution space.

3. Entropy-Balanced Policy Optimization: Stop-Gradient and Advantage Rescaling

The policy optimization phase incorporates crucial modifications for stabilizing entropy and preserving informative gradients:

• High-Entropy Token Rescaling: AEPO inserts a stop-gradient operation into the high-entropy clipping term for token-level updates. This prevents backward suppression of exploratory tokens, allowing gradients to flow and be properly rescaled in proportion to their entropy.
• Entropy-Aware Advantage Estimation: The traditional advantage estimate $\widetilde{A}^{(t)}_\text{Acc}$ is augmented by an entropy-derived term $\widetilde{A}^{(t)}_{\Delta H}$:

$$\widetilde{A}^{(t)} = \widetilde{A}^{(t)}_\text{Acc} \cdot (1 + a \cdot \widetilde{A}^{(t)}_{\Delta H})$$

where $a$ controls the weighting. High-uncertainty tokens (high entropy) are prioritized for learning, increasing the agent's capacity to adaptively explore complex action sequences.

In combination, these mechanisms maintain policy entropy at desirable levels across RL epochs, balancing exploratory capacity with exploitation stability.

4. Performance Metrics and Benchmark Results

AEPO demonstrates superior results on 14 demanding datasets covering multi-turn information seeking, knowledge-intensive reasoning, and complex web navigation. Specifically, with 1,000 RL samples and Qwen3-14B backbone:

Dataset	Pass@1	Pass@5
GAIA	47.6%	65.0%
Humanity's Last Exam	11.2%	26.0%
WebWalker	43.0%	70.0%

Such results illustrate that AEPO enables rapid improvements in sample efficiency and final generalization, especially for agentic reasoning tasks requiring nuanced exploration and integration of multiple external tools.

Further analysis confirms that AEPO improves rollout sampling diversity over baseline RL methods and preserves stable policy entropy, which is critical for long-horizon tasks with high branching complexity.

5. Comparative Advantages Over Mainstream RL Algorithms

AEPO outperforms seven mainstream RL algorithms—including vanilla RL, various clipping-optimized methods (GPPO, CISPO), and alternative agentic RL variants (ARPO, GIGPO)—by virtue of:

• Dual-phase entropy balancing: Unlike entropy-driven methods that either over-explore or indiscriminately clip gradients, AEPO orchestrates entropy allocation during both rollout and policy update.
• Adaptive resource allocation: Real-time entropy measurements dynamically set the global versus branch budget, in contrast to rigid pre-allocation in prior algorithms.
• Gradient preservation: The stop-gradient mechanism and entropy-aware advantage estimation rescue learning capacity on high-uncertainty tokens, fostering discovery of novel behaviors.

These differentiators yield more robust exploration, higher pass rates, and increased sampling diversity across agentic RL benchmarks.

6. Scalability and Stability

In scalable web agent training, AEPO provides:

• Sampling Diversity: The dynamic entropy-balanced rollout mitigates collapse and ensures trajectories cover a wide portion of the state space.
• Stable Policy Entropy: Preserving and rescaling gradients for high-entropy tokens maintains policy entropy and avoids instability or premature determinism.

Such characteristics are essential for advancing agentic RL agents capable of long-horizon, robust tool integration across diverse web interface tasks.

7. Prospects for Further Development

AEPO's foundational architecture prompts several avenues for future research:

• Further refinement of entropy allocation and penalty functions for highly dynamic or adversarial environments.
• Expansion to multi-modal agents integrating more sophisticated external toolchains.
• Investigation of advanced gradient preservation mechanisms, potentially involving adaptive or content-dependent clipping thresholds.
• Broader benchmark evaluation and adaptation to real-world web agent tasks for comprehensive assessment of scalability.
• Enhancement of entropy-aware advantage estimation via integration of techniques from other clipping-optimized RL frameworks.

This ongoing development trajectory holds promise for deploying AEPO as a general framework for robust, scalable, and agentic RL in web agent training and beyond.

---

In summary, Agentic Entropy-Balanced Policy Optimization implements coordinated entropy management across both sampling and optimization, enabling agentic RL agents to realize broad exploration, stable learning, and high performance across deeply interactive, long-horizon tasks. Its approach directly addresses challenges of high-entropy rollout collapse and gradient suppression, and is empirically validated as a foundation for future agentic RL research (Dong et al., 16 Oct 2025).