BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping (2510.18927v1)

Abstract: Reinforcement learning (RL) has recently become the core paradigm for aligning and strengthening LLMs. Yet, applying RL in off-policy settings--where stale data from past policies are used for training--improves sample efficiency, but remains challenging: policy entropy declines sharply, optimization often becomes unstable and may even collapse. Through theoretical and empirical analysis, we identify two key insights: (i) an imbalance in optimization, where negative-advantage samples dominate the policy gradient, suppressing useful behaviors and risking gradient explosions; and (ii) the derived Entropy-Clip Rule, which reveals that the fixed clipping mechanism in PPO-like objectives systematically blocks entropy-increasing updates, thereby driving the policy toward over-exploitation at the expense of exploration. Building on these insights, we propose BAlanced Policy Optimization with Adaptive Clipping (BAPO), a simple yet effective method that dynamically adjusts clipping bounds to adaptively re-balance positive and negative contributions, preserve entropy, and stabilize RL optimization. Across diverse off-policy scenarios--including sample replay and partial rollout--BAPO achieves fast, stable, and data-efficient training. On AIME 2024 and AIME 2025 benchmarks, our 7B BAPO model surpasses open-source counterparts such as SkyWork-OR1-7B, while our 32B BAPO model not only achieves state-of-the-art results among models of the same scale but also outperforms leading proprietary systems like o3-mini and Gemini-2.5-Flash-Thinking.

• The paper introduces BAPO, a dynamic adaptive clipping mechanism that ensures balanced policy optimization by regulating positive and negative token contributions.
• It addresses optimization imbalance by selectively amplifying low-probability positive tokens and mitigating gradient explosion from negative samples.
• Empirical evaluations on AIME benchmarks demonstrate that BAPO significantly improves training stability and performance across various off-policy scenarios.

BAPO: Stabilizing Off-Policy RL for LLMs via Balanced Policy Optimization with Adaptive Clipping

Introduction and Motivation

The application of reinforcement learning (RL) to LLMs has become central for aligning and enhancing model capabilities in complex reasoning, coding, and agentic tasks. Off-policy RL, which leverages stale data from previous policies, offers improved sample efficiency and compatibility with modern AI infrastructure, such as partial rollouts. However, off-policy RL introduces significant instability: policy entropy collapses, optimization becomes erratic, and training may fail entirely as data staleness increases.

Empirical analysis reveals two critical issues: (1) optimization imbalance, where negative-advantage samples dominate the policy gradient, suppressing beneficial behaviors and risking gradient explosion; and (2) the Entropy-Clip Rule, which shows that fixed clipping in PPO-like objectives systematically blocks entropy-increasing updates, driving the policy toward over-exploitation and reduced exploration.

Figure 1: As data staleness increases, off-policy RL suffers from unstable optimization, entropy collapse, and sudden training failure.

Theoretical Analysis: Imbalanced Optimization and Entropy-Clip Rule

The policy gradient in PPO-like objectives is split into contributions from positive and negative advantage tokens. Empirical results demonstrate that positive samples are a minority both in count and in their impact on the policy-gradient loss. This imbalance is exacerbated by longer trajectories on difficult queries and insufficient model capability in early training stages.

Negative low-probability tokens can dominate the loss, leading to gradient explosion and instability. The Entropy-Clip Rule, derived for PPO objectives, shows that the clipping mechanism blocks many low-probability positive tokens (which would increase entropy) while over-penalizing low-probability negatives. This exclusion of entropy-increasing updates leads to a sharp decline in policy entropy and a bottleneck in exploration.

Figure 2: Positive tokens contribute less to the policy-gradient loss and are fewer in number compared to negative tokens during training.

Figure 3: Token probabilities, importance sampling weights, and entropy are tightly coupled; low-probability tokens with extreme IS weights are often clipped, reducing entropy.

Methodology: BAPO Algorithm

BAPO (BAlanced Policy Optimization with Adaptive Clipping) introduces a dynamic clipping mechanism that adaptively adjusts the clipping bounds $c_\textnormal{low}$ and $c_\textnormal{high}$ for each update step. The goal is to ensure that the contribution of positive tokens to the policy-gradient loss meets a target threshold $\rho_0$, while preventing negative tokens from overwhelming the optimization.

The algorithm incrementally increases $c_\textnormal{high}$ and $c_\textnormal{low}$ until the positive token contribution condition is satisfied:

$\frac{|\sum_{A_t>0} \pi_{\theta_{\textnormal{rollout}}(y_t) \cdot [\min(r_t\cdot A_t, \textnormal{clip}(r_t, 0, c_\textnormal{high})\cdot A_t)]|}{|\sum_{A_t} \pi_{\theta_{\textnormal{rollout}}(y_t) \cdot [\min(r_t\cdot A_t, \textnormal{clip}(r_t, c_\textnormal{low}, c_\textnormal{high})\cdot A_t)]|} \geq \rho_0$

This adaptive strategy incorporates more low-probability positive tokens (preserving entropy) and filters out excessive negatives (preventing gradient explosion), resulting in stable and efficient RL optimization.

Figure 4: BAPO dynamically adjusts clipping bounds to balance positive and negative contributions, preserving entropy and stabilizing training.

Empirical Results

BAPO demonstrates robust improvements across diverse off-policy scenarios, including sample replay, partial rollout, and varying staleness levels. On AIME 2024 and AIME 2025 benchmarks, BAPO-trained models consistently outperform both open-source and proprietary baselines at comparable scales.

• BP-Math-32B$_{\texttt{BAPO}}$ achieves 87.1 (AIME24) and 80.0 (AIME25), surpassing Qwen3-32B and SkyWork-OR1-32B, and outperforming proprietary systems like o3-mini-medium and Gemini-2.5-Flash-Thinking.
• BP-Math-7B$_{\texttt{BAPO}}$ achieves 70.8 (AIME24) and 62.5 (AIME25), outperforming SkyWork-OR1-7B and matching Gemini-2.0-Flash-Thinking.

  Figure 5: BAPO achieves superior performance compared to baselines across multiple benchmarks and model scales.

BAPO also exhibits greater robustness in partial rollout settings, maintaining stable optimization where baseline methods degrade.

Figure 6: BAPO training dynamics show rapid reward increase, stable entropy, and balanced gradient normalization.

Implementation Considerations

BAPO is implemented as an extension of GRPO, with dynamic adjustment of clipping bounds per batch. Hyperparameters (e.g., $\rho_0$, clipping ranges, step sizes) are not finely tuned, yet the method demonstrates strong empirical performance. The algorithm is compatible with modern RL infrastructure, including experience replay and partial rollouts, and does not require complex manual hyperparameter tuning.

Resource requirements are similar to standard PPO/GRPO, with additional computation for dynamic bound adjustment. BAPO is scalable to long-context LLMs and large batch sizes, and is robust to varying degrees of data staleness.

Theoretical Implications and Future Directions

The Entropy-Clip Rule provides a formal link between clipping mechanisms and policy entropy, highlighting the importance of balancing exploration and exploitation in off-policy RL for LLMs. BAPO's adaptive clipping offers a principled approach to maintaining entropy and preventing optimization collapse.

Future research may explore token-level adaptive clipping, integration with curriculum learning, and further theoretical analysis of entropy dynamics in RL for LLMs. Extensions to multimodal and agentic LLMs, as well as distributed RL systems, are promising directions.

Figure 7: The function $f(x) = x (\log x - C)$ illustrates the relationship between token probability and entropy contribution.

Conclusion

BAPO addresses fundamental challenges in off-policy RL for LLMs by dynamically balancing positive and negative contributions and preserving policy entropy. The method achieves stable, efficient optimization and state-of-the-art performance across multiple benchmarks and model scales. Theoretical analysis and empirical validation underscore the importance of adaptive clipping for robust RL in LLMs, providing a foundation for future advances in scalable, efficient LLM alignment.