BroRL: Scaling Reinforcement Learning via Broadened Exploration (2510.01180v1)

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a key ingredient for unlocking complex reasoning capabilities in LLMs. Recent work ProRL has shown promise in scaling RL by increasing the number of training steps. However, performance plateaus after thousands of steps, with clear diminishing returns from allocating more computation to additional training. In this work, we investigate a complementary paradigm for scaling RL, BroR-Lincreasing the number of rollouts per example to hundreds to exhaustively Broaden exploration, which yields continuous performance gains beyond the saturation point observed in ProRL when scaling the number of training steps. Our approach is motivated by a mass balance equation analysis allowing us to characterize the rate of change in probability mass for correct and incorrect tokens during the reinforcement process. We show that under a one-step RL assumption, sampled rollout tokens always contribute to correct-mass expansion, while unsampled tokens outside rollouts may lead to gains or losses depending on their distribution and the net reward balance. Importantly, as the number of rollouts per example N increases, the effect of unsampled terms diminishes, ensuring overall correct-mass expansion. To validate our theoretical analysis, we conduct simulations under more relaxed conditions and find that a sufficiently large rollout size N-corresponding to ample exploration-guarantees an increase in the probability mass of all correct tokens. Empirically, BroRL revives models saturated after 3K ProRL training steps and demonstrates robust, continuous improvement, achieving state-of-the-art results for the 1.5B model across diverse benchmarks.

• The paper introduces a novel scaling approach for RL via increased rollout size, which eliminates knowledge shrinkage and revives saturated large language models.
• It employs a mass-balance analysis to decompose RL updates, ensuring stable policy improvement validated by empirical benchmarks in math, code, and reasoning.
• BroRL enhances hardware efficiency by shifting computation dynamics, achieving faster throughput and state-of-the-art performance with fewer RL steps.

BroRL: Scaling Reinforcement Learning via Broadened Exploration

Introduction

BroRL introduces a principled approach to scaling Reinforcement Learning with Verifiable Rewards (RLVR) for LLMs by increasing the number of rollouts per example, rather than merely extending the number of training steps. This work is motivated by a theoretical mass-balance analysis of RLVR update dynamics, which reveals that the diversity and breadth of exploration—operationalized as the rollout size $N$—is a critical axis for stable and efficient policy improvement. Empirical results demonstrate that BroRL overcomes the performance plateaus observed in step-scaling methods such as ProRL, achieving continuous gains in reasoning benchmarks and superior hardware efficiency.

Theoretical Foundations

The core theoretical contribution is a decomposition of the RLVR update into three terms: two always non-negative terms corresponding to sampled correct and incorrect tokens, and a third "unsampled coupling" term that can be negative and is responsible for instability and knowledge shrinkage. The key insight is that as the rollout size $N$ increases, the magnitude of the unsampled coupling term diminishes, guaranteeing that the total probability mass assigned to correct tokens ($Q_{\mathrm{pos}}$) increases monotonically.

Figure 1: This illustration shows how a single RLVR update step alters the total probability mass $\Delta Q_{\mathrm{pos}}$ for correct tokens, decomposing the update into sampled and unsampled contributions.

The mass-balance analysis formalizes the relationship between rollout size and the reliability of policy improvement. Specifically, the expected unsampled second-moment terms decay exponentially with $N$, as shown by the lemma:

$\mathbb{E}[U_2(p)] = p^2 (1-p)^N$

This result ensures that, in the large-$N$ regime, RLVR updates are dominated by the sampled terms, which always expand the correct probability mass.

Simulation and Empirical Validation

Token-level simulations using a TRPO-style surrogate objective confirm the theoretical predictions. As $N$ increases, the updates become more stable, the accumulation of correct probability mass accelerates, and knowledge shrinkage is eliminated.

Figure 2: Training dynamics of the simulator under varying rollout size $N$. Larger $N$ yields more stable updates, faster accumulation of correct mass, and eliminates knowledge shrinkage.

Empirical studies on LLMs further validate these findings. BroRL is applied to models that have saturated after 3,000 ProRL steps. By increasing $N$ from 16 to 512, BroRL revives learning and achieves state-of-the-art results for 1.5B parameter models across math, code, and reasoning benchmarks. Notably, BroRL achieves these gains with fewer RL steps and less wall-clock time, highlighting its efficiency.

Comparative Analysis: BroRL vs. ProRL

A direct comparison between BroRL and ProRL under matched compute budgets reveals three characteristic training trajectories: (1) both improve, but BroRL consistently outperforms; (2) ProRL degrades while BroRL continues to improve; (3) both fail to yield consistent gains, typically on the hardest problems.

Figure 3: Pass@1 comparison of BroRL vs. ProRL, normalized by training compute. BroRL consistently outperforms or maintains gains where ProRL stagnates or degrades.

Statistical analysis across over 10,000 benchmark instances confirms a small but highly significant advantage for BroRL ($\Delta = 0.0033$, $t = 4.84$, $p = 6.5 \times 10^{-7}$), even when starting from a strong baseline. This demonstrates that scaling rollout size is a robust and generalizable strategy for improving LLM reasoning.

System and Hardware Efficiency

BroRL's large-$N$ regime confers substantial system-level advantages. At the algorithmic level, the dynamic sampling pass rate increases (from 41% to 62%), reducing wasted computation. At the hardware level, generation throughput nearly doubles (from 36.5 to 72.4 samples/s), as the workload shifts from memory-bound to compute-bound, maximizing GPU utilization. This is achieved without increasing the number of PPO mini-batches per step, so the improvements stem from higher-quality updates rather than more frequent ones.

Practical Implications and Limitations

BroRL establishes rollout size as a critical scaling axis for RLVR, complementing the traditional focus on training depth. This has immediate implications for practitioners: allocating compute to broader exploration (larger $N$) is more effective and efficient than simply increasing the number of RL steps, especially for models that have reached a plateau. The approach is particularly well-suited for large-batch, high-throughput hardware environments.

A limitation of the current paper is the lack of a comprehensive sweep over intermediate $N$ values, which would more precisely characterize the cost-benefit trade-off and the point of diminishing returns. Future work should address this by mapping the full scaling curve of $N$ in large-scale LLMs.

Theoretical and Future Directions

The mass-balance framework provides a formal underpinning for the empirical success of large-batch RLVR. It suggests that the perceived limits of RL-based reasoning in LLMs are often artifacts of insufficient exploration rather than fundamental algorithmic barriers. This opens avenues for further research into adaptive rollout strategies, more efficient sampling, and the interplay between exploration and generalization in RLHF and RLVR settings.

Conclusion

BroRL demonstrates that scaling the width of RL updates via increased rollout size is a principled and effective strategy for advancing LLM reasoning. Theoretical analysis, simulation, and large-scale experiments converge to show that broad exploration eliminates knowledge shrinkage, revives saturated models, and achieves superior performance and efficiency. These findings establish rollout size as a primary axis for RLVR scaling and provide actionable guidance for the design of future RL-based LLM training regimes.