Effective scaling of RLVR

Determine effective methods and principles for scaling Reinforcement Learning with Verifiable Rewards (RLVR) to improve the reasoning capabilities of large language models, identifying which scaling axes and training designs yield reliable performance gains.

Background

Reinforcement Learning with Verifiable Rewards (RLVR) has recently driven progress in reasoning for LLMs, but the community lacks clear guidance on how to scale RLVR effectively. Prior work such as ProRL scaled training by increasing steps but encountered plateaus, indicating that step depth alone may be insufficient.

This paper proposes BroRL, which scales the number of rollouts per prompt, and provides theoretical and empirical support for its effectiveness. Nonetheless, the general question of how best to scale RLVR is explicitly identified as open.

References

Yet, how to effectively scale the RLVR paradigm remains an open question.

— BroRL: Scaling Reinforcement Learning via Broadened Exploration  (2510.01180 - Hu et al., 1 Oct 2025) in Section 1 (Introduction)

Several important questions remain open, we discuss some in the Limitations below (Section~\ref{sec:limitations}). The most critical is whether debate's benefits transfer to domains without verifiable ground truth. Connecting debate training to alignment-relevant behaviours beyond judge metrics (e.g., reducing scheming or deception) would further strengthen the case. And understanding why debate accuracy plateaus rather than continuing toward the RLVR roofline, and how to overcome this, could unlock further gains.

— Debate Training Reduces Reward Hacking in RLAIF  (2608.17776 - Kenton et al., 18 Aug 2026) in Section 5, subsection “Conclusion”

Since our method operates strictly at the input-trajectory level, an exciting avenue for future work is integrating weak-model prefix guidance with advanced RLVR algorithms. We hypothesize that combining our data-level exploration with objective-level regularization could yield synergistic effects, further pushing the boundaries of LLM reasoning.

— Boosting LLM Exploration via Weak-Model Guidance in RLVR  (2608.27420 - Shen et al., 27 Aug 2026) in Section Conclusions