The Art of Scaling Reinforcement Learning Compute for LLMs (2510.13786v1)

Abstract: Reinforcement learning (RL) has become central to training LLMs, yet the field lacks predictive scaling methodologies comparable to those established for pre-training. Despite rapidly rising compute budgets, there is no principled understanding of how to evaluate algorithmic improvements for scaling RL compute. We present the first large-scale systematic study, amounting to more than 400,000 GPU-hours, that defines a principled framework for analyzing and predicting RL scaling in LLMs. We fit sigmoidal compute-performance curves for RL training and ablate a wide range of common design choices to analyze their effects on asymptotic performance and compute efficiency. We observe: (1) Not all recipes yield similar asymptotic performance, (2) Details such as loss aggregation, normalization, curriculum, and off-policy algorithm primarily modulate compute efficiency without materially shifting the asymptote, and (3) Stable, scalable recipes follow predictable scaling trajectories, enabling extrapolation from smaller-scale runs. Combining these insights, we propose a best-practice recipe, ScaleRL, and demonstrate its effectiveness by successfully scaling and predicting validation performance on a single RL run scaled up to 100,000 GPU-hours. Our work provides both a scientific framework for analyzing scaling in RL and a practical recipe that brings RL training closer to the predictability long achieved in pre-training.

• The paper introduces a sigmoidal compute-performance law that reliably extrapolates RL rewards in LLM training using 400K GPU-hours of data.
• The paper validates key design choices through extensive ablations, demonstrating that methods like ScaleRL improve both efficiency and asymptotic reward.
• The paper shows that implementation tweaks such as FP32 precision and adaptive prompt sampling significantly boost RL performance and scalability.

Principled Scaling of RL Compute for LLMs: Methodology, Empirical Insights, and Practical Implications

Introduction and Motivation

This paper establishes a rigorous framework for analyzing and predicting the scaling behavior of reinforcement learning (RL) applied to LLMs. While pre-training scaling laws are well-understood and widely adopted, RL post-training remains empirically driven, with little theoretical guidance for extrapolating performance as compute budgets increase. The authors address this gap by conducting a systematic paper over 400,000 GPU-hours, introducing a sigmoidal compute-performance law for RL, and empirically validating its predictive power across multiple axes of RL training.

Sigmoidal Compute-Performance Law

The central methodological contribution is the adoption of a sigmoidal law to model the relationship between RL training compute ($C$) and expected reward ($R_C$) on held-out validation data:

$$R_C = R_0 + \frac{A - R_0}{1 + (C_{\text{mid}}/C)^B}$$

where $A$ is the asymptotic performance ceiling, $B$ is the scaling efficiency exponent, and $C_{\text{mid}}$ is the compute midpoint. This formulation captures the saturating returns observed in RL for LLMs, contrasting with the unbounded power-law fits typical in pre-training. The sigmoidal law enables reliable extrapolation from small-scale runs to large compute budgets, facilitating principled evaluation of RL recipes without exhaustive experimentation.

Figure 1: $C_{\text{mid}}$ determines the compute point at which half of the total gain is achieved; $B$ controls the curve’s steepness; $A$ is the asymptotic performance.

Empirical Study: Ablations and Recipe Construction

The authors perform extensive ablations on design choices, including off-policy algorithms, loss functions, aggregation strategies, advantage normalization, precision fixes, batch definition, and data curriculum. Key findings include:

• Off-policy algorithm: PipelineRL-$k$ yields higher compute efficiency ($B$) than PPO-off-policy-$k$, with similar asymptotic performance ($A$), due to reduced idle time and tighter feedback between generators and trainers.

Figure 2: PipelineRL-$k$ achieves higher efficiency and slightly better asymptotic pass rate compared to PPO-off-policy-$k$.

• Loss function: CISPO and GSPO outperform DAPO in asymptotic reward, with CISPO exhibiting superior robustness to hyperparameter choices.

Figure 3: CISPO and GSPO loss functions achieve higher asymptotic reward than DAPO.

• FP32 precision at LM head: Applying FP32 precision at the final layer mitigates numerical mismatches between generator and trainer, yielding a substantial boost in $A$.

Figure 3: FP32 precision at the LM head improves asymptotic reward.

• Zero-variance filtering and adaptive prompt sampling: Filtering out prompts with zero reward variance and removing prompts with pass rate $\geq 0.9$ in subsequent epochs both increase the asymptotic ceiling.

Figure 4: Zero-variance filtering and adaptive prompt sampling improve asymptotic performance.

• Aggregation and normalization: Prompt-level loss aggregation and batch-level advantage normalization are marginally superior, but most normalization strategies yield similar $A$.

Leave-One-Out Validation and Robustness

The final RL recipe, termed ScaleRL, integrates the best-performing choices. Leave-one-out (LOO) ablations confirm that each component contributes to either efficiency or stability, with the combined recipe consistently achieving the highest $A$ and $B$. The sigmoidal fits are robust, with error margins for $A$ within $\pm 0.02$ across independent runs.

Figure 5: LOO ablations show ScaleRL slightly outperforms alternatives in efficiency and asymptotic reward.

Figure 6: (a) Variance in scaling fits is low; (b) FP32 precision fix improves performance on Scout MoE.

Scaling Across Multiple Axes

The framework is validated across several scaling axes:

• Model size: Larger models (e.g., 17B$\times$16 MoE) exhibit higher asymptotic RL performance, with scaling curves remaining predictive.
• Generation length: Increasing context length raises the asymptotic ceiling, though initial efficiency is reduced.
• Batch size: Larger batches reach higher asymptotes, despite slower initial progress.
• Generations per prompt: Varying generations per prompt (with fixed total batch) has minimal impact on scaling curves at moderate batch sizes.
• Multi-task RL: Joint training on math and code domains yields parallel, predictable scaling trends.

Figure 7: RL training scaled to 100,000 GPU-hours follows the predicted sigmoidal curve; downstream performance on AIME-24 also scales predictably.

Figure 8: Long-context RL eventually surpasses smaller-context runs in both validation and downstream evaluations.

Figure 9: Larger batch sizes settle at higher asymptotes, despite slower initial training.

Figure 10: ScaleRL’s sigmoidal scaling law generalizes to multi-task RL (math+code).

Comparative Analysis with Prevalent Methods

ScaleRL is benchmarked against established RL recipes (GRPO, DAPO, Magistral, MiniMax-M1), consistently achieving higher asymptotic reward and more efficient scaling. Notably, methods that appear superior at small compute budgets may be overtaken at scale, underscoring the importance of extrapolative analysis.

Figure 11: ScaleRL surpasses prevalent RL methods in asymptotic reward and scaling efficiency.

Practical Implications and Future Directions

The sigmoidal scaling law provides a principled tool for RL practitioners to forecast performance, optimize compute allocation, and select scalable RL algorithms. The empirical findings highlight the necessity of prioritizing design choices that raise the asymptotic ceiling ($A$), with efficiency ($B$) as a secondary criterion. The methodology is robust to model scale, batch size, context length, and task mixture, supporting its generalizability.

The framework opens avenues for deriving predictive scaling laws across pre-training compute, model size, and RL training data. Future work may extend the analysis to structured rewards, generative verifiers, multi-turn RL, and agentic interaction regimes.

Conclusion

This work establishes a scientific foundation for scaling RL compute in LLMs, introducing a robust sigmoidal law for compute-performance extrapolation and empirically validating its predictive power across diverse RL training axes. The ScaleRL recipe, constructed via systematic ablations, achieves state-of-the-art scaling efficiency and asymptotic performance, outperforming prevalent methods. The framework enables cost-effective evaluation and principled design of scalable RL algorithms, with broad implications for the development and deployment of advanced LLMs.