Effect of Pretraining Scale on RL Compute Returns

Determine how pretraining choices—specifically model size and pretraining data (amount and composition)—influence the downstream performance gains achieved per unit of reinforcement learning compute during post-training on reasoning tasks with large language models. Establish a quantitative relationship that can guide compute allocation between pretraining and reinforcement learning.

Background

In contemporary LLM training, reinforcement learning is typically applied after large-scale pretraining, yet these stages are often studied in isolation. This separation leaves unclear how to divide limited compute between improving the pretrained prior and optimizing it with RL.

The paper motivates this problem by noting that experience-only RL is impractical for LLMs due to enormous action spaces and sparse rewards, making the strength of the pretrained prior crucial. A principled understanding of how model size and data scale during pretraining affect the marginal returns of RL compute would inform optimal training recipes.