Understanding Reasoning from Pretraining to Post-Training
This presentation explores a groundbreaking study that systematically maps how pretraining choices shape reinforcement learning outcomes in reasoning tasks. Using chess as a controlled testbed, the research establishes quantitative scaling laws connecting pretraining metrics to post-training performance, reveals the mechanisms by which RL reshapes model policies, and provides actionable insights for compute allocation. The findings extend to mathematical reasoning, suggesting broad applicability across domains requiring complex multi-step inference.Script
Most research treats pretraining and reinforcement learning as separate problems, but the real question is how they work together. This paper uses chess as a laboratory to quantify exactly how your pretraining choices determine what reinforcement learning can achieve.
The authors built a controlled pipeline mirroring language model training. They pretrain transformers on millions of human chess games, fine-tune on synthetic reasoning traces that show step-by-step puzzle solving, then run reinforcement learning on tactical puzzles where only complete solutions earn rewards.
The central discovery is a predictive scaling law. Models with lower pretraining loss achieve higher reinforcement learning performance at any fixed compute budget. More striking, the learning curve slope during reinforcement learning increases monotonically with pretraining token count, meaning better pretraining makes every reinforcement learning update more effective.
Mechanistically, reinforcement learning does not simply sharpen the policy uniformly. On easy puzzles, it amplifies moves that were already likely. On hard puzzles, it can surface correct moves from the distribution tail, but it also frequently promotes wrong moves into dominance, especially when rewards are sparse and errors compound across multiple steps.
Reinforcement learning improves search breadth and first-move accuracy, but struggles with depth. Even after extensive training, models rarely recover full multi-step solution lines. Coverage of ground truth moves decays sharply with each additional ply, revealing a fundamental limit in how current methods extend reasoning chains.
The scaling law transfers to mathematical reasoning with a 1 billion parameter language model, confirming that these patterns reflect general properties of reasoning systems. At low compute, invest heavily in pretraining; at high compute, shift more resources to reinforcement learning. For anyone building reasoning models, visit EmergentMind.com to explore this work and create your own video summaries of the latest research.
