Understanding Reasoning from Pretraining to Post-Training

Abstract: Reinforcement learning (RL) has become central to improving LLMs on complex reasoning tasks, yet RL post-training is largely studied in isolation from the pretraining that precedes it. As a result, two basic questions remain open: (1) how do pretraining choices (model size, data) shape the returns to RL compute, and (2) what does RL actually do to the model? These questions are difficult to study in the standard LLM setting: pretraining corpora are vast and uncontrolled, making it hard to attribute behaviors to pretraining versus RL, and systematic compute sweeps across both stages are prohibitively expensive. To address these challenges, we use chess as a controlled testbed for studying reasoning across the full pretraining-to-post-training pipeline. We follow the standard LLM training pipeline by pretraining LLMs from 5M to 1B parameters on human chess games, supervised fine-tuning on synthetic reasoning traces, and running RL on chess puzzles with verifiable rewards. Using this framework, we find that the post-RL performance at given RL compute level is well-predicted from the pretraining loss, and slope of the RL reward curves improves approximately linearly with the pretraining tokens. Beyond scaling, we find that RL does not simply sharpen the SFT policy: on easy puzzles it amplifies correct moves the SFT policy already preferred, while on hard puzzles it surfaces correct moves that were nearly absent under SFT. We further test whether our findings transfer beyond chess by training a 1B LLM on math-domain text, where the same predictive pattern emerges: longer-pretrained checkpoints reach higher post-RL performance and improve faster under RL. In sum, we provide a quantitative account of the pretraining-to-RL interface and a controlled testbed for studying the science of reasoning across the full pretraining-to-post-training pipeline.

• The paper establishes a joint scaling law that quantitatively links pretraining loss to post-RL reward performance on complex reasoning tasks.
• It demonstrates that optimal compute allocation shifts with scale, with low-compute regimes favoring extensive pretraining and high-compute regimes emphasizing RL.
• Mechanistic analysis reveals that while RL amplifies correct moves, it can also promote wrong-mode amplification, highlighting nuanced policy evolution.

Quantitative Characterization of the Pretraining-to-RL Interface in Reasoning: Insights from Chess and Math Domains

Introduction

This paper presents a comprehensive study of the interplay between pretraining and reinforcement learning (RL) in the context of complex reasoning tasks, using chess as a prototypical testbed. While RL post-training has become a standard means to enhance LLMs on challenging tasks, existing research generally considers pretraining and RL in isolation. The paper addresses two central questions: how pretraining choices (model size, data) shape the returns to RL compute, and the mechanisms by which RL restructures model policies. The authors deploy a controlled, synthetic pipeline analogous to LLM workflows—combining pretraining on human games, supervised fine-tuning (SFT) on synthetic reasoning traces, and RL on verifiable chess puzzles—to disentangle these effects and enable systematic scaling analysis.

Figure 1: A synthetic framework enables joint scaling analysis from pretraining through RL in chess, showing prediction of RL performance by pretraining and mechanistic analysis of policy evolution.

Controlled Chess Testbed and Training Pipeline

The experimental pipeline mirrors LLM training with distinct roles for each stage:

• Pretraining: Models (5M–1B parameters) are trained autoregressively on sequences of human chess moves, derived from a large-scale, quality-controlled Lichess dataset. The tokenization encodes each move in four tokens following chess notation conventions.
• Supervised Fine-Tuning: The models are fine-tuned on synthetic chains-of-thought—tree-structured reasoning traces constructed via sampling continuations from a pretraining policy and depth-first serialization. Training data is carefully decontaminated.
• Reinforcement Learning: RL is performed on a chess puzzle environment with verifiable, sparse rewards, employing Group Relative Policy Optimization (GRPO). The models are rewarded only upon execution of an entire solution line, making the environment highly sensitive to single-step errors.

Both SFT and RL are evaluated on a stratified test benchmark spanning a range of tactical puzzle difficulties.

Pretraining and RL Scaling Laws: Empirical Findings

The core contribution is the establishment of a joint scaling law connecting pretraining metrics to post-RL performance. The study employs extensive sweeps (model size, token count, RL compute) to support rigorous analysis.

Pre-RL Scaling

IsoFLOP curves demonstrate that, for any fixed model size, downstream performance on tactical puzzles continues to improve with increased pretraining tokens up to the point of compute-optimality for that size. The optimal allocation shifts as compute scales, favoring more tokens relative to parameters at lower scales, and approaching language modeling scaling at larger budgets.

Figure 2: Empirical puzzle benchmark frontiers across pretraining-RL sweeps for various model sizes, mapping out compute allocation effects.

RL Compute Allocation and Scaling Law

Allocating fixed total compute, the analysis shows that:

• In the low-compute regime, allocating a larger fraction to pretraining yields superior downstream performance—RL is highly initialization-limited.
• With higher total compute, more compute should be devoted to RL—diminishing pretraining returns make RL relatively more valuable.

Post-RL performance at a given RL compute is strongly predicted by pretraining evaluation loss. The slope of the RL learning curve increases monotonically with $\log_{10}T$ (pretraining token count), indicating that models exposed to more tokens exhibit greater sample efficiency during RL.

Figure 3: Pretraining properties (such as validation loss) predict RL scaling: lower loss yields better post-RL reward; the RL slope correlates with pretraining tokens.

A log-linear predictive law is developed:

$$R(C_{\mathrm{RL}}, N, T) = f(L_{\mathrm{pt}}(N, T)) + g(N,T)(\log_{10}C_{\mathrm{RL}} - \log_{10}C_{\mathrm{ref}})$$

where $R$ is RL reward, $N$ model size, $T$ pretraining tokens, $L_{\mathrm{pt}}$ pretraining loss, and $C_{\mathrm{RL}}$ RL compute. The parameterization robustly predicts unseen runs and enables simulation of compute-optimal frontiers.

Figure 4: Simulated compute-optimal frontiers: optimal RL-compute fraction increases with budget, matching empirical results; optimal pretraining allocation tracks Chinchilla scaling.

SFT and RL Effects on Broader Metrics

Pass@$k$ metrics highlight that RL primarily improves pass@1 (the ability to solve with one sample), while gains in pass@16 are limited and often dominated by further pretraining or SFT on reasoning traces. This supports the claim that RL tends to sharpen the existing policy rather than broaden its support, especially on easy tasks.

Figure 5: SFT with reasoning traces enhances pass@$k$ diversity; without traces, gains are concentrated on pass@1 only.

Mechanistic Analysis of Post-RL Policy Evolution

The paper provides detailed mechanistic investigations into how RL interacts with the SFT-initialized policy, considering both aggregate and per-decision effects.

• On easy puzzles, RL amplifies already high-probability correct moves (ground-truth amplification).
• On hard puzzles, RL can surface correct low-probability moves (tail discovery) but also reinforces wrong modes, sometimes promoting suboptimal moves into dominance (wrong-mode amplification).

  Figure 6: RL redistributes probability in distinct ways: ground-truth amplification dominates in easy puzzles; tail discovery and wrong-mode amplification become prominent on harder puzzles.

Case studies of policy evolution illustrate these effects, showing that even when RL discovers correct moves from the distribution tail, it does not always eliminate competing errors.

Figure 7: A hard-puzzle example demonstrates tail discovery and co-occurring wrong-mode amplification through RL stages.

Analysis of the chain-of-thought (CoT) structures generated by the models during RL reveals:

• Increased search breadth but limited gains in depth, with models struggling to recover multi-step (deep) solution continuations even after extensive RL.
• Improvement in the quality of both the committed move and the reasoning-step candidates, with coverage of ground-truth moves increasing but decaying sharply with depth.

  Figure 8: RL improves search breadth and move quality in the model's reasoning traces, but depth-limited search constrains recovery of longer continuations.

Transferability: Scaling Law Patterns in Math Reasoning

The findings are tested in the math domain, training a 1B OLMo-2 LLM on up to 200B tokens of mixed math and general corpora. Results show:

• The predictive scaling law persists: lower pretraining loss corresponds to higher post-RL performance at fixed compute.
• RL reward slope again scales with the amount of pretraining.

  Figure 9: In the math domain, the scaling law generalizes: longer pretraining produces steeper RL reward curves and higher eventual performance.

Implications and Prospects

This research has both practical and theoretical implications:

• Compute Allocation: The derived scaling law provides principled guidance for distributing compute between pretraining and RL. RL should be allocated an increasing share as total compute rises, but is only effective after sufficient pretraining exposure.
• Mechanistic Impact: RL is not a uniform sharpening; it can promote new behaviors but also amplify wrong policies, especially where payoffs are sparse or rewards are misaligned.
• Limitations: The chess testbed offers controlled, verifiable insights, but differences in vocabulary and task structure limit direct transference of quantitative exponents to natural language settings. Scaling trends may deviate at larger parameter counts or with different reward structures.
• Further Development: Enhancing RL beyond pass@1 likely requires innovations that expand correct-solution support and reduce wrong-mode amplification, not just further sharpening. Interleaved or adaptive training recipes—rather than rigid two-stage pipelines—may yield better generalization and efficiency.

Conclusion

By constructing an LLM-analogous, verifiable environment in chess, this paper delivers a quantitative account of the pretraining-to-RL interface in reasoning. A rigorous joint scaling law empirically links pretraining metrics to RL improvement rate, providing actionable compute-allocation guidance and mechanistic understanding. Extensions to the math domain indicate that these patterns reflect underlying properties of reasoning-capable models more broadly. Future directions include refined compute-allocation strategies, improved RL methods for broadening policy support, and further investigations into testbed environments for dissecting alignment and generalization challenges in LLMs.