Mechanistic Effect of RL Post-Training on Pretrained Policies

Characterize how reinforcement learning with verifiable rewards changes the behavior of pretrained large language models for reasoning by determining whether RL primarily sharpens existing action preferences or discovers new, previously low-probability solution modes, and identify the conditions under which each behavior occurs.

Background

There are competing views on how RL affects pretrained reasoning models: some studies suggest RL mainly sharpens existing preferences, while others argue RL composes new skills. The lack of step-level supervision and massive action spaces in natural language make this mechanistic question difficult to resolve.

A precise characterization of how RL reshapes the inherited policy—at the level of distributions over actions and the structure of reasoning traces—would clarify when RL provides genuine discovery versus amplification and guide post-training strategies.

References

As a result, two basic questions remain open: (1) how do pretraining choices (model size, data) shape the returns to RL compute, and (2) what does RL actually do to the model?

— Understanding Reasoning from Pretraining to Post-Training  (2607.16097 - Shen et al., 17 Jul 2026) in Abstract

While we have validated SignBalance across settings (math reasoning and search-agent) and across model scales (0.5B, 3B, and 7B), and we empirically observe that our method drives the policy toward more effective reasoning strategies, we have not yet quantitatively measured the resulting growth in reasoning ability, such as the policy's reasoning confidence and its uncertainty behavior across problem difficulty levels.

— Spurious Advantage Hidden in GRPO  (2609.04063 - Wang et al., 3 Sep 2026) in Section Limitations

RL has to discover the behaviour before it can reinforce it, so a failure under RL leaves open whether the model is incapable or merely unable to find it.

— Learning Steganography Is Easy, Learning Steganographic Reasoning Is Hard  (2609.39838 - Schulz et al., 30 Sep 2026) in Section 3.3, paragraph opening "RL has to discover the behaviour"

The first attributes the gains to data contamination \citep{yanSpuriousRewardsParadox2026}. The gains we observe on synthetically generated problems suggest, however, that the phenomenon cannot be attributed entirely to data leakage. The second looks to particular mechanisms within RL training. \citet{shaoSpuriousRewardsRethinking2026} conjecture that clipping in GRPO \citep{shaoDeepSeekMathPushingLimits2024} favors behavior to which the model already assigns high probability. \citet{chenExplorationVsExploitation2025}, however, argue that clipping bias does not explain the gains.

— Probe with Participation Trophies: Random-Reward RL as a Probe of LLM Capability  (2610.01066 - Mao et al., 1 Oct 2026) in Section 1, Introduction