Causal factors behind performance gains in RL-based reasoning models
Determine whether the performance enhancements observed in reinforcement learning–trained Large Reasoning Models (LRMs) are primarily caused by (i) increased exposure to established mathematical benchmark data during training, (ii) the greater inference-time compute allocated to thinking tokens, or (iii) genuine reasoning capabilities developed through reinforcement learning, by isolating and quantifying the contribution of each factor under controlled experimental conditions.
References
Currently, it is not clear whether the performance enhancements observed in recent RL-based reasoning (thinking) models are attributable to increased exposure to established mathematical benchmark data, to the significantly greater inference compute allocated to thinking tokens, or to reasoning capabilities developed by RL-based training?
Whether the additional token use contributes causally to the success increase cannot be determined from these aggregate measurements.
Three explanations compete: clipping that amplifies the model's prior for code-style reasoning; contamination, since from 60\,\% of each problem Qwen2.5-Math-7B reconstructs 54.60\,\% of MATH-500 against 3.80\,\% for Llama3.1-8B and only correct rewards give stable gains on a clean benchmark; and model strength, since the clipping bias is too small to carry a signal and gains appear on non-Qwen models too. The studies differ in prompts, clipping, and models, so the question is open, but they agree that a strong prior can override the $J$ analysis and that benchmarks older than the model can make any reward look useful.