Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data

Abstract: Reinforcement Learning (RL) enhances LLM reasoning, yet a paradox emerges as models scale: strong base models saturate standard benchmarks (e.g., MATH), yielding correct but homogeneous solutions. In such environments, the lack of failure cases causes the advantage signal in group-relative algorithms (e.g., GRPO) to vanish, driving policies into mode collapse. To address this, we propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration. Unlike standard sampling that follows model biases, CUTS flattens the local optimization landscape by sampling uniformly from constrained high-confidence candidates. We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance. Experiments on Qwen3 models demonstrate that our approach prevents policy degeneration and significantly boosts out-of-domain generalization. Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO, validating that maintaining diversity within the semantic manifold is critical for rigorous reasoning.

• The paper introduces Mixed-CUTS to mitigate vanishing gradients in reinforcement learning by restoring reward variance with constrained uniform top-K sampling.
• It integrates CUTS in a dual-rollout architecture, achieving up to a +15.1% increase in Pass@1 on AIME benchmarks with minimal parameter tuning.
• The approach enhances generalization on both mathematical and out-of-domain tasks, providing a scalable and cost-effective decoding-level intervention for LLMs.

Reinforcement Learning with Homogeneous Correctness: Mitigating Mode Collapse in LLM Reasoning via Decoding-Level Diversification

Motivation and Problem Diagnosis

Advancements in LLMs have led to regimes where base models, when fine-tuned on canonical reasoning datasets such as MATH, achieve high near-perfect accuracy. This ubiquity of correctness induces a previously under-explored pathology in reinforcement learning for reasoning: the saturation-induced vanishing of advantage signals in group-relative algorithms (notably GRPO). When all or most trajectories for a prompt are uniformly correct, intra-group reward variance collapses. As a consequence, policy gradients vanish and the model quickly converges to a mode-collapsed behavior, repeatedly sampling homogeneous solutions from a local optimum, with weakened generalization capacity and stunted further learning. Standard interventions, including entropy regularization, fail under these circumstances by incentivizing incoherent diversity rather than restoring meaningful exploration.

Mixed-CUTS: Constrained Uniform Top-K Sampling

To address the intrinsic limitations of RL on saturated datasets, the authors propose Constrained Uniform Top-K Sampling (CUTS), an inference-time decoding operator that instantiates structure-preserving exploration. At each decoding step, CUTS selects the Top-$K$ high-probability tokens, filters them through a minimum probability threshold $\delta$ for semantic coherence, and samples uniformly from this constrained set. This process, devoid of additional trainable parameters, forces local exploration orthogonal to the model’s current bias, enabling the generation of valid yet under-represented trajectories.

CUTS is integrated as an exploratory stream in a dual-rollout training architecture, Mixed-CUTS, which partitions each rollout group into standard exploitative and CUTS-driven explorative batches. All generated responses contribute to computing group advantages, ensuring that even in “all-correct” regimes, reward variance––and thus effective gradients––is restored.

Figure 1: The Mixed-CUTS framework: exploitative trajectories reflect current policy bias, while CUTS-enforced exploratory rollouts structurally restore intra-group advantage variance under saturated correctness.

The key theoretical result, presented via a total variance decomposition, guarantees that as long as CUTS produces trajectories with distinct expected rewards compared to the standard batch, intra-group variance is strictly bounded away from zero. Hence, informative advantage signals are structurally preserved—even in degenerate “all-correct” or “all-fail” settings where classic GRPO is ineffective.

Empirical Evaluation and Numerical Results

The efficacy of Mixed-CUTS is validated on Qwen3-1.7B and Qwen3-4B models, trained on MATH and evaluated on five benchmarks (MATH, AIME24, AIME25, AMC, GPQA). The framework consistently outperforms the standard GRPO approach, with up to +15.1% absolute gain in Pass@1 accuracy on AIME25 for Qwen3-4B (41.7% vs. 26.6%). These improvements extend to multi-sample and consistency-based measures (Pass@16, maj@16), affirming that the model not only finds the correct answer more frequently but does so robustly and repeatedly, indicating improved internal coherence instead of mere stochastic success.

The performance advantage of Mixed-CUTS persists even on hard, non-mathematical general reasoning tasks, as zero-shot transfer evaluations on MMLU-Pro and SuperGPQA show continued improvements over GRPO. A key finding is that the gain from CUTS is largest in regimes where high-quality “hard” data is scarce and the training environment is saturated.

Figure 2: Training dynamics on Qwen3-4B: Mixed-CUTS (orange) sustains higher policy entropy and produces longer reasoning chains compared to GRPO (grey), directly translating to higher out-of-domain generalization and stronger majority vote consistency.

CUTS is robust to its hyperparameters, with stable performance across a range of $K$ and $\delta$ settings as long as the candidate set remains semantically coherent. When deployed alongside high-quality hard data, gains persist but are smaller, supporting the claim that decoding-level interventions are orthogonal and complementary to increased data difficulty.

Theoretical and Practical Implications

The principal theoretical implication is a formal diagnosis and mitigation of saturation-induced collapse in RL for reasoning LLMs. By demonstrating that homogeneous correctness can represent an anti-signal in RL objectives, the results urge a careful reconsideration of standard RL pipelines in high-capacity, saturated regimes. Rather than focusing purely on objective modification (e.g., via extraneous entropy terms or altered reward criteria) or on curating ever more difficult datasets, the paper positions the decoding process as a critical and underappreciated locus for structural exploration interventions.

Practically, the findings support scalable and robust reasoning RL pipelines that do not require modification of model architecture or extensive tuning. The parameter-free nature of CUTS, operating solely at decoding time, enhances the portability and cost-effectiveness of the approach. The improvements in sample efficiency, consistency, and out-of-domain transferability are of particular value for continued scaling and deployment of reasoning-augmented LLMs.

Future Directions

The paper suggests that decoding-level structural interventions may generalize beyond mathematical reasoning to other domains, e.g., code generation and agentic multi-turn planning, where policy diversity and robust exploration are pivotal. Open theoretical questions include a more complete analysis of how CUTS-induced policy deviation interacts with global RL convergence properties and the design of formal diversity metrics beyond local flattening. There is also scope for principled integration with alternative reward structures and exploration heuristics, especially in domains with explicit multimodality or value misalignment.

Conclusion

The work delivers a comprehensive framework and empirical validation for mitigating the “too correct to learn” pathology in RL-based LLM reasoning. By introducing and integrating CUTS, the approach provides structural, parameter-free exploration that addresses vanishing-advantage mode collapse without sacrificing coherence. The strong, domain-general improvements observed suggest that decoding-time interventions will become a necessary component of effective RL for high-capacity LLMs operating under saturated training conditions.