Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a central paradigm for scaling LLM reasoning, yet its optimization often suffers from training instability and suboptimal convergence. Through a systematic dissection of clipping-based GRPO-style objectives, we identify the rigid clipping decision induced by hard clipping as a key practical bottleneck in the studied RLVR setups. Specifically, our analysis suggests that informative signals can lie in the near-boundary region just beyond the clipping threshold, and are therefore discarded by the standard hard-clipping rule. Notably, once this bottleneck is precisely identified, even simple stochastic perturbations at the boundary can recover meaningful performance gains. Building on this finding, we propose Near-boundary Stochastic Rescue (NSR), a minimal, plug-and-play modification that stochastically retains these slightly out-of-bound tokens to recover lost signals. While NSR, via stochastic sampling, can be interpreted as inducing an implicit gradient decay in expectation, our ablations reveal that its stochastic, boundary-local rescue mechanism is consistently more effective than deterministic gradient decay. Validated by extensive experiments across model sizes from 7B to 30B and both dense and MoE architectures, as a plug-and-play solution, NSR substantially improves training stability and delivers consistent gains over strong baselines such as DAPO and GSPO.

• The paper identifies the clipping mask as the primary instability source in RLVR and shows that hard clipping discards valuable near-boundary gradients.
• It introduces Near-boundary Stochastic Rescue (NSR), which stochastically reseeds marginally out-of-bound token updates to recover informative gradients.
• Extensive experiments confirm NSR improves training stability, sample efficiency, and reasoning performance across various LLM architectures.

Clipping Bottleneck and NSR: Advancing Stability in RLVR

Motivation and Diagnosis

Reinforcement Learning with Verifiable Rewards (RLVR) constitutes a foundational approach for scaling LLM reasoning, leveraging deterministic verifiers for robust supervision in domains such as mathematics and coding. Widely-used policy optimization objectives in RLVR—specifically, GRPO-style clipping (e.g., DAPO, GSPO)— impose token- and sequence-level trust region constraints via hard clipping, which zeroes out gradient flow for policy ratios outside an admissible interval. These objectives are theoretically justified but empirically, they exhibit high sensitivity to hyperparameters and significant instability, manifesting as entropy collapse, reproducibility issues, and oscillatory optimization trajectories.

A systematic analysis presented in this work reveals that the primary bottleneck is the binary clipping decision. In particular, informative learning signals may reside just outside the clipping thresholds, but hard clipping discards them wholesale, failing to discriminate marginal from severe out-of-bound violations. Empirical interventions demonstrate optimization is robust to gradient magnitude, but hypersensitive to the clipping mask; coupled perturbations that affect the mask lead to collapse, but decoupled noise (which preserves the clipping decision) stabilizes training and improves sample efficiency.

Mechanism: Near-boundary Stochastic Rescue (NSR)

The authors propose Near-boundary Stochastic Rescue (NSR), a plug-and-play mechanism that targets the clipping boundary. NSR operates by stochastically re-admitting out-of-bound token updates based on their proximity to the trust region. Specifically, the clipping mask is preserved according to the original importance ratio, but for tokens slightly out-of-bound, a stochastic executor samples a multiplicative perturbation and rescues gradients if the perturbed token falls within the trust region. NSR thus retains informative near-boundary learning signals, while conservative policy constraints remain intact for deep violations.

Mechanistically, NSR transforms the rigid binary gate into a probabilistic filter, where the admission probability decays with the degree of boundary violation. Ablation studies conclusively show that gains stem from recovering gradients in the rescue zone, not from exaggerating in-bound updates (push-out zone), highlighting that standard hard clipping is empirically over-aggressive.

Theoretical Analysis

NSR admits a rigorous mechanistic interpretation and theoretical characterization. In expectation, NSR induces implicit soft clipping: the stochastic process modulates the effective gradient for out-of-bound tokens by an inverse-square decay $O(1/r^2)$, yielding smooth attenuation rather than binary censorship. The mathematical derivations confirm this attenuation profile symmetrically applies to both upper and lower trust region bounds. Layered ablation distinguishes stochastic filtering from deterministic gradient decay: while deterministic attenuation (e.g., explicit decay $w(r) = (u/r)^k$) provides stability improvements, NSR outperforms in aggregate performance and robustness, due to its boundary-local probabilistic rejection of unreliable directions.

Empirical Results

Extensive experiments spanning hundreds of thousands of GPU hours and diverse LLM architectures (dense and MoE, 7B–30B scale) validate NSR's efficacy. NSR consistently improves both peak and aggregate performance across math benchmarks (AIME24/25, AMC) and general reasoning (GPQA, MMLU-Pro), substantially boosting Pass@1/Pass@16 metrics (e.g., >10% improvement in 7B models; statistically significant gains in larger MoEs). NSR maintains markedly lower clipped gradient fractions, converges faster, and exhibits more stable policy entropy compared to strong baselines (DAPO, GSPO). Qualitative analysis further indicates rescued gradients promote longer output sequences, facilitating deeper reasoning chains.

Crucially, NSR generalizes robustly across optimization granularities, adapting from token-level to sequence-level objectives. It introduces minimal computational overhead and only one additional hyperparameter (rescue window), offering a practical, boundary-local fix suitable for large-scale RLVR training regimes.

Implications and Future Directions

The identification of the clipping decision bottleneck and the deployment of NSR carry both pragmatic and theoretical implications. On the practical side, NSR provides immediate improvements in sample efficiency, training stability, and reasoning quality for RLVR-based LLMs, with minimal changes to established algorithms. Theoretically, NSR establishes that gradient-based RL fine-tuning benefits from boundary-local stochastic filtering and soft constraint mechanisms, rather than rigid binary gates—this principle may inform the design of future policy optimization objectives, especially in verifiable reward settings.

Unresolved questions include optimal tuning of the rescue window across algorithms and architectures, the interplay between NSR and MoE routing, and broader applicability to RL setups with learned value functions or dense rewards. Additionally, dynamic adaptation of NSR parameters, integration with entropy regularization, and formalization in model selection contexts represent promising directions.

Conclusion

This paper presents a rigorous diagnosis of instability in clipping-based RLVR, isolates the clipping mask as the principal bottleneck, and introduces NSR as an effective plug-and-play solution. NSR probabilistically rescues near-boundary tokens, recovers censored informative gradients, and induces a theoretically sound soft constraint mechanism. Empirical evaluation confirms its stability and superiority over deterministic attenuation schemes. NSR thus advances both the practice and understanding of verifiable reward RL for LLM reasoning, suggesting further development of boundary-local stochastic filtering paradigms is warranted.