Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward (2510.03222v1)

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has propelled LLMs in complex reasoning, yet its scalability is often hindered by a training bottleneck where performance plateaus as policy entropy collapses, signaling a loss of exploration. Previous methods typically address this by maintaining high policy entropy, yet the precise mechanisms that govern meaningful exploration have remained underexplored. Our analysis suggests that an unselective focus on entropy risks amplifying irrelevant tokens and destabilizing training. This paper investigates the exploration dynamics within RLVR and identifies a key issue: the gradual elimination of valuable low-probability exploratory tokens, which we term \textbf{\textit{reasoning sparks}}. We find that while abundant in pre-trained models, these sparks are systematically extinguished during RLVR due to over-penalization, leading to a degeneracy in exploration. To address this, we introduce Low-probability Regularization (Lp-Reg). Its core mechanism regularizes the policy towards a heuristic proxy distribution. This proxy is constructed by filtering out presumed noise tokens and re-normalizing the distribution over the remaining candidates. The result is a less-noisy proxy where the probability of \textit{reasoning sparks} is amplified, which then serves as a soft regularization target to shield these valuable tokens from elimination via KL divergence. Experiments show that Lp-Reg enables stable on-policy training for around 1,000 steps, a regime where baseline entropy-control methods collapse. This sustained exploration leads to state-of-the-art performance, achieving a $60.17\%$ average accuracy on five math benchmarks, an improvement of $2.66\%$ over prior methods. Code is available at https://github.com/CarlanLark/Lp-Reg.

• The paper introduces Lp-Reg, a method that selectively preserves low-probability tokens to counteract policy entropy collapse in RLVR.
• Empirical results show Lp-Reg outperforms entropy-based methods, achieving a 2.66% increase in accuracy on benchmarks like Qwen3-14B-Base.
• The approach is robust and computationally efficient, enabling scalable reinforcement learning without extensive hyperparameter tuning.

Low-Probability Token Regularization for Sustained Exploration in RLVR

Introduction

The paper "Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward" (2510.03222) addresses a critical bottleneck in scaling Reinforcement Learning with Verifiable Rewards (RLVR) for LLMs in complex reasoning tasks. The authors identify that standard RLVR training, particularly with Group-Relative Policy Optimization (GRPO), leads to a rapid collapse in policy entropy and a plateau in performance. This collapse is traced to the systematic elimination of semantically meaningful, low-probability tokens—termed "reasoning sparks"—which are essential for initiating diverse reasoning trajectories. The indiscriminate application of entropy bonuses exacerbates this issue by amplifying irrelevant noise tokens, further destabilizing training. To resolve this, the paper introduces Low-probability Regularization (Lp-Reg), a method that selectively preserves valuable low-probability tokens by regularizing the policy towards a filtered, renormalized proxy distribution.

Figure 1: Selectively preserving low-probability tokens is key to overcoming performance plateaus in reasoning RL. Lp-Reg maintains a healthy distribution of reasoning sparks, while entropy-based methods collapse or amplify noise.

Mechanism of Low-Probability Regularization

Proxy Distribution Construction

Lp-Reg operates by constructing a proxy distribution $\pi_{\text{proxy}}$ from the current policy $\pi_{\boldsymbol{\theta}}$. The process involves:

1. Filtering Noise Tokens: Tokens with probability below a threshold $\tau$ are considered noise and are excluded. The threshold can be fixed or adaptively set as a fraction of the maximum token probability (min-p threshold), enhancing robustness to distributional sharpness.
2. Probability Renormalization: The probability mass of filtered tokens is redistributed among the remaining candidates, amplifying the relative probability of reasoning sparks.

   Figure 2: Probability renormalization: $\pi_{\text{proxy}}$ assigns zero probability to tokens with $\pi_{\boldsymbol{\theta}} \leq \tau$ and renormalizes over the rest.

Regularization Objective

The Lp-Reg penalty is integrated into the GRPO framework via a forward KL divergence:

$$J(\boldsymbol{\theta}) = \text{GRPO objective} - \beta \cdot I[\text{low-probability, proxy-preserved, negative advantage}] \cdot \mathcal{D}_{\mathrm{KL}}(\pi_{\text{proxy}} \Vert \pi_{\boldsymbol{\theta}})$$

Regularization is applied only to tokens that are (i) low-probability under the current policy, (ii) preserved in the proxy, and (iii) receive negative advantage signals. This selective approach prevents over-penalization and ensures that only valuable exploratory tokens are protected.

Empirical Results and Training Dynamics

Lp-Reg demonstrates superior and stable performance across five mathematical reasoning benchmarks (AIME24, AIME25, MATH-500, OlympiadBench, Minerva Math) on both Qwen3-14B-Base and Qwen2.5-32B models. On Qwen3-14B-Base, Lp-Reg achieves $60.17\%$ average accuracy, outperforming the next best method by $2.66\%$. Notably, Lp-Reg enables stable on-policy training for $\sim$1,000 steps, a regime where entropy-control baselines collapse.

Figure 3: Training dynamics on Qwen3-14B-Base: Lp-Reg yields more stable and higher performance than entropy-based methods.

The entropy trajectory under Lp-Reg is adaptive: initial entropy decreases as the model learns core reasoning, then increases to foster exploration, and finally stabilizes. In contrast, entropy-based methods either collapse or induce excessive noise.

Ablation and Analysis

Ablation studies confirm the necessity of noise filtering and the superiority of dynamic min-p thresholds over fixed ones. Removing the filter leads to entropy explosion and performance collapse. Regularizing high-entropy tokens, rather than low-probability ones, fails to sustain exploration and degrades performance.

Figure 4: Ablation studies: Filtering with adaptive $\tau$ is critical for stable performance; fixed thresholds are less robust.

Forward KL regularization outperforms reverse KL, as the latter forces strict imitation of the heuristic proxy, constraining exploration.

Figure 5: Forward KL yields better performance than reverse KL, providing soft guidance rather than strict imitation.

Hyperparameter sensitivity analysis shows Lp-Reg is robust to the choice of percentile $\rho$ and min-p ratio $\kappa$, indicating practical deployability without extensive tuning.

Figure 6: Training dynamics of Lp-Reg with different hyperparameters demonstrate robustness.

Token-Level Exploration Dynamics

Qualitative and quantitative analyses reveal that low-probability tokens include semantically meaningful reasoning sparks (e.g., "wait", "perhaps"), while high-entropy tokens are dominated by functional or formatting artifacts. Lp-Reg maintains a healthy distribution and frequency of reasoning sparks throughout training, whereas entropy-based methods either collapse or amplify irrelevant noise.

Figure 7: Word cloud statistics: Low-probability tokens are rich in reasoning markers, high-entropy tokens are not.

Figure 8: Probability–entropy scatter plots: Lp-Reg maintains a broad, balanced distribution for reasoning sparks.

Figure 9: Frequency of explorative tokens: Lp-Reg sustains higher usage of reasoning sparks during training.

A persistent probabilistic gap exists between reasoning sparks and noise tokens, justifying the use of a probability threshold for filtering.

Implementation Considerations

• Computational Overhead: Lp-Reg introduces minimal overhead, as proxy construction and KL regularization are lightweight and parallelizable.
• Integration: The method is compatible with standard policy gradient frameworks (GRPO, PPO) and can be applied in both on-policy and off-policy settings.
• Scaling: Lp-Reg is robust across model scales (14B, 32B) and does not require extensive hyperparameter tuning.
• Deployment: The approach is suitable for production RLVR pipelines, especially where sustained exploration and reasoning diversity are critical.

Implications and Future Directions

The findings challenge the conventional reliance on entropy as a proxy for exploration in RL for LLMs. By demonstrating that meaningful exploration is driven by the selective preservation of reasoning sparks, the work suggests that future RL algorithms should incorporate token-level semantic filtering rather than indiscriminate entropy maximization. The Lp-Reg framework can be extended to other domains where exploration quality is paramount, such as code generation, scientific reasoning, and multi-agent dialogue. Further research may explore adaptive, context-aware filtering strategies and the integration of external semantic priors to enhance proxy construction.

Conclusion

This paper provides a rigorous analysis of exploration collapse in RLVR for LLMs, identifying the elimination of reasoning sparks as the root cause. The proposed Lp-Reg method selectively preserves valuable low-probability tokens, enabling stable and superior performance in complex reasoning tasks. The approach is robust, computationally efficient, and broadly applicable, marking a significant step towards scalable, high-quality RL for LLMs.