Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models (2509.26114v1)

Abstract: Reinforcement learning with verifiable rewards (RLVR) has recently emerged as the leading approach for enhancing the reasoning capabilities of LLMs. However, RLVR is prone to entropy collapse, where the LLM quickly converges to a near-deterministic form, hindering exploration and progress during prolonged RL training. In this work, we reveal that the clipping mechanism in PPO and GRPO induces biases on entropy. Through theoretical and empirical analyses, we show that clip-low increases entropy, while clip-high decreases it. Further, under standard clipping parameters, the effect of clip-high dominates, resulting in an overall entropy reduction even when purely random rewards are provided to the RL algorithm. Our findings highlight an overlooked confounding factor in RLVR: independent of the reward signal, the clipping mechanism influences entropy, which in turn affects the reasoning behavior. Furthermore, our analysis demonstrates that clipping can be deliberately used to control entropy. Specifically, with a more aggressive clip-low value, one can increase entropy, promote exploration, and ultimately prevent entropy collapse in RLVR training.

• The paper demonstrates that clip-low increases policy entropy while clip-high decreases it, affecting the exploration capacity in RL training of LLMs.
• It employs rigorous theoretical and empirical analyses using models like Qwen and Llama across random and true reward settings to validate its findings.
• Practical implications include tuning clipping parameters to balance exploration and exploitation without requiring extra entropy or KL regularization.

Entropy Dynamics Induced by Clipping in RL Training of LLMs

Introduction

This paper provides a rigorous theoretical and empirical analysis of the impact of clipping mechanisms—specifically, clip-low and clip-high—on policy entropy during reinforcement learning with verifiable rewards (RLVR) for LLMs. The authors demonstrate that clip-low increases entropy, promoting exploration, while clip-high decreases entropy, leading to entropy collapse. These findings are substantiated both in toy settings with random rewards and in practical RLVR scenarios for mathematical reasoning tasks. The work clarifies a previously underexplored confounding factor in RLVR: the clipping mechanism itself, independent of the reward signal, exerts a direct influence on entropy and, consequently, on the exploration-exploitation trade-off in LLM reasoning.

Theoretical Analysis of Clipping Effects

The paper formalizes RLVR as an MDP where the LLM policy $\pi_\theta$ generates token sequences, and the objective is to maximize expected reward. The standard policy gradient estimator (REINFORCE) is unbiased under random rewards, but practical RL algorithms such as PPO and GRPO introduce clipping to stabilize training. The authors analyze the first-order effect of clipping on policy entropy under the assumption of random, symmetric rewards.

The key theoretical results are:

• Clip-low (lower bound on negative advantages) increases entropy.
• Clip-high (upper bound on positive advantages) decreases entropy.
• Under typical symmetric clipping ($$\varepsilon_{\mathrm{low}} = \varepsilon_{\mathrm{high}}$$), the entropy-reducing effect of clip-high dominates, even with random rewards.

These claims are supported by first-order Taylor expansions of entropy with respect to policy updates, separating the contributions of clip-low and clip-high. The conditions under which these effects hold are empirically validated.

Figure 1: Empirical estimates of $\mathbb{E}[Q]-\mathbb{E}[Q\,|\,X_k]$ and $\mathbb{E}[Q]-\mathbb{E}[Q\,|\,Y_k]$ during RL training with random rewards, confirming the theoretical sign conditions for entropy change.

Figure 2: Estimated values of the log-conditional entropy terms, showing persistent positivity and thus validating the theoretical predictions for both Qwen and Llama models.

Empirical Validation: Random Reward Setting

The authors conduct extensive experiments using the verl framework, training Qwen2.5-1.5B-Instruct and Llama3-8B-Instruct models on GSM8K with random Bernoulli rewards. The results confirm the theoretical predictions:

• Strengthening clip-low (decreasing $\varepsilon_{\mathrm{low}}$) increases entropy.
• Strengthening clip-high (decreasing $\varepsilon_{\mathrm{high}}$) decreases entropy.
• With symmetric clipping, entropy consistently decreases, regardless of the reward distribution or model family.

Figure 3: Change of policy entropy during RL training with random rewards and different clipping settings, demonstrating the opposing effects of clip-low and clip-high.

Figure 4: Entropy change across different base models and reward distributions, showing that entropy minimization is a general effect of clipped RL training.

RLVR with True Rewards: Mathematical Reasoning Tasks

Extending the analysis to RLVR with true rewards, the paper shows that the entropy-reducing effect persists, but can be counteracted by tuning the clipping parameters. Specifically:

• Disabling clip-high and reducing $\varepsilon_{\mathrm{low}}$ can maintain or even increase entropy, preventing entropy collapse.
• Controlled entropy via clipping preserves or improves pass@k metrics (exploration) without degrading mean@k (single-response accuracy).

Figure 5: Entropy change during RLVR with GSM8K and Qwen2.5-3B-Instruct, showing the effect of ablating and tuning clipping mechanisms.

Figure 6: RLVR performance on GSM8K for Qwen2.5-3B-Instruct and Llama3-8B-Instruct, showing that higher entropy correlates with improved pass@8 (exploration) while mean@8 remains stable.

Figure 7: RLVR performance on AMC and MATH-500 for Qwen2.5-7B-Instruct, demonstrating simultaneous improvement in mean@32 and pass@32 metrics via entropy control.

Figure 8: Policy entropy change during RL training with Llama base models, confirming the generality of the clipping-induced entropy dynamics.

Implementation and Practical Implications

The findings have direct implications for RLVR training of LLMs:

• Clipping parameters ($\varepsilon_{\mathrm{low}}$, $\varepsilon_{\mathrm{high}}$) should be treated as primary controls for policy entropy.
• Aggressive clip-low can be used to prevent entropy collapse and maintain exploration, especially in prolonged RLVR training.
• Entropy control via clipping is effective even without explicit KL or entropy regularization losses.
• Performance optimization should consider both mean@k and pass@k metrics, as excessive entropy reduction narrows the reasoning boundary and degrades exploration.

The paper provides detailed experimental configurations and ablation studies, enabling reproducibility and practical adoption in RLVR pipelines for LLMs.

Theoretical and Empirical Trade-offs

• Sample Efficiency vs. Entropy Control: While unclipped REINFORCE avoids entropy bias, it is less sample-efficient and less stable than PPO/GRPO with clipping.
• Exploration vs. Exploitation: Excessive entropy reduction (clip-high dominance) leads to deterministic policies and reduced exploration, while excessive entropy increase (aggressive clip-low) can cause entropy explosion and unstable training.
• Model-Agnostic Effects: The entropy dynamics induced by clipping are observed across Qwen, Llama, and Olmo model families, and are robust to reward distribution.

Future Directions

The work suggests several avenues for further research:

• Refinement of theoretical analysis to relax assumptions and address edge cases where the sign conditions may not hold.
• Automated tuning of clipping parameters to optimize both exploration and exploitation dynamically during RLVR.
• Investigation of entropy-performance trade-offs in broader reasoning domains and with more complex reward structures.
• Integration with other exploration-promoting techniques such as intrinsic motivation and reward shaping.

Conclusion

This paper establishes that the clipping mechanism in PPO and GRPO exerts a direct, controllable influence on policy entropy in RLVR for LLMs. Clip-low increases entropy and promotes exploration, while clip-high decreases entropy and accelerates entropy collapse. These effects are robust across model families and reward distributions, and can be leveraged to maintain exploration and prevent reasoning boundary degradation during RLVR. The results provide a principled foundation for entropy control in RL training of LLMs, with significant implications for both theoretical understanding and practical deployment.