When Sharpening Becomes Collapse: Sampling Bias and Semantic Coupling in RL with Verifiable Rewards

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) is a central paradigm for turning LLMs into reliable problem solvers, especially in logic-heavy domains. Despite its empirical success, it remains unclear whether RLVR elicits novel capabilities or merely sharpens the distribution over existing knowledge. We study this by formalizing over-sharpening, a phenomenon where the policy collapses onto limited modes, suppressing valid alternatives. At a high level, we discover finite-batch updates intrinsically bias learning toward sampled modes, triggering a collapse that propagates globally via semantic coupling. To mitigate this, we propose inverse-success advantage calibration to prioritize difficult queries and distribution-level calibration to diversify sampling via a memory network. Empirical evaluations validate that our strategies can effectively improve generalization.

• The paper identifies two mechanisms—sampling bias and semantic coupling—involved in RL Verifiable Reward that cause collapse, namely, over-sharpening of correct-answer distributions which can then suppress valid yet unsampled reasoning modes.
• Sampling bias occurs when finite batches suppress correct yet unsampled modes, while semantic coupling propagates this suppression to related queries.
• The proposed calibration strategies, inverse-success advantage calibration (IAC) and distribution-level calibration (DLC), mitigate collapse and improve performance across multiple benchmarks.

This paper develops a formal account of how Reinforcement Learning with Verifiable Rewards (RLVR) reshapes the output distribution of LLMs, identifying two mechanisms—sampling bias and semantic coupling—that convert benign distribution sharpening into over-sharpening, where correct but unsampled reasoning modes are suppressed. Building on this analysis, the authors propose two calibration strategies, inverse-success advantage calibration (IAC) and distribution-level calibration (DLC), and demonstrate consistent gains over GRPO, RLOO, DrGRPO, Reinforce++, and DAPO across six mathematical reasoning benchmarks and three Qwen3 backbones.

Formalizing moderate versus over-sharpening

The paper abstracts the output space for a query $q$ into $K_1$ mutually exclusive correct modes and $K_2$ incorrect modes, with binary verifiable rewards yielding non-negative advantages $A_+ \ge 0$ for correct modes and $A_- < 0$ for incorrect ones. A policy exhibits moderate sharpening if every correct mode's probability increases relative to the reference model $\pi_{\text{ref}}$, and over-sharpening if any correct mode's probability decreases. The authors position entropy collapse as the asymptotic limit of severe over-sharpening, thereby giving that widely discussed phenomenon a precise distributional characterization.

In the idealized infinite-sample regime, the KL-regularized RLVR objective admits a closed-form maximizer proportional to $\pi_{\text{ref}}(o|q)\exp(A(q,o)/\beta)$. A corollary shows this optimal policy always produces moderate sharpening: since the partition function is bounded above by $e^{A_+/\beta}$, no correct mode is ever suppressed. This result matters because it isolates the pathology: whatever causes collapse in practice cannot be attributed to the objective itself.

Sampling bias in finite batches

The finite-batch regime changes the picture fundamentally. With a group of $G$ Monte Carlo samples, the batch-optimal policy takes the same exponential form but with each advantage weighted by its empirical count $N_i$. For unsampled modes ($K_1$0), the update reduces to uniform scaling by $K_1$1: all unsampled modes move together, and they are suppressed whenever the batch partition function satisfies $K_1$2.

The central theoretical result bounds $K_1$3 from below. Under normalized binary advantages, the bound simplifies to

$K_1$4

where $K_1$5 measures the reference model's confidence gap between sampled correct and incorrect modes. Whenever $K_1$6—i.e., the model already performs well on the query—the partition function exceeds one and all unsampled modes are uniformly suppressed. This yields a sharp conclusion: standard advantage normalization, whose factor $K_1$7 vanishes on skewed batches, can only slow collapse; it does not remove the structural cause, because once the policy becomes confident, the optimization landscape inherently favors suppressing unvisited modes. Existing sample-filtering heuristics are reinterpreted as implicit attempts to keep $K_1$8 small.

Semantic coupling propagates collapse

Because parameters are shared, updates at one query affect semantically related queries. Using an NTK-style linearization, the induced logit shift at $K_1$9 equals $K_2$0, where $K_2$1 is the empirical gradient kernel on the batch. Under two assumptions—gradient correlations decaying by a factor $K_2$2 when shifting from $K_2$3 to $K_2$4, and diagonal dominance of self-alignment over cross-alignment—the paper derives upper bounds on logit increases for batch-unseen modes and lower bounds for batch-seen modes at $K_2$5. Their ratio $K_2$6 characterizes when over-sharpening transfers to $K_2$7: it grows with concentration of the update budget on a single overlapping mode, feature specificity ($K_2$8), and group size $K_2$9.

A counterintuitive prediction follows: larger group sizes, usually expected to stabilize training, worsen cross-query collapse. Controlled experiments confirm both mechanisms. In a softmax classification toy problem with nested Cat/Persian/Siamese classes trained only on Persian inputs, the policy collapses onto a single label under every advantage estimator tested (raw reward, mean-shifted, normalized, RLOO, Reinforce++); which label wins is determined by early sampling noise, and larger $A_+ \ge 0$0 merely delays collapse. Momentum and AdamW accelerate it dramatically—collapse occurs within roughly 20 steps under momentum regardless of group size. Held-out Siamese accuracy degrades faster with higher embedding similarity and larger $A_+ \ge 0$1, matching the predicted growth of $A_+ \ge 0$2 through the empirically measured kernel ratio $A_+ \ge 0$3.

Mitigation via inverse-success advantage calibration

IAC targets the partition-function mechanism directly. Reweighting advantages by $A_+ \ge 0$4 would exactly cancel the bias term in the bound on $A_+ \ge 0$5, but direct implementation suffers from extreme variance in sequence-level probabilities and from the fact that negative samples often contain partially correct reasoning. The practical variant therefore calibrates only positive samples:

$A_+ \ge 0$6

leaving negatives untouched. The factor $A_+ \ge 0$7 proxies query difficulty, so hard queries receive larger effective weight; $A_+ \ge 0$8 recovers the vanilla GRPO objective. Sensitivity analysis shows performance peaking near $A_+ \ge 0$9, with mild degradation beyond, and IAC consistently outperforming a filtering baseline that discards the easiest 25% of queries—hard filtering discards useful gradient signal entirely, whereas IAC rescales influence smoothly.

Distribution-level calibration and its cost

DLC addresses the feedback loop whereby suppressed modes almost never reappear in rollouts. An auxiliary memory network $A_- < 0$0 learns to predict the empirical frequency of $A_- < 0$1 pairs, and sampling proceeds from logits $A_- < 0$2, down-weighting frequently sampled modes. In the toy setting, DLC combined with IAC arrests collapse entirely and maintains diversity even under AdamW. However, the authors concede a substantial practical limitation: without kernel-level support for synchronized multi-model sampling, DLC incurs roughly a $A_- < 0$3 rollout overhead, so IAC alone is the recommended default and DLC an optional augmentation.

Empirical results

Training on DeepScaleR (~40.3k problems) with Qwen3-4B/8B/14B backbones, IAC achieves the best AVG@8 and PASS@8 in nearly all settings. Representative numbers for Qwen3-14B:

Method	AVG@8 avg	PASS@8 avg
GRPO	69.71	76.42
RLOO	69.95	76.92
Reinforce++	70.28	77.01
DrGRPO	70.18	77.03
IAC	71.15	78.81

Gains concentrate on harder benchmarks (AIME24 PASS@8 rises from 71.83–73.18 for baselines to 78.07), consistent with the claim that calibrated sharpening helps most when solving requires exploring multiple reasoning modes rather than amplifying a dominant pattern. Compatibility experiments show additive benefits: DAPO+IAC improves LightEval PASS@8 from 76.62% to 84.80%, and adding DLC raises it further to 85.15%. Entropy traces corroborate the mechanism: DAPO alone drives policy entropy to ~0.08 (near-deterministic), while DAPO+IAC stabilizes between 0.20 and 0.24.

Limitations and open questions

Several caveats bear directly on the strength of the claims. The semantic coupling analysis rests on two assumptions—kernel decay by a fixed factor $A_- < 0$4 and diagonal dominance—that are tractability devices rather than verified properties of LLMs; the transfer-kernel assumption can be relaxed to a bounded interval, but only at the cost of a scaling factor in the ratio. The theory is developed for binary verifiable rewards and single-turn settings, so extension to richer reward structures and multi-turn tool use remains open. Empirically, evaluation covers mathematical reasoning exclusively, leaving generalization of the calibration strategies to other verifiable domains untested. Finally, the $A_- < 0$5 rollout overhead of DLC means the more complete fix for the sampling-feedback loop is not the one deployed at scale, and whether cheaper approximations of the memory prior can match it is unresolved.

Conclusion

The paper provides a unified account of when RLVR acts as benign sharpening and when it destroys useful diversity: infinite-sample objectives provably preserve all correct modes, while finite batches suppress unsampled ones once the policy grows confident, and shared parameters propagate that suppression across semantically related queries. The proposed calibrations—one lightweight, one expensive—keep training in a moderate-sharpening regime and yield measurable improvements on both average accuracy and pass-rate metrics, including additive gains atop DAPO. The main open question is whether the analysis and remedies extend beyond binary-reward, single-turn mathematical reasoning.