Inverse-Success Advantage Calibration in RLVR

• The paper introduces inverse-success advantage calibration as a policy-gradient correction method that mitigates the over-sharpening effect in RLVR by adjusting advantages based on empirical success rates.
• It mathematically defines a calibration weight to downscale the influence of over-represented successful trajectories, thereby preserving unsampled yet valid solution modes.
• Empirical results across benchmarks show that IAC improves metrics like AVG@8 and PASS@8, demonstrating its robustness and effectiveness against mode collapse.

Inverse-success advantage calibration is a policy-gradient correction technique introduced to address the over-sharpening phenomenon in reinforcement learning with verifiable rewards (RLVR). This method explicitly targets the risk of finite-batch sampling bias that drives the policy to collapse onto a restricted set of modes, thereby suppressing diverse and potentially correct outputs. By dynamically adjusting the advantages of successful trajectories based on empirical success rates, inverse-success advantage calibration ensures fair treatment across sampled and unsampled modes and maintains the retention of valid solution candidates throughout RLVR training (Fan et al., 22 Jan 2026).

1. Sampling Bias and Over-Sharpening in RLVR

Within the RLVR paradigm, a model is updated by sampling a batch of trajectories for each query and assigning advantages based on a verifiable reward signal. The finite-batch regime introduces an "over-sharpening" effect: as soon as the model achieves high accuracy on a query (i.e., the empirical success rate $p^+$ approaches 1), the normalization factor (or partition function) becomes dominated by a few sampled modes. According to Theorem 3 of (Fan et al., 22 Jan 2026), the batch-partition function

$$Z'(q) = \sum_k \pi_\text{ref}(o_k|q)\exp\left(\frac{N_k A_k}{\beta G}\right)$$

can grow large, causing probabilities of all unsampled modes—including semantically correct ones—to rapidly decay. This structural failure mode leads to the permanent loss of valuable solution trajectories, particularly in logic-heavy settings where solution diversity is critical.

2. Mathematical Formulation of Inverse-Success Advantage Calibration

Inverse-success advantage calibration (IAC) modifies the policy-gradient update by down-weighting the contribution of successful trajectories in direct proportion to their empirical prevalence. For each query $q$ and batch of $G$ rollouts $o^{(1)},\ldots,o^{(G)}\sim\pi_\theta(\cdot|q)$, with binary rewards $r_s\in\{0,1\}$ and computed advantages $A_s$, let $S^+=\{s:r_s=1\}$ and $p^+=|S^+|/G$. The vanilla generalized reinforce plus off-policy (GRPO) gradient is:

$$\sum_{s=1}^G A_s \nabla_\theta \log \pi_\theta(o^{(s)}|q)$$

IAC introduces a calibration strength parameter $$Z'(q) = \sum_k \pi_\text{ref}(o_k|q)\exp\left(\frac{N_k A_k}{\beta G}\right)$$0 and computes

$$Z'(q) = \sum_k \pi_\text{ref}(o_k|q)\exp\left(\frac{N_k A_k}{\beta G}\right)$$1

The calibrated advantage is

$$Z'(q) = \sum_k \pi_\text{ref}(o_k|q)\exp\left(\frac{N_k A_k}{\beta G}\right)$$2

or equivalently per-sample, $$Z'(q) = \sum_k \pi_\text{ref}(o_k|q)\exp\left(\frac{N_k A_k}{\beta G}\right)$$3 if $$Z'(q) = \sum_k \pi_\text{ref}(o_k|q)\exp\left(\frac{N_k A_k}{\beta G}\right)$$4, $$Z'(q) = \sum_k \pi_\text{ref}(o_k|q)\exp\left(\frac{N_k A_k}{\beta G}\right)$$5 otherwise. This adjustment ensures that, when the query is nearly solved, the positive-advantage contributions are suppressed to offset the risk of global mode collapse.

3. Calibrated Policy-Gradient Update and Implementation

For a mini-batch of $$Z'(q) = \sum_k \pi_\text{ref}(o_k|q)\exp\left(\frac{N_k A_k}{\beta G}\right)$$6 queries, each with $$Z'(q) = \sum_k \pi_\text{ref}(o_k|q)\exp\left(\frac{N_k A_k}{\beta G}\right)$$7 rollouts, IAC is realized via the following steps:

• Identify the sets of successes $$Z'(q) = \sum_k \pi_\text{ref}(o_k|q)\exp\left(\frac{N_k A_k}{\beta G}\right)$$8 and failures $$Z'(q) = \sum_k \pi_\text{ref}(o_k|q)\exp\left(\frac{N_k A_k}{\beta G}\right)$$9 for each query $q$0.
• Compute the empirical batch success rate $q$1 and calibration weight $q$2.
• For each successful trajectory, scale its advantage: $q$3 if $q$4; otherwise, leave it unchanged.
• The aggregate policy-gradient becomes:

$q$5

This update is accompanied by algorithmic pseudocode specifying batch processing, dynamic weighting, and KL regularization to the reference policy. An optional extension incorporates distribution-level calibration via a memory network $q$6, which tracks and regularizes empirical sampling frequencies by modifying the policy logits before sampling.

4. Theoretical Guarantee Against Over-Sharpening

IAC is constructed such that, after each finite-batch update, all unsampled modes retain their original probability mass relative to the reference distribution. The lower-bound proof (Theorem 3) shows

$q$7

which directly yields, for any unsampled mode $q$8 with $q$9, the update rule

$G$0

Consequently, no unsampled correct mode is ever suppressed, and the structural cause of over-sharpening is eliminated in the finite-batch regime. This property provides robustness, ensuring that correct behaviors are never inadvertently forgotten, independent of regret guarantees.

5. Empirical Validation Across Mathematical Reasoning Benchmarks

IAC has been empirically evaluated on six benchmarks: GSM8K, MATH500, LightEval, AIME24, Minerva, and O-Bench, using Qwen3 models at 4B, 8B, and 14B parameter scales. Key metrics include AVG@8 (average correct rate across eight samples per query) and PASS@8 (fraction of queries with at least one correct answer in eight tries). Across all benchmarks and model sizes, IAC consistently outperforms strong RLVR baselines such as GRPO, RLOO, Reinforce++, and DrGRPO. For example, Qwen3-14B achieves AVG@8 improving from 70.28% (best baseline) to 71.15% and PASS@8 from 77.03% to 78.81%. Gains are accentuated on more challenging tasks (AIME24, Minerva, O-Bench). Furthermore, IAC is complementary to DAPO entropy-safe clipping: on LightEval, DAPO+IAC achieves PASS@8 of 84.80%, surpassing DAPO alone at 76.62%, with further improvement (85.15%) upon introducing distribution-level calibration (memory network, $G$1).

6. Calibration Strength, Entropy, and Semantic-Coupling Analyses

Experimental ablations demonstrate that calibration strength $G$2 critically tunes performance. $G$3 reduces to GRPO; values in $G$4 yield steady improvements, peaking around $G$5. Excessive suppression beyond $G$6 degrades performance. Policy entropy analyses reveal that DAPO alone causes entropy collapse ($G$7), while DAPO+IAC stabilizes entropy in the $G$8 range, preventing premature determinization. Toy semantic-coupling simulations further validate that IAC (at $G$9) dramatically delays the collapse of semantically linked modes, and with memory-based distribution-level calibration, collapse is nearly arrested.

7. Significance and Generality

Inverse-success advantage calibration provides a principled, lightweight mechanism for mitigating the collapse of valid solution modes under RLVR finite-batch updates. It is agnostic to model architecture and baseline selection, easily integrated into existing RLVR pipelines, and does not require problem-specific tuning apart from the calibration strength $o^{(1)},\ldots,o^{(G)}\sim\pi_\theta(\cdot|q)$0. Empirical and theoretical analysis jointly substantiate its role in preserving solution diversity and improving final task performance, while being complementary to entropy-based and distribution-level regularization techniques (Fan et al., 22 Jan 2026).