---
title: Inverse-Success Advantage Calibration in RLVR
url: https://www.emergentmind.com/topics/inverse-success-advantage-calibration
type: topic
---

# Inverse-Success Advantage Calibration in RLVR

Inverse-success advantage calibration is a policy-gradient correction technique introduced to address the over-sharpening phenomenon in reinforcement learning with verifiable rewards (RLVR). This method explicitly targets the risk of finite-batch sampling bias that drives the policy to collapse onto a restricted set of modes, thereby suppressing diverse and potentially correct outputs. By dynamically adjusting the advantages of successful trajectories based on empirical success rates, inverse-success advantage calibration ensures fair treatment across sampled and unsampled modes and maintains the retention of valid solution candidates throughout RLVR training [2601.15609].

## 1. Sampling Bias and Over-Sharpening in RLVR

Within the RLVR paradigm, a model is updated by sampling a batch of trajectories for each query and assigning advantages based on a verifiable reward signal. The finite-batch regime introduces an "over-sharpening" effect: as soon as the model achieves high accuracy on a query (i.e., the empirical success rate $p^+$ approaches 1), the normalization factor (or partition function) becomes dominated by a few sampled modes. According to Theorem 3 of [2601.15609], the batch-partition function
$$
Z'(q) = \sum_k \pi_\text{ref}(o_k|q)\exp\left(\frac{N_k A_k}{\beta G}\right)
$$
can grow large, causing probabilities of all unsampled modes—including semantically correct ones—to rapidly decay. This structural failure mode leads to the permanent loss of valuable solution trajectories, particularly in logic-heavy settings where solution diversity is critical.

## 2. Mathematical Formulation of Inverse-Success Advantage Calibration

Inverse-success advantage calibration (IAC) modifies the policy-gradient update by down-weighting the contribution of successful trajectories in direct proportion to their empirical prevalence. For each query $q$ and batch of $G$ rollouts $o^{(1)},\ldots,o^{(G)}\sim\pi_\theta(\cdot|q)$, with binary rewards $r_s\in\{0,1\}$ and computed advantages $A_s$, let $S^+=\{s:r_s=1\}$ and $p^+=|S^+|/G$. The vanilla generalized reinforce plus off-policy (GRPO) gradient is:
$$
\sum_{s=1}^G A_s \nabla_\theta \log \pi_\theta(o^{(s)}|q)
$$
IAC introduces a calibration strength parameter $\alpha\ge0$ and computes
$$
w(q) = (G - |S^+|)^\alpha = (1-p^+)^\alpha G^\alpha
$$
The calibrated advantage is
$$
\tilde{A}_s =
  \begin{cases}
    A_s \cdot w(q), & \text{if } s\in S^+ \\
    A_s,             & \text{otherwise}
  \end{cases}
$$
or equivalently per-sample, $w_s = (G-|S^+|)^\alpha$ if $r_s=1$, $w_s=1$ otherwise. This adjustment ensures that, when the query is nearly solved, the positive-advantage contributions are suppressed to offset the risk of global mode collapse.

## 3. Calibrated Policy-Gradient Update and Implementation

For a mini-batch of $B$ queries, each with $G$ rollouts, IAC is realized via the following steps:
- Identify the sets of successes $S^+_b$ and failures $S^-_b$ for each query $q_b$.
- Compute the empirical batch success rate $p^+_b$ and calibration weight $w_b=(G-|S^+_b|)^\alpha$.
- For each successful trajectory, scale its advantage: $\tilde{A}_{b,s}=A_{b,s} w_b$ if $s\in S^+_b$; otherwise, leave it unchanged.
- The aggregate policy-gradient becomes:
$$
\nabla_\theta J_\text{IAC} = \frac{1}{B G}\sum_{b=1}^B\sum_{s=1}^G \tilde{A}_{b,s}\nabla_\theta\log \pi_\theta(o_{b,s}|q_b) 
- \beta\nabla_\theta D_\text{KL}(\pi_\theta(\cdot|q_b)\|\pi_\text{ref}(\cdot|q_b))
$$
This update is accompanied by algorithmic pseudocode specifying batch processing, dynamic weighting, and KL regularization to the reference policy. An optional extension incorporates distribution-level calibration via a memory network $m_\phi$, which tracks and regularizes empirical sampling frequencies by modifying the policy logits before sampling.

## 4. Theoretical Guarantee Against Over-Sharpening

IAC is constructed such that, after each finite-batch update, all unsampled modes retain their original probability mass relative to the reference distribution. The lower-bound proof (Theorem 3) shows
$$
Z'_\text{IAC}(q) \ge 1 + \frac{1}{\beta G}\sum_s N_s \tilde{A}_s = 1
$$
which directly yields, for any unsampled mode $\lambda$ with $N_\lambda=0$, the update rule
$$
\pi_\text{new}(\lambda|q) = \pi_\text{ref}(\lambda|q)/Z'_\text{IAC}(q) = \pi_\text{ref}(\lambda|q)
$$
Consequently, no unsampled correct mode is ever suppressed, and the structural cause of over-sharpening is eliminated in the finite-batch regime. This property provides robustness, ensuring that correct behaviors are never inadvertently forgotten, independent of regret guarantees.

## 5. Empirical Validation Across Mathematical Reasoning Benchmarks

IAC has been empirically evaluated on six benchmarks: GSM8K, MATH500, LightEval, AIME24, Minerva, and O-Bench, using Qwen3 models at 4B, 8B, and 14B parameter scales. Key metrics include AVG@8 (average correct rate across eight samples per query) and PASS@8 (fraction of queries with at least one correct answer in eight tries). Across all benchmarks and model sizes, IAC consistently outperforms strong RLVR baselines such as GRPO, RLOO, Reinforce++, and DrGRPO. For example, Qwen3-14B achieves AVG@8 improving from 70.28% (best baseline) to 71.15% and PASS@8 from 77.03% to 78.81%. Gains are accentuated on more challenging tasks (AIME24, Minerva, O-Bench). Furthermore, IAC is complementary to DAPO entropy-safe clipping: on LightEval, DAPO+IAC achieves PASS@8 of 84.80%, surpassing DAPO alone at 76.62%, with further improvement (85.15%) upon introducing distribution-level calibration (memory network, $\mu=0.5$).

## 6. Calibration Strength, Entropy, and Semantic-Coupling Analyses

Experimental ablations demonstrate that calibration strength $\alpha$ critically tunes performance. $\alpha=0$ reduces to GRPO; values in $[0.5,1.5]$ yield steady improvements, peaking around $\alpha=1.0$. Excessive suppression beyond $\alpha\approx2$ degrades performance. Policy entropy analyses reveal that DAPO alone causes entropy collapse ($\approx 0.08$), while DAPO+IAC stabilizes entropy in the $[0.20,0.24]$ range, preventing premature determinization. Toy semantic-coupling simulations further validate that IAC (at $\alpha=1$) dramatically delays the collapse of semantically linked modes, and with memory-based distribution-level calibration, collapse is nearly arrested.

## 7. Significance and Generality

Inverse-success advantage calibration provides a principled, lightweight mechanism for mitigating the collapse of valid solution modes under RLVR finite-batch updates. It is agnostic to model architecture and baseline selection, easily integrated into existing RLVR pipelines, and does not require problem-specific tuning apart from the calibration strength $\alpha$. Empirical and theoretical analysis jointly substantiate its role in preserving solution diversity and improving final task performance, while being complementary to entropy-based and distribution-level regularization techniques [2601.15609].

Source: https://www.emergentmind.com/topics/inverse-success-advantage-calibration