CalibAdv: Calibrating Advantage in GRPO

• CalibAdv is an advantage calibration technique that refines reward signals in GRPO-based deep search training by addressing the mis-penalization of correct intermediate steps.
• It employs a silver-document proxy and soft penalization to rebalance negative versus positive advantages at both intermediate and final answer stages.
• Experimental studies show an 11.80% relative F1 improvement and enhanced stability across deep search benchmarks, preventing training collapse.

CalibAdv is an advantage calibration method for GRPO-based training of deep search agents. It is designed for a setting in which an agent conducts multi-turn interactions with a search engine, producing trajectories that include reasoning, search queries, retrieved documents, and a final answer. The method targets two coupled pathologies identified in deep search training: mis-penalization of correct intermediate steps and instability caused by dominance of negative advantages. In the formulation introduced in "Negative Advantage Is a Double-Edged Sword: Calibrating Advantage in GRPO for Deep Search" (Wu et al., 20 Apr 2026), CalibAdv modifies the advantage signal rather than replacing GRPO itself: it downscales excessive negative advantages on partially correct intermediate steps, rebalances positive and negative advantages at the answer step, and isolates the special > token from unstable advantage assignment.

1. Problem setting and motivation

CalibAdv is situated in deep search, where a policy interacts with a retriever over multiple turns and is trained with Group Relative Policy Optimization (GRPO) (Wu et al., 20 Apr 2026). In this regime, standard training uses only the final answer reward, together with a format reward, and propagates the resulting advantage over the whole rollout. The paper states that this effectively assigns the same sign and magnitude to all tokens or steps in the trajectory.

The reward definition used in the paper is answer F1 with a format gate:

$T_{\text{answer}}=\frac{2\,IN}{PN+RN}$

where $IN$ is the number of overlapping words between predicted and reference answers, $PN$ is predicted-answer length, and $RN$ is reference length. The format reward is

$$T_{\text{format}}= \begin{cases} 1, & \text{if format is correct}\ 0, & \text{otherwise} \end{cases}$$

and the final reward is

$$Y_{\text{final}} = T_{\text{answer}} \cdot T_{\text{format}}.$$

The central diagnosis is that this reward propagation is too coarse-grained for deep search. A rollout can contain correct intermediate searches and still receive a negative final reward if the final answer is wrong. The paper names this failure mode mis-penalization of correct steps. It further argues that, over training, negative advantages can dominate, and that this imbalance is associated with rising entropy, higher perplexity, garbled text, and ultimately catastrophic collapse into word-level repetition or unusable outputs (Wu et al., 20 Apr 2026).

2. Failure modes of standard GRPO in deep search

The paper attributes the observed instability to two structural properties of the GRPO signal. First, intermediate-step correctness and final-answer reward are substantially mismatched. Second, the optimization can drift into a regime in which negative advantages outweigh positive ones for extended periods (Wu et al., 20 Apr 2026).

The first issue is a credit-assignment problem. Because intermediate steps are not separately judged, a useful search action can be punished solely because the terminal answer is wrong. In deep search, this matters because trajectories are composite objects: retrieved evidence, query reformulations, and reasoning traces may be locally correct even when the end answer fails. A plausible implication is that standard rollout-level reward assignment suppresses useful retrieval behavior together with genuinely harmful behavior.

The second issue is a stability problem. The paper explicitly characterizes negative advantage as a double-edged sword. It is necessary for suppressing bad behavior and steering the agent away from incorrect search actions, but if it is applied too broadly or too strongly, especially to correct intermediate steps, it harms exploration and language generation quality. The reported training traces connect an increasing negative/positive advantage ratio to entropy spikes, falling token probabilities, rising perplexity, and collapse (Wu et al., 20 Apr 2026).

This framing is important because CalibAdv does not remove negative learning signals. Instead, it calibrates where and how strongly they are applied.

3. Core mechanism of CalibAdv

CalibAdv calibrates advantages in two places: intermediate steps and the final answer step (Wu et al., 20 Apr 2026).

For intermediate steps, the method constructs a silver-document proxy for step correctness. For a question $q$, let $R_{\text{correct}}$ be the set of correct rollouts, and let $D_r$ be the retrieved documents from rollout $r$. The silver document set is

$IN$0

For an intermediate step $IN$1, with retrieved documents $IN$2, the correctness score is

$IN$3

This score serves as a proxy for how much useful information the step retrieved.

The original step advantage $IN$4 is then calibrated by the soft penalization rule

$IN$5

If a step has negative advantage but overlaps strongly with the silver-document set, the penalty is reduced. If it is uninformative, the negative signal remains near full strength. The paper is explicit that this is attenuation, not removal, of negative advantage.

The second component rebalances the answer-step signal. For the final answer step in a group $IN$6, the positive and negative advantages are denoted $IN$7 and $IN$8. The method defines

$IN$9

and rebalances the final answer’s advantage via

$PN$0

where $PN$1 controls the relative influence of the rebalanced signal. The paper reports that $PN$2 gives the best tradeoff: values below $PN$3 lead to negative-advantage dominance and collapse, while values above $PN$4 overemphasize positive advantages and can reduce the reward ceiling (Wu et al., 20 Apr 2026).

A third practical component is the handling of the special <think> token. CalibAdv prepends it outside the rollout so that it does not absorb unstable advantage signals. The paper links this token-level decoupling to improved stability.

4. Integration into GRPO and algorithmic role

CalibAdv is not presented as a separate optimizer or a new RL objective. It is a modification to the advantage used by GRPO (Wu et al., 20 Apr 2026). Training proceeds with sampled rollouts as in standard GRPO, but the advantage signal is altered before the policy update.

The paper describes three modifications to the training pipeline. Intermediate-step advantages are replaced by the calibrated values from the soft penalization rule. The final answer step is rebalanced using the negative-to-positive ratio and the scaling coefficient $PN$5. The <think> token is handled outside the shared rollout prefix. In that sense, the GRPO objective remains intact in spirit, but the credit assignment fed into it is changed step by step.

This design places CalibAdv between pure process supervision and unmodified outcome supervision. It does not require extra annotations, external LLM labeling, or additional sampling overhead, yet it injects step-sensitive structure into the learning signal (Wu et al., 20 Apr 2026). The method therefore targets a specific weakness of rollout-level reinforcement learning in search-based QA: the mismatch between the granularity of behavior and the granularity of reward.

5. Experimental regime and empirical findings

The reported experiments train three backbone models as deep search agents: Qwen2.5-7B-Base, Qwen2.5-3B-Base, and Llama-3.2-3B-Instruct (Wu et al., 20 Apr 2026). The retrieval stack uses a Wikipedia dump from Dec. 20, 2018, E5-Base-V2 as the dense retriever, and ms-marco-MiniLM-L12-v2 as the reranker, with a top-20 retrieval stage followed by reranking to top-3. Training uses the training splits of HotpotQA and 2WikiMultiHopQA.

Evaluation is conducted on seven benchmarks: Natural Questions, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle. The first three are single-hop and the last four are multi-hop. The main metrics are F1 score for answer correctness and Collapse Point for stability, defined as the training step at which performance sharply declines. The paper also uses High PPL Ratio in ablations, defined as the proportion of outputs with perplexity greater than $PN$6, as a proxy for language degradation (Wu et al., 20 Apr 2026).

CalibAdv is compared against Search-R1, SimpleTIR, LLD, StepSearch, MT-GRPO, and GiGPO. The main result is that CalibAdv consistently improves both performance and stability. Relative to standard GRPO/Search-R1, it yields an 11.80% relative improvement in F1 on average and reaches the final training step without collapse on all reported models (Wu et al., 20 Apr 2026).

The headline average F1 values reported in the paper are:

• Qwen2.5-7B-Base: Search-R1 $PN$7, CalibAdv $PN$8
• Qwen2.5-3B-Base: Search-R1 $PN$9, CalibAdv $RN$0
• Llama-3.2-3B-Instruct: Search-R1 $RN$1, CalibAdv $RN$2

The paper also emphasizes that CalibAdv outperforms the process-reward baselines while avoiding catastrophic collapse.

6. Ablations, interpretation, and scope

The ablation study is organized around three progressively added components: PreThink, SoftPen, and AdvReb (Wu et al., 20 Apr 2026). PreThink prepends <think>. SoftPen applies soft penalization to intermediate steps. AdvReb performs final-step advantage rebalance.

The reported progression is structurally informative. PreThink alone improves stability and lets training finish an epoch, but language degradation remains. Adding SoftPen improves F1 and reduces the proportion of high-perplexity outputs, which the paper interprets as evidence that correcting misattributed penalties on intermediate steps matters. Adding AdvReb eliminates high-PPL outputs entirely and further improves F1, indicating that balancing positive and negative advantages is critical for long-horizon stability (Wu et al., 20 Apr 2026).

A common misconception would be to treat the method as an argument against negative advantage itself. The paper explicitly rejects that interpretation. Negative advantage is necessary, but dangerous when it is uncontrolled, coarse, and imbalanced. The contribution of CalibAdv is therefore not to neutralize punishment signals, but to calibrate them so that they better match the structure of deep search trajectories.

Within that framing, CalibAdv can be understood as a credit-assignment correction for GRPO in search-intensive QA. It fine-grains credit over intermediate search steps using a silver-document proxy, softens erroneous penalties on partially correct steps, rebalances the final answer step to prevent punishment-only dynamics, and stabilizes formatting by handling <think> separately (Wu et al., 20 Apr 2026). This suggests that its broader significance lies in showing that training instability in deep search is not only an optimization issue, but also a problem of how reward-derived advantages are distributed over multi-turn behavior.