---
title: Winner Advantage Policy Optimization (WAPO)
url: https://www.emergentmind.com/topics/winner-advantage-policy-optimization-wapo
type: topic
---

# Winner Advantage Policy Optimization (WAPO)

Winner Advantage Policy Optimization (WAPO) is a reinforcement-learning-with-verifiable-rewards (RLVR) method for reasoning-focused language models that was introduced as “a simple online clipped policy-gradient objective that updates only on positive-advantage completions” after a token-level analysis of GRPO-style instability [2606.16154]. In the formulation proposed in “A Gradient Perspective on RLVR Stability and Winner Advantage Policy Optimization” [2606.16154], WAPO keeps the core GRPO machinery—group-relative advantages, importance ratios, clipping, and online rollouts—but masks out all non-positive-advantage completions. The resulting method is motivated by a gradient taxonomy in which stability depends jointly on the sign of the advantage and the sampled token’s probability under the current policy, and it is reported to improve training stability while matching or outperforming strong RLVR baselines across mathematical reasoning and multi-hop question answering benchmarks [2606.16154].

## 1. RLVR setting and the collapse problem

WAPO is defined in the RLVR setting in which a language model policy $\pi_\theta$ generates a completion $y$ for a prompt $x$, and a verifier or rule-based checker computes a scalar reward $r(x,y)$ [2606.16154]. The objective is
$$
J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot\mid x)}[r(x,y)].
$$
The paper studies two reasoning-oriented regimes. In mathematical reasoning, the model is instructed to produce
$$
y = \texttt{<think>}~\text{reasoning}~\texttt{</think>}~\texttt{Answer: } \text{answer},
$$
with binary reward
$$
R_{\text{math}}(y,a) = I_{\text{fmt}}(y)\, I_{\text{ans}}(y,a).
$$
In multi-hop QA, the model interacts with a search environment using `<think>`, `<search>`, `<information>`, and `<answer>`, with reward
$$
R_{\text{QA}}(y,a) = \frac{0.3\, R_{\text{fmt}}(y) + R_{\text{F1}}(y,a)}{1.3}.
$$

The immediate background is GRPO-style optimization. For each prompt, the rollout policy $\pi_{\theta_{\text{old}}}$ generates a group of completions $\{y_i\}_{i=1}^G$, rewards $\{r_i\}$ are converted into group-relative advantages, typically
$$
A_i = \frac{r_i - \bar r}{\operatorname{std}(\{r_j\}) + \epsilon},\quad
\bar r = \tfrac{1}{G}\sum_j r_j,
$$
and the current policy is updated through token-level importance ratios
$$
\rho_{ij}(\theta) = \frac{\pi_\theta(y_{ij}\mid x,y_{i,<j})}{\pi_{\theta_{\text{old}}}(y_{ij}\mid x,y_{i,<j})}.
$$

The WAPO paper characterizes GRPO-style RLVR as “fragile and prone to collapse” and distinguishes two collapse modes: **high-entropy collapse**, in which outputs become long, diverse, and task-irrelevant, and **low-entropy collapse**, in which outputs become repetitive, short, or templated [2606.16154]. Prior explanations emphasized policy mismatch and weak trust-region control, but the paper’s central claim is that instability can also be understood directly through token-level gradient dynamics. That framing is what leads to WAPO.

## 2. Token-level gradient taxonomy

The analysis starts from a single next-token distribution with logits $z \in \mathbb{R}^V$, softmax $p = \text{softmax}(z)$, sampled token $s$ with probability $p_s$, and a sequence-level advantage $A \in \mathbb{R}$ [2606.16154]. The paper studies gradient descent on the advantage-weighted negative log-likelihood
$$
\ell_s(z) = -A \log p_s.
$$
For a small step size $\eta>0$, the first-order change in the probability of a non-sampled token $i\neq s$ is
$$
\Delta p_i = \eta A p_i\bigl( C(p) - p_s - p_i \bigr) + O(\eta^2),
$$
where
$$
C(p)=\sum_{j=1}^V p_j^2.
$$
This yields a threshold-based distinction between **peak** and **valley** tokens:
$$
\text{Peak: } p_s \ge C(p), \qquad \text{Valley: } p_s < C(p).
$$
Combined with the sign of $A$, the paper defines four regimes:

1. **Pos-peak**: $A>0$, $p_s \ge C(p)$  
2. **Pos-valley**: $A>0$, $p_s < C(p)$  
3. **Neg-peak**: $A<0$, $p_s \ge C(p)$  
4. **Neg-valley**: $A<0$, $p_s < C(p)$

The entropy analysis is equally central. With
$$
H(p) = -\sum_i p_i \log p_i,
$$
the first-order entropy change is
$$
\Delta H = -\eta A \Big[ \big(p_s\log p_s-\sum_i p_i^2\log p_i\big) + H(p)\big(p_s-C(p)\big) \Big] +O(\eta^2).
$$
The paper reports that for valley tokens the bracketed term is negative, implying that **Pos-valley increases entropy** and **Neg-valley decreases entropy**. For peak tokens, especially max-probability tokens, the bracketed term tends to be positive, implying that **Pos-peak decreases entropy** and **Neg-peak increases entropy** [2606.16154].

This leads to the empirical taxonomy that motivates WAPO. The paper reports that **Pos-peak** is stable and reduces entropy but plateaus in reward; **Pos-valley** and **Neg-peak** increase entropy and drive the model toward high-entropy collapse; **Neg-valley** is initially useful but later tends toward low-entropy collapse [2606.16154]. Masking by only sign or only peak/valley produces further structure: **Pos-only** behaves comparably to DAPO, **Neg-only** and **Peak-only** tend to collapse, and **Valley-only** can be stable but is under-explored and under-sampled. WAPO is the policy-level consequence of this analysis: keep positive-advantage updates and remove negative-advantage updates.

## 3. Formal objective and algorithmic structure

WAPO uses the same group-relative advantage as standard GRPO and then applies a positive-part mask. Given rollout advantages $A_i$, it defines
$$
A_i^+ = \max(A_i, 0).
$$
“Updates only on positive-advantage completions” means that only terms with $A_i^+>0$ contribute to the policy-gradient objective; completions with $A_i \le 0$ are masked out [2606.16154].

The WAPO objective is
$$
\mathcal{J}_{\mathrm{WAPO}}(\theta) =
\mathbb{E}_{x\sim\mathcal{D}} \Bigg[
\frac{1}{GT} \sum_{i=1}^{G} \sum_{j=1}^{T_i}
A_i^+\cdot \min\bigl(\rho_{ij}(\theta),\,1{+}\epsilon\bigr)
\;\Big|\; \exists\,i \text{ s.t. } A_i^+>0
\Bigg].
$$
Several implementation choices are specific. First, clipping is only on the upper side, at $1+\epsilon$, because $A_i^+\ge 0$ makes the lower clipping bound irrelevant. Second, normalization is over group size $G$ and a fixed maximum sequence length $T$, not per-sequence length $T_i$; the paper states that this avoids length bias observed in Dr.GRPO and PSR/RAFT++ [2606.16154]. Third, groups with no positive-advantage completions are dropped from the policy-gradient term.

The relationship to GRPO is explicit. Standard GRPO uses
$$
\mathcal{L}_{\text{GRPO}}(\theta) = - \frac{1}{G} \sum_{i=1}^G \frac{1}{T_i} \sum_{j=1}^{T_i}
 \min\bigl( \rho_{ij}(\theta) A_i,\; \operatorname{clip}(\rho_{ij}(\theta), 1-\epsilon, 1+\epsilon)\, A_i \bigr).
$$
WAPO differs in three ways: it uses $A_i^+$ rather than $A_i$, it removes the lower clipping bound, and it normalizes with a fixed $T$ rather than $T_i$ [2606.16154]. The paper summarizes this succinctly as “GRPO with a ReLU on the advantage and slightly asymmetric clipping.”

The training procedure otherwise remains GRPO-like. The paper describes online RLVR with LoRA adapters of rank $8$ and alpha $32$, stepwise batches of $256$ rollouts, group size $G=8$ per prompt, micro-batch size $1$, learning rate $1 \times 10^{-5}$, and maximum sequence length $T = 8192$ [2606.16154]. No explicit KL regularization to a reference model is highlighted; stability is controlled mainly through clipping, group-relative normalization, and positive-only masking.

## 4. Binary-reward analysis and success-probability scaling

Beyond the local token taxonomy, the WAPO paper gives a binary-reward analysis for a fixed prompt $x$ with reward $r(x,y)\in\{0,1\}$ and success probability
$$
q_x = \Pr_{y\sim\pi_\theta(\cdot|x)}[r(x,y) = 1].
$$
In this setting, the objective is
$$
J_x(\theta) = \mathbb{E}[r(x,y)] = q_x.
$$
The paper states that the true policy gradient can be approximated by
$$
\nabla q_x \approx \frac{1}{G} \sum_{i=1}^{G} \sum_{j=1}^{T_i} r_i\nabla\log p_{ij},
$$
where $p_{ij}=\pi_\theta(y_{ij}\mid x,y_{i,<j})$.

Now define group-centered positive-only advantage
$$
A_i^+ = (r_i - \bar r)\mathbbm{1}[r_i = 1] = (1 - \bar r) r_i,
$$
with $\bar r = \tfrac{1}{G}\sum_i r_i$. Using $\bar r \approx q_x$, the paper derives
$$
\frac{1}{G} \sum_{i,j} A_i^+ \nabla \log p_{ij} \approx (1-q_x)\,\nabla q_x.
$$
This is one of the paper’s strongest theoretical claims: the WAPO update is aligned with the true gradient of success probability and is scaled by the adaptive factor $1-q_x$ [2606.16154].

The interpretation is direct. When the model rarely succeeds on a prompt, $q_x$ is small and the multiplicative factor is close to $1$, so the update remains strong. As the model approaches near-perfect success on that prompt, $q_x \to 1$ and the factor approaches $0$, so the update saturates. The paper also considers an alternative normalization yielding $(1-q_x)/q_x$, which would upweight very hard prompts more aggressively, but adopts the bounded $1-q_x$ factor as the default because it is more conservative under noisy rare successes [2606.16154].

This adaptive behavior is closely tied to the method’s “winner” interpretation. Positive-advantage completions function as winners within each rollout group, but the strength of winner reinforcement is not constant across prompts. It decreases automatically as a prompt becomes easy and remains large when success is rare.

## 5. Empirical behavior across benchmarks and baselines

The WAPO paper compares against GRPO, DAPO, GSPO, PSR, and RAFT++ across four benchmarks and three model families: Qwen3-4B, SmolLM3-3B, and Gemma3-4B [2606.16154]. The reported headline result is a better stability–performance tradeoff. WAPO is described as stable in all tested scenarios, whereas DAPO exhibits multiple collapses; the paper gives the specific example that on OTT-QA with Qwen3-4B, DAPO collapses within $100$ steps, and it also notes that GRPO collapses on NuminaMath-LEAN with Gemma3-4B [2606.16154].

On harder multi-hop QA tasks, the gains are substantial. For OTT-QA EM (average@32 best checkpoint), the paper reports:

- **Qwen3-4B**: WAPO $37.44$ vs. next-best GSPO $27.57$ $(+9.9)$  
- **Gemma3-4B**: WAPO $13.67$ vs. next-best GSPO $10.50$ $(+3.2)$  

For Hotpot-QA EM, it reports:

- **Qwen3-4B**: WAPO $45.84$ vs. GSPO $41.30$ $(+4.5)$  
- **Gemma3-4B**: WAPO $34.91$ vs. GSPO $24.36$ $(+10.6)$  

The evaluation curves are said to show that DAPO often collapses, GRPO and GSPO saturate earlier, and WAPO continues to improve over training steps with smooth reward trajectories [2606.16154].

On mathematical reasoning, the picture is more conservative. WAPO is reported as “generally competitive, occasionally slightly below the best GRPO/GSPO,” with Math-500 within approximately $1$–$2\%$ of the top baseline and NuminaMath-LEAN initially lagging but closing much of the gap later [2606.16154]. The paper’s interpretation is that the $1-q_x$ scaling factor makes WAPO more conservative early on, especially when many prompts are easy and quickly saturated.

A common concern with positive-only updates is loss of exploration. The paper addresses this with pass@k results. For multi-hop QA, WAPO is reported to achieve the best pass@1 and the best or tied-best pass@k across $k=1..32$. For math, pass@k is described as competitive; the paper gives the example that on NuminaMath-LEAN with SmolLM3-3B, WAPO pass@16 $= 81.02\%$, comparable to GSPO’s $81.0\%$ [2606.16154]. The stated explanation is that WAPO preserves Pos-valley updates, which are entropy-increasing but tied to verified good rollouts rather than arbitrary drift.

Out-of-domain results are also included. The paper reports that on **Hotpot-QA $\to$ 2-wiki**, WAPO is best across all three model families, while on **NuminaMath-LEAN $\to$ AIME’25**, WAPO is competitive and at higher $k$ catches up or is comparable [2606.16154].

## 6. Relation to adjacent advantage designs, misconceptions, and open problems

WAPO belongs to a broader family of work on how advantage design affects RL for language models, but several nearby methods operate in meaningfully different regimes. “Mixed Advantage Policy Optimization” studies two GRPO pathologies—**advantage reversion** and **advantage mirror**—and proposes a certainty-dependent mixture of deviation-based and mean-based advantages [2509.18849]. That paper does not define WAPO, but it explicitly notes that “a winner-based method that normalizes by variance or uses z-scores around the winner could suffer the same phenomenon” [2509.18849]. This suggests that not all winner-oriented objectives are equivalent: WAPO’s sign masking addresses instability caused by negative-advantage regimes, whereas MAPO addresses instability and misallocation caused by the shape of the advantage function itself.

A second neighboring line is pairwise preference optimization. “AdaDPO” derives per-preference-pair coefficients $\beta_w$ and $\beta_l$ that equalize winner and loser gradient magnitudes in DPO-style losses and shows that the same mechanism can be biased with a factor $k$ to favor winners more strongly [2605.28440]. This is not RLVR and not WAPO, but it formalizes a distinct meaning of “winner advantage”: direct control of preferred-versus-dispreferred gradient magnitudes in probability space. The contrast is useful. WAPO retains group-relative advantages and online policy gradients; AdaDPO modifies pairwise contrastive losses.

A third neighboring method is “Decoupled Advantage Policy Optimization,” which targets overthinking in large reasoning models by attenuating token-level advantages on inefficient reasoning segments inside correct trajectories [2510.15374]. That paper explicitly states that it “does not introduce or name anything as ‘Winner Advantage Policy Optimization (WAPO)’” [2510.15374]. The relationship is conceptual rather than terminological: DEPO refines token-level credit assignment within winners, whereas WAPO makes a coarser sign-based decision about which trajectories should update the policy at all.

These distinctions address a common misconception: WAPO is not simply any winner-only or winner-biased update rule. In the paper’s own positioning, WAPO differs from PSR and RAFT++ precisely because it keeps GRPO-style group-normalized advantages and importance-ratio clipping rather than reinforcing successes without group centering or with sequence-level normalization that induces short-answer bias [2606.16154].

The WAPO paper also states several limitations and open questions. It assumes that positive-advantage completions are sufficiently informative; under very coarse or noisy rewards, useful information in negative rollouts may be lost. The token taxonomy shows that negative-advantage tokens can be harmful on average, but the paper notes that some may carry useful gradient signal and that selecting them safely is non-trivial. Experiments are reported at approximately $3$–$4$B scale, so larger models and MoE architectures remain unstudied. Interactions with explicit KL constraints and other off-policy corrections are also identified as not yet fully analyzed. Proposed extensions include alternative positive-only normalizations such as $(1-q_x)/q_x$, refined selection of negative-advantage tokens, combining WAPO with advanced trust-region mechanisms, and applying it beyond math and QA to code generation, text-to-SQL, and other verifiable tasks [2606.16154].

In that sense, WAPO is best understood as a specific answer to a specific instability diagnosis. It is a GRPO-family RLVR method in which winners are defined by positive group-relative advantage, negative-advantage trajectories are removed from the policy-gradient update, and the resulting update is justified both by token-level entropy dynamics and by a binary-reward derivation aligned with the gradient of success probability [2606.16154].

Source: https://www.emergentmind.com/topics/winner-advantage-policy-optimization-wapo