Winner Advantage Policy Optimization (WAPO)

• The paper introduces a clipped policy-gradient update that uses only positive-advantage completions to mitigate collapse in RLVR settings.
• It employs group-relative advantages, token-level importance ratios, and asymmetric clipping to balance entropy and maintain stable training dynamics.
• Empirical results show that WAPO outperforms or matches baselines across multi-hop QA and mathematical reasoning benchmarks by adaptively scaling updates.

Winner Advantage Policy Optimization (WAPO) is a reinforcement-learning-with-verifiable-rewards (RLVR) method for reasoning-focused LLMs that was introduced as “a simple online clipped policy-gradient objective that updates only on positive-advantage completions” after a token-level analysis of GRPO-style instability (YSS et al., 15 Jun 2026). In the formulation proposed in “A Gradient Perspective on RLVR Stability and Winner Advantage Policy Optimization” (YSS et al., 15 Jun 2026), WAPO keeps the core GRPO machinery—group-relative advantages, importance ratios, clipping, and online rollouts—but masks out all non-positive-advantage completions. The resulting method is motivated by a gradient taxonomy in which stability depends jointly on the sign of the advantage and the sampled token’s probability under the current policy, and it is reported to improve training stability while matching or outperforming strong RLVR baselines across mathematical reasoning and multi-hop question answering benchmarks (YSS et al., 15 Jun 2026).

1. RLVR setting and the collapse problem

WAPO is defined in the RLVR setting in which a LLM policy $\pi_\theta$ generates a completion $y$ for a prompt $x$, and a verifier or rule-based checker computes a scalar reward $r(x,y)$ (YSS et al., 15 Jun 2026). The objective is

$$J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot\mid x)}[r(x,y)].$$

The paper studies two reasoning-oriented regimes. In mathematical reasoning, the model is instructed to produce

$$y = \texttt{<think>}~\text{reasoning}~\texttt{</think>}~\texttt{Answer: } \text{answer},$$

with binary reward

$$R_{\text{math}}(y,a) = I_{\text{fmt}}(y)\, I_{\text{ans}}(y,a).$$

In multi-hop QA, the model interacts with a search environment using >, <search>, <information>, and <answer>, with reward

$$R_{\text{QA}}(y,a) = \frac{0.3\, R_{\text{fmt}}(y) + R_{\text{F1}}(y,a)}{1.3}.$$

The immediate background is GRPO-style optimization. For each prompt, the rollout policy $\pi_{\theta_{\text{old}}}$ generates a group of completions $\{y_i\}_{i=1}^G$, rewards $y$0 are converted into group-relative advantages, typically

$y$1

and the current policy is updated through token-level importance ratios

$y$2

The WAPO paper characterizes GRPO-style RLVR as “fragile and prone to collapse” and distinguishes two collapse modes: high-entropy collapse, in which outputs become long, diverse, and task-irrelevant, and low-entropy collapse, in which outputs become repetitive, short, or templated (YSS et al., 15 Jun 2026). Prior explanations emphasized policy mismatch and weak trust-region control, but the paper’s central claim is that instability can also be understood directly through token-level gradient dynamics. That framing is what leads to WAPO.

2. Token-level gradient taxonomy

The analysis starts from a single next-token distribution with logits $y$3, softmax $y$4, sampled token $y$5 with probability $y$6, and a sequence-level advantage $y$7 (YSS et al., 15 Jun 2026). The paper studies gradient descent on the advantage-weighted negative log-likelihood

$y$8

For a small step size $y$9, the first-order change in the probability of a non-sampled token $x$0 is

$x$1

where

$x$2

This yields a threshold-based distinction between peak and valley tokens:

$x$3

Combined with the sign of $x$4, the paper defines four regimes:

1. Pos-peak: $x$5, $x$6
2. Pos-valley: $x$7, $x$8
3. Neg-peak: $x$9, $r(x,y)$0
4. Neg-valley: $r(x,y)$1, $r(x,y)$2

The entropy analysis is equally central. With

$r(x,y)$3

the first-order entropy change is

$r(x,y)$4

The paper reports that for valley tokens the bracketed term is negative, implying that Pos-valley increases entropy and Neg-valley decreases entropy. For peak tokens, especially max-probability tokens, the bracketed term tends to be positive, implying that Pos-peak decreases entropy and Neg-peak increases entropy (YSS et al., 15 Jun 2026).

This leads to the empirical taxonomy that motivates WAPO. The paper reports that Pos-peak is stable and reduces entropy but plateaus in reward; Pos-valley and Neg-peak increase entropy and drive the model toward high-entropy collapse; Neg-valley is initially useful but later tends toward low-entropy collapse (YSS et al., 15 Jun 2026). Masking by only sign or only peak/valley produces further structure: Pos-only behaves comparably to DAPO, Neg-only and Peak-only tend to collapse, and Valley-only can be stable but is under-explored and under-sampled. WAPO is the policy-level consequence of this analysis: keep positive-advantage updates and remove negative-advantage updates.

3. Formal objective and algorithmic structure

WAPO uses the same group-relative advantage as standard GRPO and then applies a positive-part mask. Given rollout advantages $r(x,y)$5, it defines

$r(x,y)$6

“Updates only on positive-advantage completions” means that only terms with $r(x,y)$7 contribute to the policy-gradient objective; completions with $r(x,y)$8 are masked out (YSS et al., 15 Jun 2026).

The WAPO objective is

$r(x,y)$9

Several implementation choices are specific. First, clipping is only on the upper side, at $$J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot\mid x)}[r(x,y)].$$0, because $$J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot\mid x)}[r(x,y)].$$1 makes the lower clipping bound irrelevant. Second, normalization is over group size $$J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot\mid x)}[r(x,y)].$$2 and a fixed maximum sequence length $$J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot\mid x)}[r(x,y)].$$3, not per-sequence length $$J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot\mid x)}[r(x,y)].$$4; the paper states that this avoids length bias observed in Dr.GRPO and PSR/RAFT++ (YSS et al., 15 Jun 2026). Third, groups with no positive-advantage completions are dropped from the policy-gradient term.

The relationship to GRPO is explicit. Standard GRPO uses

$$J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot\mid x)}[r(x,y)].$$5

WAPO differs in three ways: it uses $$J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot\mid x)}[r(x,y)].$$6 rather than $$J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot\mid x)}[r(x,y)].$$7, it removes the lower clipping bound, and it normalizes with a fixed $$J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot\mid x)}[r(x,y)].$$8 rather than $$J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot\mid x)}[r(x,y)].$$9 (YSS et al., 15 Jun 2026). The paper summarizes this succinctly as “GRPO with a ReLU on the advantage and slightly asymmetric clipping.”

The training procedure otherwise remains GRPO-like. The paper describes online RLVR with LoRA adapters of rank $$y = \texttt{<think>}~\text{reasoning}~\texttt{</think>}~\texttt{Answer: } \text{answer},$$0 and alpha $$y = \texttt{<think>}~\text{reasoning}~\texttt{</think>}~\texttt{Answer: } \text{answer},$$1, stepwise batches of $$y = \texttt{<think>}~\text{reasoning}~\texttt{</think>}~\texttt{Answer: } \text{answer},$$2 rollouts, group size $$y = \texttt{<think>}~\text{reasoning}~\texttt{</think>}~\texttt{Answer: } \text{answer},$$3 per prompt, micro-batch size $$y = \texttt{<think>}~\text{reasoning}~\texttt{</think>}~\texttt{Answer: } \text{answer},$$4, learning rate $$y = \texttt{<think>}~\text{reasoning}~\texttt{</think>}~\texttt{Answer: } \text{answer},$$5, and maximum sequence length $$y = \texttt{<think>}~\text{reasoning}~\texttt{</think>}~\texttt{Answer: } \text{answer},$$6 (YSS et al., 15 Jun 2026). No explicit KL regularization to a reference model is highlighted; stability is controlled mainly through clipping, group-relative normalization, and positive-only masking.

4. Binary-reward analysis and success-probability scaling

Beyond the local token taxonomy, the WAPO paper gives a binary-reward analysis for a fixed prompt $$y = \texttt{<think>}~\text{reasoning}~\texttt{</think>}~\texttt{Answer: } \text{answer},$$7 with reward $$y = \texttt{<think>}~\text{reasoning}~\texttt{</think>}~\texttt{Answer: } \text{answer},$$8 and success probability

$$y = \texttt{<think>}~\text{reasoning}~\texttt{</think>}~\texttt{Answer: } \text{answer},$$9

In this setting, the objective is

$$R_{\text{math}}(y,a) = I_{\text{fmt}}(y)\, I_{\text{ans}}(y,a).$$0

The paper states that the true policy gradient can be approximated by

$$R_{\text{math}}(y,a) = I_{\text{fmt}}(y)\, I_{\text{ans}}(y,a).$$1

where $$R_{\text{math}}(y,a) = I_{\text{fmt}}(y)\, I_{\text{ans}}(y,a).$$2.

Now define group-centered positive-only advantage

$$R_{\text{math}}(y,a) = I_{\text{fmt}}(y)\, I_{\text{ans}}(y,a).$$3

with $$R_{\text{math}}(y,a) = I_{\text{fmt}}(y)\, I_{\text{ans}}(y,a).$$4. Using $$R_{\text{math}}(y,a) = I_{\text{fmt}}(y)\, I_{\text{ans}}(y,a).$$5, the paper derives

$$R_{\text{math}}(y,a) = I_{\text{fmt}}(y)\, I_{\text{ans}}(y,a).$$6

This is one of the paper’s strongest theoretical claims: the WAPO update is aligned with the true gradient of success probability and is scaled by the adaptive factor $$R_{\text{math}}(y,a) = I_{\text{fmt}}(y)\, I_{\text{ans}}(y,a).$$7 (YSS et al., 15 Jun 2026).

The interpretation is direct. When the model rarely succeeds on a prompt, $$R_{\text{math}}(y,a) = I_{\text{fmt}}(y)\, I_{\text{ans}}(y,a).$$8 is small and the multiplicative factor is close to $$R_{\text{math}}(y,a) = I_{\text{fmt}}(y)\, I_{\text{ans}}(y,a).$$9, so the update remains strong. As the model approaches near-perfect success on that prompt, $$R_{\text{QA}}(y,a) = \frac{0.3\, R_{\text{fmt}}(y) + R_{\text{F1}}(y,a)}{1.3}.$$0 and the factor approaches $$R_{\text{QA}}(y,a) = \frac{0.3\, R_{\text{fmt}}(y) + R_{\text{F1}}(y,a)}{1.3}.$$1, so the update saturates. The paper also considers an alternative normalization yielding $$R_{\text{QA}}(y,a) = \frac{0.3\, R_{\text{fmt}}(y) + R_{\text{F1}}(y,a)}{1.3}.$$2, which would upweight very hard prompts more aggressively, but adopts the bounded $$R_{\text{QA}}(y,a) = \frac{0.3\, R_{\text{fmt}}(y) + R_{\text{F1}}(y,a)}{1.3}.$$3 factor as the default because it is more conservative under noisy rare successes (YSS et al., 15 Jun 2026).

This adaptive behavior is closely tied to the method’s “winner” interpretation. Positive-advantage completions function as winners within each rollout group, but the strength of winner reinforcement is not constant across prompts. It decreases automatically as a prompt becomes easy and remains large when success is rare.

5. Empirical behavior across benchmarks and baselines

The WAPO paper compares against GRPO, DAPO, GSPO, PSR, and RAFT++ across four benchmarks and three model families: Qwen3-4B, SmolLM3-3B, and Gemma3-4B (YSS et al., 15 Jun 2026). The reported headline result is a better stability–performance tradeoff. WAPO is described as stable in all tested scenarios, whereas DAPO exhibits multiple collapses; the paper gives the specific example that on OTT-QA with Qwen3-4B, DAPO collapses within $$R_{\text{QA}}(y,a) = \frac{0.3\, R_{\text{fmt}}(y) + R_{\text{F1}}(y,a)}{1.3}.$$4 steps, and it also notes that GRPO collapses on NuminaMath-LEAN with Gemma3-4B (YSS et al., 15 Jun 2026).

On harder multi-hop QA tasks, the gains are substantial. For OTT-QA EM (average@32 best checkpoint), the paper reports:

• Qwen3-4B: WAPO $$R_{\text{QA}}(y,a) = \frac{0.3\, R_{\text{fmt}}(y) + R_{\text{F1}}(y,a)}{1.3}.$$5 vs. next-best GSPO $$R_{\text{QA}}(y,a) = \frac{0.3\, R_{\text{fmt}}(y) + R_{\text{F1}}(y,a)}{1.3}.$$6 $$R_{\text{QA}}(y,a) = \frac{0.3\, R_{\text{fmt}}(y) + R_{\text{F1}}(y,a)}{1.3}.$$7
• Gemma3-4B: WAPO $$R_{\text{QA}}(y,a) = \frac{0.3\, R_{\text{fmt}}(y) + R_{\text{F1}}(y,a)}{1.3}.$$8 vs. next-best GSPO $$R_{\text{QA}}(y,a) = \frac{0.3\, R_{\text{fmt}}(y) + R_{\text{F1}}(y,a)}{1.3}.$$9 $\pi_{\theta_{\text{old}}}$0

For Hotpot-QA EM, it reports:

• Qwen3-4B: WAPO $\pi_{\theta_{\text{old}}}$1 vs. GSPO $\pi_{\theta_{\text{old}}}$2 $\pi_{\theta_{\text{old}}}$3
• Gemma3-4B: WAPO $\pi_{\theta_{\text{old}}}$4 vs. GSPO $\pi_{\theta_{\text{old}}}$5 $\pi_{\theta_{\text{old}}}$6

The evaluation curves are said to show that DAPO often collapses, GRPO and GSPO saturate earlier, and WAPO continues to improve over training steps with smooth reward trajectories (YSS et al., 15 Jun 2026).

On mathematical reasoning, the picture is more conservative. WAPO is reported as “generally competitive, occasionally slightly below the best GRPO/GSPO,” with Math-500 within approximately $\pi_{\theta_{\text{old}}}$7–$\pi_{\theta_{\text{old}}}$8 of the top baseline and NuminaMath-LEAN initially lagging but closing much of the gap later (YSS et al., 15 Jun 2026). The paper’s interpretation is that the $\pi_{\theta_{\text{old}}}$9 scaling factor makes WAPO more conservative early on, especially when many prompts are easy and quickly saturated.

A common concern with positive-only updates is loss of exploration. The paper addresses this with pass@k results. For multi-hop QA, WAPO is reported to achieve the best pass@1 and the best or tied-best pass@k across $\{y_i\}_{i=1}^G$0. For math, pass@k is described as competitive; the paper gives the example that on NuminaMath-LEAN with SmolLM3-3B, WAPO pass@16 $\{y_i\}_{i=1}^G$1, comparable to GSPO’s $\{y_i\}_{i=1}^G$2 (YSS et al., 15 Jun 2026). The stated explanation is that WAPO preserves Pos-valley updates, which are entropy-increasing but tied to verified good rollouts rather than arbitrary drift.

Out-of-domain results are also included. The paper reports that on Hotpot-QA $\{y_i\}_{i=1}^G$3 2-wiki, WAPO is best across all three model families, while on NuminaMath-LEAN $\{y_i\}_{i=1}^G$4 AIME’25, WAPO is competitive and at higher $\{y_i\}_{i=1}^G$5 catches up or is comparable (YSS et al., 15 Jun 2026).

6. Relation to adjacent advantage designs, misconceptions, and open problems

WAPO belongs to a broader family of work on how advantage design affects RL for LLMs, but several nearby methods operate in meaningfully different regimes. “Mixed Advantage Policy Optimization” studies two GRPO pathologies—advantage reversion and advantage mirror—and proposes a certainty-dependent mixture of deviation-based and mean-based advantages (Huang et al., 23 Sep 2025). That paper does not define WAPO, but it explicitly notes that “a winner-based method that normalizes by variance or uses z-scores around the winner could suffer the same phenomenon” (Huang et al., 23 Sep 2025). This suggests that not all winner-oriented objectives are equivalent: WAPO’s sign masking addresses instability caused by negative-advantage regimes, whereas MAPO addresses instability and misallocation caused by the shape of the advantage function itself.

A second neighboring line is pairwise preference optimization. “AdaDPO” derives per-preference-pair coefficients $\{y_i\}_{i=1}^G$6 and $\{y_i\}_{i=1}^G$7 that equalize winner and loser gradient magnitudes in DPO-style losses and shows that the same mechanism can be biased with a factor $\{y_i\}_{i=1}^G$8 to favor winners more strongly (Chen et al., 27 May 2026). This is not RLVR and not WAPO, but it formalizes a distinct meaning of “winner advantage”: direct control of preferred-versus-dispreferred gradient magnitudes in probability space. The contrast is useful. WAPO retains group-relative advantages and online policy gradients; AdaDPO modifies pairwise contrastive losses.

A third neighboring method is “Decoupled Advantage Policy Optimization,” which targets overthinking in large reasoning models by attenuating token-level advantages on inefficient reasoning segments inside correct trajectories (Tan et al., 17 Oct 2025). That paper explicitly states that it “does not introduce or name anything as ‘Winner Advantage Policy Optimization (WAPO)’” (Tan et al., 17 Oct 2025). The relationship is conceptual rather than terminological: DEPO refines token-level credit assignment within winners, whereas WAPO makes a coarser sign-based decision about which trajectories should update the policy at all.

These distinctions address a common misconception: WAPO is not simply any winner-only or winner-biased update rule. In the paper’s own positioning, WAPO differs from PSR and RAFT++ precisely because it keeps GRPO-style group-normalized advantages and importance-ratio clipping rather than reinforcing successes without group centering or with sequence-level normalization that induces short-answer bias (YSS et al., 15 Jun 2026).

The WAPO paper also states several limitations and open questions. It assumes that positive-advantage completions are sufficiently informative; under very coarse or noisy rewards, useful information in negative rollouts may be lost. The token taxonomy shows that negative-advantage tokens can be harmful on average, but the paper notes that some may carry useful gradient signal and that selecting them safely is non-trivial. Experiments are reported at approximately $\{y_i\}_{i=1}^G$9–$y$00B scale, so larger models and MoE architectures remain unstudied. Interactions with explicit KL constraints and other off-policy corrections are also identified as not yet fully analyzed. Proposed extensions include alternative positive-only normalizations such as $y$01, refined selection of negative-advantage tokens, combining WAPO with advanced trust-region mechanisms, and applying it beyond math and QA to code generation, text-to-SQL, and other verifiable tasks (YSS et al., 15 Jun 2026).

In that sense, WAPO is best understood as a specific answer to a specific instability diagnosis. It is a GRPO-family RLVR method in which winners are defined by positive group-relative advantage, negative-advantage trajectories are removed from the policy-gradient update, and the resulting update is justified both by token-level entropy dynamics and by a binary-reward derivation aligned with the gradient of success probability (YSS et al., 15 Jun 2026).