---
title: 'ASymPO: Asymmetric-Scale Policy Optimization'
url: https://www.emergentmind.com/topics/asymmetric-scale-policy-optimization-asympo
type: topic
---

# ASymPO: Asymmetric-Scale Policy Optimization

Searching arXiv for the cited ASymPO paper and closely related asymmetric policy optimization work to ground the article in current literature.
Asymmetric-Scale Policy Optimization (ASymPO) is a current-policy-only objective for asynchronous large language model post-training that addresses a specific instability arising from stale rollouts: positive- and negative-advantage responses can be evaluated under the current policy at different negative log-probability scales, so zero-sum group-relative advantages no longer imply balanced loss contributions. The method normalizes each response’s token loss by that response’s current average token negative log-probability, requires no behavior-policy probabilities, restores exact response-level zero-sum balance in the forward loss, and preserves a nonzero learning signal in the backward pass [2606.03070].

## 1. Problem setting and motivation

ASymPO is formulated for **asynchronous group-relative reinforcement learning** in LLM post-training. In this setting, rollout workers generate responses from a stale behavior policy $\pi_b$, the learner updates a newer policy $\pi_\theta$, and responses are optimized later, after $\pi_\theta$ may have drifted from $\pi_b$. The operational benefit is higher throughput through decoupling response generation from policy optimization. The cost is **distribution drift**, because the sampled data and the optimizing policy are no longer synchronized [2606.03070].

Standard behavior-corrected methods such as PPO and GRPO typically control this drift with behavior-policy probabilities, importance ratios, or clipping. The paper emphasizes that this requires token-aligned behavior log-probabilities, policy version bookkeeping, and recomputation at training precision to avoid numerical mismatch. ASymPO is motivated by the question of whether asynchronous group-relative RL can instead be stabilized using only current-policy probabilities, thereby removing the behavior-information interface between rollout and learner systems [2606.03070].

A central design point is that ASymPO is **not** presented as a general solution to all off-policy effects. The paper’s target is narrower: it isolates a failure mode in current-policy-only asynchronous training and proposes a normalization that fixes that specific imbalance without recovering behavior-policy probabilities or importance weights [2606.03070].

## 2. Scale imbalance under stale rollouts

The paper defines group-relative advantages for a prompt $x$ using a group of $G$ sampled responses with rewards $r_g$:
$$
A_g = r_g - \hat r, \qquad \hat r = \frac{1}{G}\sum_{j=1}^G r_j,
$$
so that
$$
\sum_{g=1}^G A_g = 0.
$$
This zero-sum structure is intended to stabilize learning by making positive and negative advantages cancel in aggregate [2606.03070].

For a policy $\pi_\mu$, the response-level average token negative log-probability is defined as
$$
S_{\mu,g} = -\frac{1}{m_g}\sum_{i=1}^{m_g}\log \pi_\mu(a_{g,i}\mid x,a_{g,<i}).
$$
Under the current policy, the naive loss is
$$
\mathcal{L}_{\mathrm{naive}}(\theta) = \frac{1}{G}\sum_{g=1}^G A_g S_{\theta,g}.
$$
The paper’s diagnosis is that, although the advantages sum to zero, the products $A_g S_{\theta,g}$ need not balance because $S_{\theta,g}$ varies across responses. In asynchronous training, stale negative-advantage responses can become much less probable under the current policy than positive responses, making their current $S_{\theta,g}$ larger and causing the negative side of the loss to dominate [2606.03070].

This diagnosis is the **scale-imbalance failure mode**. The paper states that stale responses can be evaluated at very different current-policy loss scales, so zero-sum advantages no longer imply balanced loss contributions. A related appendix interpretation is that the positive/negative contribution ratio under the naive loss is
$$
R_{\mathrm{naive}}=\frac{\bar S_{\mathcal{N}}}{\bar S_{\mathcal{P}}},
$$
whereas the corresponding ratio for ASymPO is exactly $1$. This identifies the instability as a mismatch in response-level scales rather than merely as a generic off-policy issue [2606.03070].

Behavior-corrected methods avoid this specific pathology differently. Because they compare $\pi_\theta$ with $\pi_b$, responses share a common reference scale, and clipping keeps the current-policy scale near the behavior-policy scale. The paper gives a theorem stating, roughly, that if the behavior-policy scales are already fairly balanced and token ratios are clipped to $[1-\epsilon,1+\epsilon]$, then the loss imbalance remains bounded by a term of the form
$$
\left|\Delta_\theta\right| \le 2B(\delta_b+\tau_\epsilon).
$$
This supplies a contrast: behavior correction preserves approximate balance through explicit reference to $\pi_b$, whereas ASymPO restores balance using only current-policy quantities [2606.03070].

## 3. Objective, normalization, and response-level balancing

The paper first defines a general scaled current-policy objective:
$$
\mathcal{L}_C(\theta) = -\frac{1}{G}\sum_{g=1}^G\frac{1}{m_g}\sum_{i=1}^{m_g}A_g C_g\log p_{g,i}(\theta)
= \frac{1}{G}\sum_{g=1}^G A_g C_g S_{\theta,g},
$$
where
$$
p_{g,i}(\theta)=\pi_\theta(a_{g,i}\mid x,a_{g,<i}).
$$
ASymPO chooses the coefficient
$$
C_g^{\mathrm{AsymPO}}=\frac{1}{\operatorname{sg}(S_{\theta,g})},
$$
with $\operatorname{sg}(\cdot)$ denoting stop-gradient. Its loss is therefore
$$
\mathcal{L}_{\mathrm{AsymPO}}(\theta) = \frac{1}{G}\sum_{g=1}^G A_g \frac{S_{\theta,g}}{\operatorname{sg}(S_{\theta,g})},
$$
or, at token level,
$$
\mathcal{L}_{\mathrm{AsymPO}}(\theta) = -\frac{1}{G}\sum_{g=1}^G \frac{1}{m_g}\sum_{i=1}^{m_g} A_g \frac{\log p_{g,i}(\theta)}{\operatorname{sg}(S_{\theta,g})}.
$$
Because the denominator is numerically equal to $S_{\theta,g}$ in the forward pass but detached in backpropagation, the forward response-level contribution is normalized to unit scale while gradients still flow [2606.03070].

The immediate balancing effect is explicit. Since
$$
\frac{S_{\theta,g}}{\operatorname{sg}(S_{\theta,g})}=1
\quad\text{in the forward pass},
$$
the response-level loss becomes
$$
\mathcal{L}_{\mathrm{AsymPO}}(\theta)=\frac{1}{G}\sum_{g=1}^G A_g = 0
$$
whenever $\sum_g A_g=0$. The paper further states
$$
\sum_{g\in\mathcal{P}} A_g\frac{S_{\theta,g}}{\operatorname{sg}(S_{\theta,g})} = \sum_{g\in\mathcal{P}} A_g, \qquad
\sum_{g\in\mathcal{N}} |A_g|\frac{S_{\theta,g}}{\operatorname{sg}(S_{\theta,g})} = \sum_{g\in\mathcal{N}} |A_g|.
$$
Accordingly, positive and negative contributions are exactly balanced at the response level [2606.03070].

The paper also introduces **Scaled Policy Optimization (SPO)** as a simpler baseline. SPO uses a fixed coefficient for negative-advantage responses,
$$
C_g^{\mathrm{SPO}}=
\begin{cases}
1, & A_g\ge 0,\\
\alpha, & A_g<0,
\end{cases}
\qquad \alpha\in(0,1),
$$
which yields a sign-dependent but non-adaptive scaling rule. The contrast is direct: SPO uses a fixed negative scaling, while ASymPO uses an adaptive response-dependent scalar determined by each response’s current scale [2606.03070].

## 4. Gradient behavior, scope, and theoretical claims

ASymPO’s backward pass remains nontrivial because the denominator is detached:
$$
\nabla_\theta \mathcal{L}_{\mathrm{AsymPO}}(\theta) =
\frac{1}{G}\sum_{g=1}^G \frac{A_g}{\operatorname{sg}(S_{\theta,g})}\nabla_\theta S_{\theta,g}.
$$
The paper explicitly identifies this as a **nonzero learning signal**. The forward objective enforces response-level balance, but the gradient still scales each response update by the inverse of its current average token negative log-probability [2606.03070].

The main formal statement is an exact balance theorem: assuming $\sum_g A_g=0$ and $S_{\theta,g}>0$,
$$
\sum_{g=1}^G A_g\frac{S_{\theta,g}}{\operatorname{sg}(S_{\theta,g})}=0.
$$
This is stronger than the approximate balance guaranteed by behavior-corrected clipping: ASymPO restores exact balance in the forward loss without any reference to behavior-policy probabilities [2606.03070].

Two interpretive cautions are explicit. First, ASymPO is **deliberately not an importance-sampling method**. It does not estimate $\pi_\theta/\pi_b$, does not clip importance ratios, and does not require behavior probabilities at all. Second, the paper is careful to say that ASymPO does **not** solve all distribution drift; it addresses only the particular imbalance created when current-policy response scales differ sharply across stale positive- and negative-advantage responses [2606.03070].

This bounded scope is important in avoiding a common misconception. ASymPO is not presented as a full off-policy correction substitute for PPO-style behavior correction. Rather, it is a current-policy-only stabilization mechanism for asynchronous group-relative RL when behavior information is unavailable or operationally undesirable. A plausible implication is that its strongest use case is systems in which rollout–learner decoupling is valuable enough to justify targeting a narrower stability criterion than full behavior correction.

## 5. Experimental evaluation in asynchronous mathematical reasoning

The experiments study **asynchronous mathematical reasoning post-training** with verifiable rewards. The reported models are **Qwen3-1.7B-Base**, **Qwen3-4B-Base**, and **LLaMA-3.2-3B-Instruct**. Training uses randomly sampled **4k problems** from **MATH**, with a supplementary experiment on **4k examples from DAPO-Math-17K**. The training protocol is asynchronous RL with **VeRL**, group size $G=8$ rollouts per prompt, ppo mini batch size **32**, train batch size **512**, staleness threshold **0.5**, max prompt length **1024**, max response length **4096**, learning rate **$2\times 10^{-6}$**, and **3 epochs**; SPO uses $\alpha=0.2$. The compared methods are **Naive current-policy loss**, **GPG**, **GRPO**, **SPO**, and **ASymPO**. Evaluation benchmarks are **AIME24**, **AIME25**, **MATH500**, **AMC23**, **GSM8K**, and **Minerva-Math**, with **mean@8** and **pass@8** as metrics [2606.03070].

A major empirical finding is the stability pattern. The paper reports that **naive loss collapses during training** and **GPG also collapses**, while **GRPO, SPO, and ASymPO remain stable**. The same collapse pattern is reported across model families, and the supplementary DAPO-Math-17K experiment shows the same qualitative result: naive loss and GPG collapse, whereas GRPO, SPO, and ASymPO remain stable [2606.03070].

The quantitative picture is model-dependent rather than uniformly dominated by ASymPO. For **Qwen3-1.7B-Base**, mean@8 average is **37.45** for GRPO, **36.41** for SPO, and **36.98** for ASymPO; pass@8 average is **54.56** for GRPO, **53.92** for SPO, and **53.17** for ASymPO. For **LLaMA-3.2-3B-Instruct**, mean@8 average is **30.84** for GRPO, **29.61** for SPO, and **31.70** for ASymPO, while pass@8 average is **46.09**, **45.25**, and **49.19**, respectively. For **Qwen3-4B-Base**, GRPO is strongest overall, with mean@8 average **45.52** versus **45.45** for SPO and **45.07** for ASymPO, and pass@8 average **60.90** versus **58.06** and **58.87** [2606.03070].

The supplementary DAPO-Math-17K experiment further clarifies the comparison. On **Qwen3-1.7B-Base**, ASymPO achieves the best **mean@8** average at **38.66**, compared with **37.18** for GRPO and **38.13** for SPO, while **pass@8** average is best for SPO at **57.53**, with ASymPO at **56.83** and GRPO at **54.70**. The paper’s own summary is therefore measured: removing behavior-policy probabilities does not automatically break training if scale is controlled; ASymPO stabilizes asynchronous current-policy-only training; and ASymPO is often competitive with GRPO, sometimes better, especially on **LLaMA-3.2-3B-Instruct** [2606.03070].

## 6. Relationship to adjacent asymmetric policy optimization methods

The literature contains several distinct methods whose names or mechanisms are similar enough to invite conflation. ASymPO, in the specific sense of **Asymmetric-Scale Policy Optimization for asynchronous LLM post-training without behavior information**, is best distinguished by its response-level current-policy normalization and its explicit rejection of behavior-policy probabilities [2606.03070].

| Method | Core asymmetry | Setting |
|---|---|---|
| ASymPO | Normalize each response by its current average token negative log-probability | Asynchronous current-policy-only LLM post-training |
| OWPO | Verifier determines direction; reference policy modulates magnitude | RLVR for self-evolving LLMs |
| APO | DADS for positive samples; STCR for negative samples | GRPO-style RL for MLLMs |
| AsyPPO | Large actor with lightweight mini-critics and uncertainty gating | PPO-style RL4LLM reasoning |
| ASPO | Flip IS ratios for positive-advantage tokens | GRPO-style OSRL for math and code |

A useful conceptual antecedent is the unified framework of scaled policy updates in which policy optimization is organized by a gradient form and a scalar scaling function $f(\Delta_O,\Delta_R)$. That framework explicitly treats methods such as PPO as members of a parameterized family of approximate gradient updates, and it shows how sign-dependent clipping and nonlinear scaling functions can induce asymmetric behavior in the $(\Delta_O,\Delta_R)$ plane [2206.08499]. This suggests a broader mathematical backdrop for ASymPO: the method can be read as an especially sharp instance of sample-dependent scaling, specialized to asynchronous current-policy loss geometry rather than to importance-ratio correction.

Among LLM-specific methods, **OWPO** is close in spirit but not in objective. OWPO separates update direction from update magnitude: the verifier determines the direction, while the reference policy adjusts the magnitude through asymmetric reweighting, combining **Accelerated Alignment** for inferior deviations and **Gain Locking** for superior deviations. Its asymmetry is therefore reference-relative and verifier-aware, not stale-response-scale normalization [2605.22156]. **APO** for MLLMs uses a different asymmetry again: **Difficulty-Adaptive Divergence Shaping (DADS)** for positive samples and **Suboptimal Trajectory Complexity Regularization (STCR)** for negative samples, targeting general-task preservation and overthinking control rather than asynchronous behavior-free training [2506.21655].

Two later methods are adjacent in name but orthogonal in mechanics. **AsyPPO** restores critics through lightweight mini-critics trained on disjoint prompt shards and uses inter-critic uncertainty to mask advantages and filter entropy regularization; its asymmetry is architectural, between a large actor and small critics, rather than scale normalization of response losses [2510.01656]. **ASPO** instead identifies a token-weight mismatch in GRPO-style outcome-supervised RL and flips the importance-sampling ratio for positive-advantage tokens, adding soft dual clipping to stabilize extreme updates; its asymmetry is token-level and sign-specific, not response-level and behavior-free [2510.06062].

The resulting taxonomy matters because the shared prefix “asymmetric” does not denote a single method family. In the specific usage attached to [2606.03070], ASymPO refers to a response-scale balancing rule for asynchronous current-policy-only post-training. A plausible implication is that the recent literature uses asymmetry in at least four technically distinct senses: reference-relative scaling, sample-type-specific regularization, actor–critic architectural asymmetry, and current-policy response-scale normalization.

Source: https://www.emergentmind.com/topics/asymmetric-scale-policy-optimization-asympo