ASymPO: Asymmetric-Scale Policy Optimization

• ASymPO is a normalization method that balances token loss scales in asynchronous LLM post-training to counter instability from stale responses.
• It normalizes each response’s average token negative log-probability using current-policy values, eliminating the need for behavior-policy probabilities.
• ASymPO restores exact forward loss balance while preserving a nonzero learning signal during backpropagation in group-relative reinforcement learning.

Searching arXiv for the cited ASymPO paper and closely related asymmetric policy optimization work to ground the article in current literature. Asymmetric-Scale Policy Optimization (ASymPO) is a current-policy-only objective for asynchronous LLM post-training that addresses a specific instability arising from stale rollouts: positive- and negative-advantage responses can be evaluated under the current policy at different negative log-probability scales, so zero-sum group-relative advantages no longer imply balanced loss contributions. The method normalizes each response’s token loss by that response’s current average token negative log-probability, requires no behavior-policy probabilities, restores exact response-level zero-sum balance in the forward loss, and preserves a nonzero learning signal in the backward pass (Liu et al., 2 Jun 2026).

1. Problem setting and motivation

ASymPO is formulated for asynchronous group-relative reinforcement learning in LLM post-training. In this setting, rollout workers generate responses from a stale behavior policy $\pi_b$, the learner updates a newer policy $\pi_\theta$, and responses are optimized later, after $\pi_\theta$ may have drifted from $\pi_b$. The operational benefit is higher throughput through decoupling response generation from policy optimization. The cost is distribution drift, because the sampled data and the optimizing policy are no longer synchronized (Liu et al., 2 Jun 2026).

Standard behavior-corrected methods such as PPO and GRPO typically control this drift with behavior-policy probabilities, importance ratios, or clipping. The paper emphasizes that this requires token-aligned behavior log-probabilities, policy version bookkeeping, and recomputation at training precision to avoid numerical mismatch. ASymPO is motivated by the question of whether asynchronous group-relative RL can instead be stabilized using only current-policy probabilities, thereby removing the behavior-information interface between rollout and learner systems (Liu et al., 2 Jun 2026).

A central design point is that ASymPO is not presented as a general solution to all off-policy effects. The paper’s target is narrower: it isolates a failure mode in current-policy-only asynchronous training and proposes a normalization that fixes that specific imbalance without recovering behavior-policy probabilities or importance weights (Liu et al., 2 Jun 2026).

2. Scale imbalance under stale rollouts

The paper defines group-relative advantages for a prompt $x$ using a group of $G$ sampled responses with rewards $r_g$:

$$A_g = r_g - \hat r, \qquad \hat r = \frac{1}{G}\sum_{j=1}^G r_j,$$

so that

$\sum_{g=1}^G A_g = 0.$

This zero-sum structure is intended to stabilize learning by making positive and negative advantages cancel in aggregate (Liu et al., 2 Jun 2026).

For a policy $\pi_\mu$, the response-level average token negative log-probability is defined as

$\pi_\theta$0

Under the current policy, the naive loss is

$\pi_\theta$1

The paper’s diagnosis is that, although the advantages sum to zero, the products $\pi_\theta$2 need not balance because $\pi_\theta$3 varies across responses. In asynchronous training, stale negative-advantage responses can become much less probable under the current policy than positive responses, making their current $\pi_\theta$4 larger and causing the negative side of the loss to dominate (Liu et al., 2 Jun 2026).

This diagnosis is the scale-imbalance failure mode. The paper states that stale responses can be evaluated at very different current-policy loss scales, so zero-sum advantages no longer imply balanced loss contributions. A related appendix interpretation is that the positive/negative contribution ratio under the naive loss is

$\pi_\theta$5

whereas the corresponding ratio for ASymPO is exactly $\pi_\theta$6. This identifies the instability as a mismatch in response-level scales rather than merely as a generic off-policy issue (Liu et al., 2 Jun 2026).

Behavior-corrected methods avoid this specific pathology differently. Because they compare $\pi_\theta$7 with $\pi_\theta$8, responses share a common reference scale, and clipping keeps the current-policy scale near the behavior-policy scale. The paper gives a theorem stating, roughly, that if the behavior-policy scales are already fairly balanced and token ratios are clipped to $\pi_\theta$9, then the loss imbalance remains bounded by a term of the form

$\pi_\theta$0

This supplies a contrast: behavior correction preserves approximate balance through explicit reference to $\pi_\theta$1, whereas ASymPO restores balance using only current-policy quantities (Liu et al., 2 Jun 2026).

3. Objective, normalization, and response-level balancing

The paper first defines a general scaled current-policy objective:

$\pi_\theta$2

where

$\pi_\theta$3

ASymPO chooses the coefficient

$\pi_\theta$4

with $\pi_\theta$5 denoting stop-gradient. Its loss is therefore

$\pi_\theta$6

or, at token level,

$\pi_\theta$7

Because the denominator is numerically equal to $\pi_\theta$8 in the forward pass but detached in backpropagation, the forward response-level contribution is normalized to unit scale while gradients still flow (Liu et al., 2 Jun 2026).

The immediate balancing effect is explicit. Since

$\pi_\theta$9

the response-level loss becomes

$\pi_b$0

whenever $\pi_b$1. The paper further states

$\pi_b$2

Accordingly, positive and negative contributions are exactly balanced at the response level (Liu et al., 2 Jun 2026).

The paper also introduces Scaled Policy Optimization (SPO) as a simpler baseline. SPO uses a fixed coefficient for negative-advantage responses,

$\pi_b$3

which yields a sign-dependent but non-adaptive scaling rule. The contrast is direct: SPO uses a fixed negative scaling, while ASymPO uses an adaptive response-dependent scalar determined by each response’s current scale (Liu et al., 2 Jun 2026).

4. Gradient behavior, scope, and theoretical claims

ASymPO’s backward pass remains nontrivial because the denominator is detached:

$\pi_b$4

The paper explicitly identifies this as a nonzero learning signal. The forward objective enforces response-level balance, but the gradient still scales each response update by the inverse of its current average token negative log-probability (Liu et al., 2 Jun 2026).

The main formal statement is an exact balance theorem: assuming $\pi_b$5 and $\pi_b$6,

$\pi_b$7

This is stronger than the approximate balance guaranteed by behavior-corrected clipping: ASymPO restores exact balance in the forward loss without any reference to behavior-policy probabilities (Liu et al., 2 Jun 2026).

Two interpretive cautions are explicit. First, ASymPO is deliberately not an importance-sampling method. It does not estimate $\pi_b$8, does not clip importance ratios, and does not require behavior probabilities at all. Second, the paper is careful to say that ASymPO does not solve all distribution drift; it addresses only the particular imbalance created when current-policy response scales differ sharply across stale positive- and negative-advantage responses (Liu et al., 2 Jun 2026).

This bounded scope is important in avoiding a common misconception. ASymPO is not presented as a full off-policy correction substitute for PPO-style behavior correction. Rather, it is a current-policy-only stabilization mechanism for asynchronous group-relative RL when behavior information is unavailable or operationally undesirable. A plausible implication is that its strongest use case is systems in which rollout–learner decoupling is valuable enough to justify targeting a narrower stability criterion than full behavior correction.

5. Experimental evaluation in asynchronous mathematical reasoning

The experiments study asynchronous mathematical reasoning post-training with verifiable rewards. The reported models are Qwen3-1.7B-Base, Qwen3-4B-Base, and LLaMA-3.2-3B-Instruct. Training uses randomly sampled 4k problems from MATH, with a supplementary experiment on 4k examples from DAPO-Math-17K. The training protocol is asynchronous RL with VeRL, group size $\pi_b$9 rollouts per prompt, ppo mini batch size 32, train batch size 512, staleness threshold 0.5, max prompt length 1024, max response length 4096, learning rate $x$0, and 3 epochs; SPO uses $x$1. The compared methods are Naive current-policy loss, GPG, GRPO, SPO, and ASymPO. Evaluation benchmarks are AIME24, AIME25, MATH500, AMC23, GSM8K, and Minerva-Math, with mean@8 and pass@8 as metrics (Liu et al., 2 Jun 2026).

A major empirical finding is the stability pattern. The paper reports that naive loss collapses during training and GPG also collapses, while GRPO, SPO, and ASymPO remain stable. The same collapse pattern is reported across model families, and the supplementary DAPO-Math-17K experiment shows the same qualitative result: naive loss and GPG collapse, whereas GRPO, SPO, and ASymPO remain stable (Liu et al., 2 Jun 2026).

The quantitative picture is model-dependent rather than uniformly dominated by ASymPO. For Qwen3-1.7B-Base, mean@8 average is 37.45 for GRPO, 36.41 for SPO, and 36.98 for ASymPO; pass@8 average is 54.56 for GRPO, 53.92 for SPO, and 53.17 for ASymPO. For LLaMA-3.2-3B-Instruct, mean@8 average is 30.84 for GRPO, 29.61 for SPO, and 31.70 for ASymPO, while pass@8 average is 46.09, 45.25, and 49.19, respectively. For Qwen3-4B-Base, GRPO is strongest overall, with mean@8 average 45.52 versus 45.45 for SPO and 45.07 for ASymPO, and pass@8 average 60.90 versus 58.06 and 58.87 (Liu et al., 2 Jun 2026).

The supplementary DAPO-Math-17K experiment further clarifies the comparison. On Qwen3-1.7B-Base, ASymPO achieves the best mean@8 average at 38.66, compared with 37.18 for GRPO and 38.13 for SPO, while pass@8 average is best for SPO at 57.53, with ASymPO at 56.83 and GRPO at 54.70. The paper’s own summary is therefore measured: removing behavior-policy probabilities does not automatically break training if scale is controlled; ASymPO stabilizes asynchronous current-policy-only training; and ASymPO is often competitive with GRPO, sometimes better, especially on LLaMA-3.2-3B-Instruct (Liu et al., 2 Jun 2026).

6. Relationship to adjacent asymmetric policy optimization methods

The literature contains several distinct methods whose names or mechanisms are similar enough to invite conflation. ASymPO, in the specific sense of Asymmetric-Scale Policy Optimization for asynchronous LLM post-training without behavior information, is best distinguished by its response-level current-policy normalization and its explicit rejection of behavior-policy probabilities (Liu et al., 2 Jun 2026).

Method	Core asymmetry	Setting
ASymPO	Normalize each response by its current average token negative log-probability	Asynchronous current-policy-only LLM post-training
OWPO	Verifier determines direction; reference policy modulates magnitude	RLVR for self-evolving LLMs
APO	DADS for positive samples; STCR for negative samples	GRPO-style RL for MLLMs
AsyPPO	Large actor with lightweight mini-critics and uncertainty gating	PPO-style RL4LLM reasoning
ASPO	Flip IS ratios for positive-advantage tokens	GRPO-style OSRL for math and code

A useful conceptual antecedent is the unified framework of scaled policy updates in which policy optimization is organized by a gradient form and a scalar scaling function $x$2. That framework explicitly treats methods such as PPO as members of a parameterized family of approximate gradient updates, and it shows how sign-dependent clipping and nonlinear scaling functions can induce asymmetric behavior in the $x$3 plane (Gummadi et al., 2022). This suggests a broader mathematical backdrop for ASymPO: the method can be read as an especially sharp instance of sample-dependent scaling, specialized to asynchronous current-policy loss geometry rather than to importance-ratio correction.

Among LLM-specific methods, OWPO is close in spirit but not in objective. OWPO separates update direction from update magnitude: the verifier determines the direction, while the reference policy adjusts the magnitude through asymmetric reweighting, combining Accelerated Alignment for inferior deviations and Gain Locking for superior deviations. Its asymmetry is therefore reference-relative and verifier-aware, not stale-response-scale normalization (Yang et al., 21 May 2026). APO for MLLMs uses a different asymmetry again: Difficulty-Adaptive Divergence Shaping (DADS) for positive samples and Suboptimal Trajectory Complexity Regularization (STCR) for negative samples, targeting general-task preservation and overthinking control rather than asynchronous behavior-free training (Hong et al., 26 Jun 2025).

Two later methods are adjacent in name but orthogonal in mechanics. AsyPPO restores critics through lightweight mini-critics trained on disjoint prompt shards and uses inter-critic uncertainty to mask advantages and filter entropy regularization; its asymmetry is architectural, between a large actor and small critics, rather than scale normalization of response losses (Liu et al., 2 Oct 2025). ASPO instead identifies a token-weight mismatch in GRPO-style outcome-supervised RL and flips the importance-sampling ratio for positive-advantage tokens, adding soft dual clipping to stabilize extreme updates; its asymmetry is token-level and sign-specific, not response-level and behavior-free (Wang et al., 7 Oct 2025).

The resulting taxonomy matters because the shared prefix “asymmetric” does not denote a single method family. In the specific usage attached to (Liu et al., 2 Jun 2026), ASymPO refers to a response-scale balancing rule for asynchronous current-policy-only post-training. A plausible implication is that the recent literature uses asymmetry in at least four technically distinct senses: reference-relative scaling, sample-type-specific regularization, actor–critic architectural asymmetry, and current-policy response-scale normalization.