Token-Level Double-Sided Importance Sampling (DIS)

• The paper introduces an asymmetric token weighting mechanism that flips positive token importance sampling ratios to better align gradient updates.
• It details the use of dual-clipping and cumulative token correction to balance bias–variance tradeoffs in policy optimization.
• Empirical results demonstrate enhanced performance on math and code benchmarks, highlighting stabilized training and improved metrics.

Searching arXiv for the cited work and closely related importance-sampling policy optimization papers. Token-level double-sided importance sampling (DIS) denotes a class of token-level importance-weighting schemes for LLM policy optimization that treat the two sides of the importance-ratio spectrum differently rather than applying a single monotonic rule to all tokens. In the literature summarized here, the closest explicit formulation is the asymmetric treatment introduced by "ASPO: Asymmetric Importance Sampling Policy Optimization" (Wang et al., 7 Oct 2025), which argues that in GRPO-style outcome-supervised RL (OSRL), token-level importance sampling (IS) ratios cease to function primarily as classical off-policy correction and instead act as token weights in the gradient. From that perspective, a double-sided design means that negative-advantage and positive-advantage tokens are handled differently, with additional two-sided clipping to control extreme ratios. A separate line of work, "Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective" (Zhang et al., 8 May 2026), studies token-level, full-sequence, and cumulative token IS from a bias–variance viewpoint and uses double-sided clipping in log-space with position-adaptive bounds. Taken together, these works situate token-level DIS at the intersection of token weighting, off-policy correction, and clipping-based stabilization in LLM RL.

1. Conceptual framing

In GRPO-style OSRL, the per-token importance ratio is written as

$$r_t^i(\theta)=\frac{\pi_\theta(o_t^i \mid q,o_{<t}^i)}{\pi_{\theta_{\text{old}}}(o_t^i \mid q,o_{<t}^i)}.$$

The central claim of ASPO is that, once a response-level advantage is shared across tokens and token clipping is applied locally, this quantity is no longer serving its classical off-policy correction role in a meaningful way; instead, it effectively behaves like a token weight in the gradient (Wang et al., 7 Oct 2025).

Within that framing, the relevant asymmetry is between negative-advantage tokens and positive-advantage tokens. For negative-advantage tokens, PPO/GRPO clipping is described as matching the desired intuition: tokens that are less probable under the current policy tend to receive stronger corrective updates, while tokens already over-favored are down-weighted. For positive-advantage tokens, the behavior is described as inverted: tokens with higher current probability are given larger weights, while tokens with lower current probability are suppressed (Wang et al., 7 Oct 2025).

This is the sense in which token-level DIS is "double-sided" in the ASPO discussion. It is not presented as generic unbiased importance sampling correction. Rather, it is a two-sided or asymmetric token-weighting rule in which the positive and negative regions are treated differently to better match the intended optimization dynamics. The paper explicitly distinguishes this from conventional double-sided importance sampling understood as balanced, two-sided importance weighting for sampling correction (Wang et al., 7 Oct 2025).

A related but distinct notion of two-sidedness appears in CTPO. There, the emphasis is not on advantage-sign asymmetry but on correcting token-level policy gradients with a cumulative prefix ratio and clipping that has both lower and upper bounds in log-space. This is also a double-sided treatment, but it is motivated by prefix correction and variance control rather than by positive/negative token asymmetry (Zhang et al., 8 May 2026).

2. Standard token-level IS and the asymmetry problem

The standard PPO/GRPO token-level clipped surrogate uses

$$\min\left(r_t^i(\theta)\hat A_t^i,\;\operatorname{clip}(r_t^i(\theta),1-\varepsilon,1+\varepsilon)\hat A_t^i\right).$$

ASPO identifies a weight mismatch in this construction (Wang et al., 7 Oct 2025).

For negative-advantage tokens, the reported behavior aligns with the intended corrective effect. For positive-advantage tokens, however, the weighting is described as reversed relative to that intuition: already high-probability tokens receive larger weights, while lagging low-probability tokens receive smaller ones. The paper attributes several consequences to this mismatch: over-updating already confident tokens, under-updating lagging tokens, and a self-reinforcing loop associated with entropy collapse, repetition, and premature convergence to local optima (Wang et al., 7 Oct 2025).

The compact technical intuition given in the paper is that, in GRPO/PPO-Clip, token-level IS ratios act as weights; for negative-advantage tokens the weighting is sensible, while for positive-advantage tokens it is inverted, favoring already high-probability tokens and suppressing low-probability ones. This motivates a double-sided treatment in which the positive side is explicitly modified (Wang et al., 7 Oct 2025).

CTPO studies the same broader design space from a different angle. It argues that the standard token-level ratio

$$r_t = \frac{\pi_\theta(a_t \mid s_t)}{\pi_b(a_t \mid s_t)}$$

is cheap and stable, but only corrects the current action probability and not the prefix-state distribution. In that sense, token-level IS is too local and biases off-policy gradients because it ignores prefix mismatch (Zhang et al., 8 May 2026). This is a different criticism from ASPO’s positive/negative asymmetry, but both papers treat token-level ratios as central objects whose local behavior strongly shapes training dynamics.

3. Asymmetric ratio flipping as a token-level DIS mechanism

ASPO preserves the GRPO-style masking and clipping logic for negative-advantage tokens and changes the ratio used for positive-advantage tokens (Wang et al., 7 Oct 2025).

For negative-advantage tokens, ASPO uses the original ratio: $\hat r_t^i = r_t^i.$

For positive-advantage tokens, it uses

$$\hat{r}_t^i = \frac{\pi_{\theta_{\text{old}}}(o_t^i \mid q, o_{<t}^i)\,\pi_{\theta}(o_t^i \mid q, o_{<t}^i)} {\operatorname{sg}\!\left(\pi_{\theta}^2(o_t^i \mid q, o_{<t}^i)\right)},$$

which is intended to behave algebraically like

$$\hat r_t^i \approx \frac{\pi_{\theta_{\text{old}}}(o_t^i \mid q,o_{<t}^i)}{\pi_{\theta}(o_t^i \mid q,o_{<t}^i)},$$

namely the inverse of the standard IS ratio, with the denominator stop-grad’d in implementation (Wang et al., 7 Oct 2025).

This inversion changes the weighting rule on the positive side: lower current probability implies larger weight. The paper presents this as the key mechanism by which the update direction for positive tokens is aligned with the learning dynamics already seen for negative ones (Wang et al., 7 Oct 2025).

In the paper’s own comparison to DIS, the similarity is that ASPO does not treat all tokens with a single monotonic ratio rule. Negative-advantage tokens keep the usual IS ratio, while positive-advantage tokens receive the flipped reciprocal ratio. That is explicitly described as a two-sided or asymmetric treatment of token weights. The difference from conventional DIS is equally explicit: ASPO is not about variance-reduced off-policy estimation or symmetric correction of sampling mismatch, but about reassigning token-level learning weights (Wang et al., 7 Oct 2025).

A plausible implication is that, within this line of work, token-level DIS is best understood not as a single named algorithm but as a design principle: the two sides of the token-ratio or token-advantage regime are assigned different weighting behavior to avoid pathological reinforcement of already dominant tokens.

4. Dual-clipping and gradient-flow preservation

ASPO supplements ratio flipping with dual-clipping. After the positive-token ratio is flipped, extreme values can occur on the positive side, so positive tokens also require dual-clipping. The stated goal is to constrain extreme ratios without killing gradients (Wang et al., 7 Oct 2025).

The mechanism is described conceptually rather than through a single compact final equation. Its components are: retaining the original token masking rules for tokens already sufficiently updated, flipping positive-token IS ratios, and applying soft dual-clipping to positive tokens whose flipped ratios become extreme. The paper contrasts soft clipping with hard clipping: soft clipping clips the value but preserves the gradient, whereas hard clipping clips both value and gradient (Wang et al., 7 Oct 2025).

The paper also discusses a soft-clipping variant used by CISPO: $$\operatorname{sg}\!\left(\operatorname{clip}\big(r_t^i(\theta),1-\varepsilon_{\text{low}},1+\varepsilon_{\text{high}}\big)\right),$$ where $\operatorname{sg}(\cdot)$ denotes stop-gradient, so the value is clipped while the gradient is preserved (Wang et al., 7 Oct 2025). In ASPO, this conceptual model motivates the use of soft dual-clipping once the positive side has been inverted.

The reported clipping thresholds are

$$\varepsilon_{\text{low}}=0.2,\qquad \varepsilon_{\text{high}}=0.28.$$

The text indicates hard masking for tokens satisfying conditions such as $r_t^i(\theta) < 1-\varepsilon_{\text{low}}$ when $$\min\left(r_t^i(\theta)\hat A_t^i,\;\operatorname{clip}(r_t^i(\theta),1-\varepsilon,1+\varepsilon)\hat A_t^i\right).$$0 and $$\min\left(r_t^i(\theta)\hat A_t^i,\;\operatorname{clip}(r_t^i(\theta),1-\varepsilon,1+\varepsilon)\hat A_t^i\right).$$1 when $$\min\left(r_t^i(\theta)\hat A_t^i,\;\operatorname{clip}(r_t^i(\theta),1-\varepsilon,1+\varepsilon)\hat A_t^i\right).$$2, while noting formatting corruption in the PDF and identifying the intended meaning as standard PPO-style asymmetric clipping with separate lower and upper thresholds (Wang et al., 7 Oct 2025).

CTPO provides a related but differently motivated double-sided clipping construction. It clips the cumulative token ratio with lower and upper bounds in log-space: $$\min\left(r_t^i(\theta)\hat A_t^i,\;\operatorname{clip}(r_t^i(\theta),1-\varepsilon,1+\varepsilon)\hat A_t^i\right).$$3 This is explicitly described as effectively double-sided in log-space, with a lower bound $$\min\left(r_t^i(\theta)\hat A_t^i,\;\operatorname{clip}(r_t^i(\theta),1-\varepsilon,1+\varepsilon)\hat A_t^i\right).$$4 and an upper bound $$\min\left(r_t^i(\theta)\hat A_t^i,\;\operatorname{clip}(r_t^i(\theta),1-\varepsilon,1+\varepsilon)\hat A_t^i\right).$$5, mirroring PPO-style two-sided clipping but with position-adaptive bounds (Zhang et al., 8 May 2026).

5. Relation to cumulative token importance ratios

CTPO introduces the cumulative token IS ratio

$$\min\left(r_t^i(\theta)\hat A_t^i,\;\operatorname{clip}(r_t^i(\theta),1-\varepsilon,1+\varepsilon)\hat A_t^i\right).$$6

This is presented as the ratio of prefix trajectory probabilities and as the exact prefix correction needed for per-token policy-gradient terms in the deterministic token-generation MDP formulation (Zhang et al., 8 May 2026).

The paper compares three IS regimes:

Ratio type	Form	Position in the bias–variance tradeoff
Token-level	$$\min\left(r_t^i(\theta)\hat A_t^i,\;\operatorname{clip}(r_t^i(\theta),1-\varepsilon,1+\varepsilon)\hat A_t^i\right).$$7	Low variance, but biased
Full-sequence	$$\min\left(r_t^i(\theta)\hat A_t^i,\;\operatorname{clip}(r_t^i(\theta),1-\varepsilon,1+\varepsilon)\hat A_t^i\right).$$8	Unbiased, but high variance
Cumulative token	$$\min\left(r_t^i(\theta)\hat A_t^i,\;\operatorname{clip}(r_t^i(\theta),1-\varepsilon,1+\varepsilon)\hat A_t^i\right).$$9	Unbiased prefix correction with lower variance than full-sequence

For $$r_t = \frac{\pi_\theta(a_t \mid s_t)}{\pi_b(a_t \mid s_t)}$$0, the paper states that

$$r_t = \frac{\pi_\theta(a_t \mid s_t)}{\pi_b(a_t \mid s_t)}$$1

in general, because $$r_t = \frac{\pi_\theta(a_t \mid s_t)}{\pi_b(a_t \mid s_t)}$$2 ignores prefix-state mismatch (Zhang et al., 8 May 2026). By contrast, the cumulative ratio is claimed to provide unbiased prefix correction for each token-level gradient term: $$r_t = \frac{\pi_\theta(a_t \mid s_t)}{\pi_b(a_t \mid s_t)}$$3

This establishes a different sense in which token-level DIS may be interpreted. In ASPO, double-sidedness refers to asymmetric treatment across positive and negative advantages. In CTPO, double-sidedness appears in clipping, while the main innovation is the replacement of local token ratios with cumulative prefix ratios. The two papers therefore address distinct pathologies of token-level IS: one focuses on asymmetry in token weighting under OSRL, the other on bias from inadequate prefix correction and variance from unnecessary suffix terms (Wang et al., 7 Oct 2025, Zhang et al., 8 May 2026).

6. Optimization dynamics, empirical evidence, and benchmark results

ASPO provides both analytical and experimental evidence for its asymmetric design. The analytical argument emphasizes the effective gradient weight. For original GRPO, that weight is proportional to

$$r_t = \frac{\pi_\theta(a_t \mid s_t)}{\pi_b(a_t \mid s_t)}$$4

whereas for ASPO the positive region yields a flipped term positively correlated with

$$r_t = \frac{\pi_\theta(a_t \mid s_t)}{\pi_b(a_t \mid s_t)}$$5

The intended learning dynamic is therefore that lower current token probability implies larger gradient weight (Wang et al., 7 Oct 2025).

The paper reports a diagnostic comparison between GRPO with original IS and GRPO with all IS weights set to $$r_t = \frac{\pi_\theta(a_t \mid s_t)}{\pi_b(a_t \mid s_t)}$$6. It finds little difference in final test accuracy, while the no-IS version is smoother and more stable. This is used to support the claim that the original token IS weights are not functioning as classical distribution correction but rather as token weighting that can be misallocated (Wang et al., 7 Oct 2025).

A further positive-token reweighting experiment replaces token-level IS weights for positive tokens with response-level average IS weights. The reported effects are substantially smoother training, slower entropy decline, moderated repetition and KL growth, and improved pass@8 later in training. This is presented as evidence that original positive-token IS ratios are a source of instability (Wang et al., 7 Oct 2025).

For the full ASPO method, the paper evaluates on math and code benchmarks using DeepSeek-R1-Distill-Qwen-1.5B and reports gains of +12.5% over the base model on math and +17.0% on coding. It also reports that ASPO outperforms DAPO, DeepScaleR-1.5B, DeepCoder-1.5B, FastCuRL-1.5B-V3, and Nemotron-1.5B averaged across benchmarks (Wang et al., 7 Oct 2025). In training dynamics, ASPO is reported to produce more gradual entropy decrease, slower increase in repetition rate, slower growth in clip ratio, more stable KL divergence, and better late-stage performance (Wang et al., 7 Oct 2025).

CTPO evaluates cumulative token IS in tool-integrated reasoning, with up to 5 turns of model-environment interaction, maximum response length 8,000 tokens, Qwen3-4B and Qwen3-14B, DeepScaleR training data of about 40K math problems, and avg@32 evaluation on AIME 2025, AIME 2026, HMMT 2025, and BRUMO 2025 (Zhang et al., 8 May 2026). The reported averages are:

Model	GRPO	GSPO	CTPO
Qwen3-4B	43.2	47.7	51.4
Qwen3-14B	54.6	55.5	58.8

CTPO is reported to improve over GSPO by 3.7 points average on Qwen3-4B and 3.3 points average on Qwen3-14B. An ablation further reports that replacing adaptive clipping with fixed clipping reduces performance from 58.8 avg to 55.7 avg (Zhang et al., 8 May 2026).

7. Interpretation, scope, and limitations

The two papers imply different meanings for the term token-level DIS, and distinguishing them avoids a common misconception. A generic reading of DIS might suggest unbiased importance sampling correction on both sides of the ratio spectrum. ASPO explicitly rejects that framing for its own method: it argues that classical IS is not very important in the OSRL setting it studies and that the ratio mainly acts as a token-weighting device. Its double-sidedness is therefore a deliberately asymmetric optimization heuristic over token weights, not a claim of unbiased off-policy estimation (Wang et al., 7 Oct 2025).

CTPO, by contrast, is explicitly concerned with off-policy correction. It claims that cumulative token IS is the theoretically principled solution to the token-level versus full-sequence bias–variance dilemma: token-level IS is biased because it ignores prefix mismatch, full-sequence IS is exact but unnecessarily noisy, and cumulative token IS is the exact prefix correction needed for token-level policy gradients while avoiding suffix variance (Zhang et al., 8 May 2026).

The theoretical scope of CTPO is also qualified. The cumulative-ratio unbiasedness result is stated for token-level policy-gradient terms under the standard deterministic token-generation MDP. The cleaner closed-form variance expression assumes independence of per-token ratios, and clipping remains a bias-inducing stabilization mechanism as in PPO/GRPO (Zhang et al., 8 May 2026). The empirical scope is further limited to tool-integrated mathematical reasoning on two Qwen3 scales (Zhang et al., 8 May 2026).

ASPO’s claims are likewise scoped to GRPO-style OSRL with response-level shared advantages and token-level clipping. Its central argument is that the local token ratio functions as a weight and that the positive side is misallocated under standard PPO/GRPO logic. This suggests that the relevance of its asymmetric double-sided rule depends on the same structural conditions: sequence-level rewards distributed uniformly across tokens, local clipping, and optimization regimes where entropy collapse and repetition emerge from self-reinforcing token weighting (Wang et al., 7 Oct 2025).

Taken together, the literature indicates that token-level DIS is not a single settled method but a family resemblance across approaches that use two-sided treatment of token ratios. One branch, exemplified by ASPO, uses asymmetric weighting rules across positive and negative advantages to alter learning dynamics. Another, exemplified by CTPO, uses prefix-corrected cumulative ratios with two-sided clipping to manage the bias–variance structure of token-level policy gradients (Wang et al., 7 Oct 2025, Zhang et al., 8 May 2026).