Scaled Policy Optimization in Reinforcement Learning

• Scaled Policy Optimization (SPO) is a reinforcement learning framework that stabilizes asynchronous training by applying fixed asymmetric scaling to negative advantages.
• It encompasses distinct methods, including a current-policy-only objective for LLM post-training and an SMC-based approach for efficient policy distillation.
• SPO mitigates scale imbalance by uniformly downweighting stale negative responses, achieving competitive performance against behavior-corrected baselines.

Scaled Policy Optimization (SPO) is an overloaded term in contemporary reinforcement learning. In the most explicit usage, it denotes a current-policy-only, group-relative reinforcement learning objective for asynchronous large-language-model post-training that stabilizes learning without behavior-policy probabilities by asymmetrically downweighting negative-advantage responses with a fixed coefficient (Liu et al., 2 Jun 2026). A separate usage associates “scaled” with the efficient accelerator scaling properties of “SPO: Sequential Monte Carlo Policy Optimisation,” where Sequential Monte Carlo (SMC) planning produces improved action targets that are then distilled into a parametric policy through forward-KL projection inside an Expert Iteration loop (Macfarlane et al., 2024). The term therefore requires careful disambiguation.

1. Terminology and scope

In the asynchronous post-training literature, Scaled Policy Optimization is defined as a fixed negative-scaling baseline introduced alongside ASymPO. Its purpose is to address a specific failure mode—scale imbalance—when stale responses are evaluated only under the current policy and behavior-policy probabilities are unavailable (Liu et al., 2 Jun 2026). In that setting, SPO is not a generic synonym for PPO-style optimization, nor is it a trust-region method in the classical TRPO sense; it is a current-policy-only objective with asymmetric response-level scaling.

The acronym is also used for unrelated methods. “SPO” denotes “Sequential Monte Carlo Policy Optimisation” in model-based reinforcement learning (Macfarlane et al., 2024), “Simple Policy Optimization” in first-order KL-aware policy optimization (Xie et al., 2024), “Segment Policy Optimization” in reinforcement learning for LLMs (Guo et al., 29 May 2025), “Soft Policy Optimization” in off-policy sequence-model training (Cohen et al., 7 Mar 2025), “Single-stream Policy Optimization” in group-free RLVR (Xu et al., 16 Sep 2025), and “Sinkhorn Policy Optimization” in metric-aware trust-region policy optimization (Song et al., 2023). This suggests that “Scaled Policy Optimization” is not a standardized field-wide label, but a paper-specific designation whose meaning depends on context.

2. Group-relative formulation in asynchronous LLM post-training

The Scaled Policy Optimization objective introduced in asynchronous post-training is defined in a group-relative setting. For a prompt $x$ and response $y_g=(a_{g,1},\dots,a_{g,m_g})$, the current-policy token probabilities and average token negative log-probability are

$$p_{g,i}(\theta)=\pi_\theta(a_{g,i}\mid x,a_{g,<i}),\quad S_{\theta,g}=-\frac{1}{m_g}\sum_{i=1}^{m_g}\log p_{g,i}(\theta).$$

With group-relative rewards $r_g$ and zero-sum advantages

$$A_g=r_g-\frac{1}{G}\sum_{j=1}^G r_j,\qquad \sum_g A_g=0,$$

the paper considers the general scaled objective

$$\mathcal{L}_{C}(\theta) =-\frac{1}{G}\sum_{g=1}^G\frac{1}{m_g}\sum_{i=1}^{m_g}A_g C_g\log p_{g,i}(\theta) =\frac{1}{G}\sum_{g=1}^G A_g C_g S_{\theta,g}.$$

This formulation is explicitly current-policy-only. The learner requires sampled tokens and scalar rewards per response, then recomputes all token log-probabilities under $\pi_\theta$. It does not require behavior-policy log-probabilities, policy-version tags, or importance ratios. No special token alignment or numerical consistency with the rollout system is needed beyond standard deterministic tokenization and model evaluation on the learner (Liu et al., 2 Jun 2026).

The resulting objective is tailored to asynchronous pipelines in which rollout workers may sample responses under stale policies while the learner trains a newer current policy. In such pipelines, the relevant scale is the current-policy average token negative log-probability $S_{\theta,g}$, which becomes the quantity through which stale-response mismatch manifests.

3. Fixed asymmetric negative scaling

SPO instantiates the general scaled objective with a fixed asymmetric rule:

$$C_g^{\mathrm{SPO}}= \begin{cases} 1, & A_g\geq 0,\ \alpha, & A_g<0, \end{cases}$$

where $\alpha\in(0,1)$ is a hyperparameter. Equivalently, if $y_g=(a_{g,1},\dots,a_{g,m_g})$0 for $y_g=(a_{g,1},\dots,a_{g,m_g})$1 and $y_g=(a_{g,1},\dots,a_{g,m_g})$2 for $y_g=(a_{g,1},\dots,a_{g,m_g})$3, then SPO minimizes

$y_g=(a_{g,1},\dots,a_{g,m_g})$4

The asymmetry is the essential design choice. Positive-advantage terms are unscaled, whereas negative-advantage terms are downweighted by the fixed factor $y_g=(a_{g,1},\dots,a_{g,m_g})$5. The update direction is preserved—positive responses are reinforced and negative responses are suppressed—but negative updates are made weaker (Liu et al., 2 Jun 2026).

Because $y_g=(a_{g,1},\dots,a_{g,m_g})$6 does not depend on $y_g=(a_{g,1},\dots,a_{g,m_g})$7, the gradient has the simple form

$y_g=(a_{g,1},\dots,a_{g,m_g})$8

This preserves the rollout–learner interface of current-policy-only training while introducing a deterministic response-level correction. Relative to adaptive normalization schemes, the rule is deliberately minimal: one scalar $y_g=(a_{g,1},\dots,a_{g,m_g})$9 governs all negative responses.

4. Scale imbalance and comparison with related objectives

The motivation for SPO is a failure mode identified in asynchronous training without behavior information. If stale responses are evaluated under the current policy, the naive current-policy-only loss

$$p_{g,i}(\theta)=\pi_\theta(a_{g,i}\mid x,a_{g,<i}),\quad S_{\theta,g}=-\frac{1}{m_g}\sum_{i=1}^{m_g}\log p_{g,i}(\theta).$$0

can become unstable when stale negative-advantage responses have much larger current negative log-probability than positive responses. In the notation of the paper, instability appears when

$$p_{g,i}(\theta)=\pi_\theta(a_{g,i}\mid x,a_{g,<i}),\quad S_{\theta,g}=-\frac{1}{m_g}\sum_{i=1}^{m_g}\log p_{g,i}(\theta).$$1

even though $$p_{g,i}(\theta)=\pi_\theta(a_{g,i}\mid x,a_{g,<i}),\quad S_{\theta,g}=-\frac{1}{m_g}\sum_{i=1}^{m_g}\log p_{g,i}(\theta).$$2 (Liu et al., 2 Jun 2026). Zero-sum advantages therefore do not imply balanced loss contributions once responses are evaluated at different current-policy scales.

SPO addresses this by shrinking the negative side uniformly:

$$p_{g,i}(\theta)=\pi_\theta(a_{g,i}\mid x,a_{g,<i}),\quad S_{\theta,g}=-\frac{1}{m_g}\sum_{i=1}^{m_g}\log p_{g,i}(\theta).$$3

The paper’s interpretation is that fixed negative scaling reduces the tendency of stale negatives to dominate. If the naive negative-to-positive contribution ratio is $$p_{g,i}(\theta)=\pi_\theta(a_{g,i}\mid x,a_{g,<i}),\quad S_{\theta,g}=-\frac{1}{m_g}\sum_{i=1}^{m_g}\log p_{g,i}(\theta).$$4, SPO scales it to approximately $$p_{g,i}(\theta)=\pi_\theta(a_{g,i}\mid x,a_{g,<i}),\quad S_{\theta,g}=-\frac{1}{m_g}\sum_{i=1}^{m_g}\log p_{g,i}(\theta).$$5, bringing it closer to unity when negatives are over-scaled (Liu et al., 2 Jun 2026).

This mechanism differs from both behavior-corrected and adaptive current-policy-only approaches. Behavior-corrected methods such as PPO- or GRPO-style objectives use behavior-policy probabilities, importance ratios, and clipping, providing a common reference scale and bounding drift. ASymPO, introduced in the same paper, instead normalizes each response by its own current scale,

$$p_{g,i}(\theta)=\pi_\theta(a_{g,i}\mid x,a_{g,<i}),\quad S_{\theta,g}=-\frac{1}{m_g}\sum_{i=1}^{m_g}\log p_{g,i}(\theta).$$6

so that the forward loss exactly inherits the group’s zero-sum balance, while the gradient normalizes each response’s update by its own current scale (Liu et al., 2 Jun 2026). SPO does not preserve exact response-level zero-sum balance. It approximates balance by uniformly shrinking the negative side rather than matching each response’s scale individually.

The paper also states a clear limitation. Because the coefficient is fixed rather than adaptive, SPO cannot precisely counteract response-dependent scale variation, which is the root cause identified in the asynchronous setting. This is the sense in which ASymPO is presented as a more direct correction of the same failure mode.

5. Training procedure, hyperparameters, and empirical behavior

The algorithmic procedure is straightforward. Rollout workers sample $$p_{g,i}(\theta)=\pi_\theta(a_{g,i}\mid x,a_{g,<i}),\quad S_{\theta,g}=-\frac{1}{m_g}\sum_{i=1}^{m_g}\log p_{g,i}(\theta).$$7 responses per prompt and compute scalar rewards $$p_{g,i}(\theta)=\pi_\theta(a_{g,i}\mid x,a_{g,<i}),\quad S_{\theta,g}=-\frac{1}{m_g}\sum_{i=1}^{m_g}\log p_{g,i}(\theta).$$8. The learner forms group-relative advantages

$$p_{g,i}(\theta)=\pi_\theta(a_{g,i}\mid x,a_{g,<i}),\quad S_{\theta,g}=-\frac{1}{m_g}\sum_{i=1}^{m_g}\log p_{g,i}(\theta).$$9

recomputes $r_g$0 and $r_g$1 under the current policy, applies the SPO scaling rule, computes $r_g$2, and updates $r_g$3 with the chosen optimizer (Liu et al., 2 Jun 2026).

In the reported experiments, the negative scaling factor is $r_g$4, following prior intuition from RIFT. Sensitivity is explicit: $r_g$5 reduces to the naive loss and collapses in this setting, whereas $r_g$6 removes negatives entirely and becomes RFT-style positive-only training, which underperforms because it does not suppress low-reward responses. The group size is $r_g$7. The implementation uses VeRL with ppo mini-batch size $r_g$8, train batch size $r_g$9, staleness threshold $$A_g=r_g-\frac{1}{G}\sum_{j=1}^G r_j,\qquad \sum_g A_g=0,$$0, learning rate $$A_g=r_g-\frac{1}{G}\sum_{j=1}^G r_j,\qquad \sum_g A_g=0,$$1, training for $$A_g=r_g-\frac{1}{G}\sum_{j=1}^G r_j,\qquad \sum_g A_g=0,$$2 epochs, maximum prompt length $$A_g=r_g-\frac{1}{G}\sum_{j=1}^G r_j,\qquad \sum_g A_g=0,$$3 tokens, and maximum response length $$A_g=r_g-\frac{1}{G}\sum_{j=1}^G r_j,\qquad \sum_g A_g=0,$$4 tokens. The models are Qwen3-1.7B-Base, Qwen3-4B-Base, and LLaMA-3.2-3B-Instruct. Training uses a randomly sampled $$A_g=r_g-\frac{1}{G}\sum_{j=1}^G r_j,\qquad \sum_g A_g=0,$$5k subset of MATH, with a supplementary experiment on a $$A_g=r_g-\frac{1}{G}\sum_{j=1}^G r_j,\qquad \sum_g A_g=0,$$6k subset of DAPO-Math-17K (Liu et al., 2 Jun 2026).

The stability findings are categorical. The naive current-policy loss and GPG collapsed across all three model families, yielding no usable final checkpoints. GRPO, SPO, and ASymPO maintained stable training rewards. Final performance is competitive with behavior-corrected baselines, but not uniformly superior. On Qwen3-1.7B-Base trained on MATH 4k, GRPO reports mean@8 $$A_g=r_g-\frac{1}{G}\sum_{j=1}^G r_j,\qquad \sum_g A_g=0,$$7 and pass@8 $$A_g=r_g-\frac{1}{G}\sum_{j=1}^G r_j,\qquad \sum_g A_g=0,$$8, while SPO reports mean@8 $$A_g=r_g-\frac{1}{G}\sum_{j=1}^G r_j,\qquad \sum_g A_g=0,$$9 and pass@8 $$\mathcal{L}_{C}(\theta) =-\frac{1}{G}\sum_{g=1}^G\frac{1}{m_g}\sum_{i=1}^{m_g}A_g C_g\log p_{g,i}(\theta) =\frac{1}{G}\sum_{g=1}^G A_g C_g S_{\theta,g}.$$0. On LLaMA-3.2-3B-Instruct, GRPO reports mean@8 $$\mathcal{L}_{C}(\theta) =-\frac{1}{G}\sum_{g=1}^G\frac{1}{m_g}\sum_{i=1}^{m_g}A_g C_g\log p_{g,i}(\theta) =\frac{1}{G}\sum_{g=1}^G A_g C_g S_{\theta,g}.$$1 and pass@8 $$\mathcal{L}_{C}(\theta) =-\frac{1}{G}\sum_{g=1}^G\frac{1}{m_g}\sum_{i=1}^{m_g}A_g C_g\log p_{g,i}(\theta) =\frac{1}{G}\sum_{g=1}^G A_g C_g S_{\theta,g}.$$2, while SPO reports mean@8 $$\mathcal{L}_{C}(\theta) =-\frac{1}{G}\sum_{g=1}^G\frac{1}{m_g}\sum_{i=1}^{m_g}A_g C_g\log p_{g,i}(\theta) =\frac{1}{G}\sum_{g=1}^G A_g C_g S_{\theta,g}.$$3 and pass@8 $$\mathcal{L}_{C}(\theta) =-\frac{1}{G}\sum_{g=1}^G\frac{1}{m_g}\sum_{i=1}^{m_g}A_g C_g\log p_{g,i}(\theta) =\frac{1}{G}\sum_{g=1}^G A_g C_g S_{\theta,g}.$$4. On Qwen3-4B-Base, GRPO reports mean@8 $$\mathcal{L}_{C}(\theta) =-\frac{1}{G}\sum_{g=1}^G\frac{1}{m_g}\sum_{i=1}^{m_g}A_g C_g\log p_{g,i}(\theta) =\frac{1}{G}\sum_{g=1}^G A_g C_g S_{\theta,g}.$$5 and pass@8 $$\mathcal{L}_{C}(\theta) =-\frac{1}{G}\sum_{g=1}^G\frac{1}{m_g}\sum_{i=1}^{m_g}A_g C_g\log p_{g,i}(\theta) =\frac{1}{G}\sum_{g=1}^G A_g C_g S_{\theta,g}.$$6, while SPO reports mean@8 $$\mathcal{L}_{C}(\theta) =-\frac{1}{G}\sum_{g=1}^G\frac{1}{m_g}\sum_{i=1}^{m_g}A_g C_g\log p_{g,i}(\theta) =\frac{1}{G}\sum_{g=1}^G A_g C_g S_{\theta,g}.$$7 and pass@8 $$\mathcal{L}_{C}(\theta) =-\frac{1}{G}\sum_{g=1}^G\frac{1}{m_g}\sum_{i=1}^{m_g}A_g C_g\log p_{g,i}(\theta) =\frac{1}{G}\sum_{g=1}^G A_g C_g S_{\theta,g}.$$8. In the supplementary Qwen3-1.7B-Base experiment on DAPO-17K 4k, SPO reports mean@8 $$\mathcal{L}_{C}(\theta) =-\frac{1}{G}\sum_{g=1}^G\frac{1}{m_g}\sum_{i=1}^{m_g}A_g C_g\log p_{g,i}(\theta) =\frac{1}{G}\sum_{g=1}^G A_g C_g S_{\theta,g}.$$9 and pass@8 $\pi_\theta$0, exceeding the GRPO numbers of mean@8 $\pi_\theta$1 and pass@8 $\pi_\theta$2 (Liu et al., 2 Jun 2026).

The empirical takeaway stated in the paper is correspondingly narrow. SPO eliminates the collapse of naive current-policy-only training and produces competitive final performance relative to GRPO, sometimes trailing slightly and sometimes comparable, depending on model and metric. Its role is therefore best understood as a simple stabilizing baseline for behavior-free asynchronous training rather than as a universally dominant replacement for behavior-corrected objectives.

6. Broader usage of “scaled” in SPO and relation to Sequential Monte Carlo Policy Optimisation

A distinct usage of the phrase appears in “SPO: Sequential Monte Carlo Policy Optimisation,” a model-based reinforcement learning algorithm grounded within the Expectation Maximisation framework (Macfarlane et al., 2024). There, the method is a policy-iteration scheme in which SMC planning produces expert targets and these are distilled into a parametric policy. The key idea is to replace sequential tree search with a particle-based search that is trivially parallelizable on modern accelerators, scales to large budgets, and works in both discrete and continuous action spaces without modifications.

In that line of work, planning is cast through control as inference and an EM-style decomposition. The E-step approximates the posterior over high-reward trajectories via SMC, and the M-step fits the policy to the induced improved action distribution through forward-KL projection. The paper’s clarification is explicit: “SPO as presented here is the Sequential Monte Carlo Policy Optimization operator embedded in Expert Iteration: SMC planning produces an improved policy $\pi_\theta$3, and a scaled policy optimization step projects $\pi_\theta$4 onto $\pi_\theta$5 via forward KL.” The “scaled” aspect refers to the ability to scale search computation efficiently—particles and horizon—on accelerators, yielding favorable wall-clock characteristics relative to sequential search methods such as MCTS (Macfarlane et al., 2024).

That SMC-based SPO is architecturally different from the asynchronous current-policy-only SPO of ASymPO. Its search runs $\pi_\theta$6 particle rollouts in parallel for horizon $\pi_\theta$7, uses a soft-advantage weighting scheme, extracts the improved policy as the marginal over first actions, and trains the policy by minimizing either forward KL in discrete spaces or weighted negative log-likelihood in continuous spaces. The reported main hyperparameters are $\pi_\theta$8, $\pi_\theta$9, resampling period $S_{\theta,g}$0, and resampling temperature $S_{\theta,g}$1. The paper emphasizes $S_{\theta,g}$2 time per decision, $S_{\theta,g}$3 memory for current states, weights, and initial actions, and strong TPU utilization through batched tensor operations (Macfarlane et al., 2024).

The coexistence of these two usages is significant for terminology. In one case, Scaled Policy Optimization is an explicit current-policy-only asynchronous RL objective with fixed asymmetric negative scaling (Liu et al., 2 Jun 2026). In the other, “scaled” refers to the efficient scaling properties of SMC-based planning and policy distillation inside a model-based Expert Iteration framework (Macfarlane et al., 2024). The shared acronym therefore masks materially different assumptions, objectives, and update rules.