---
title: Scaled Policy Optimization in Reinforcement Learning
url: https://www.emergentmind.com/topics/scaled-policy-optimization-spo
type: topic
---

# Scaled Policy Optimization in Reinforcement Learning

Scaled Policy Optimization (SPO) is an overloaded term in contemporary reinforcement learning. In the most explicit usage, it denotes a current-policy-only, group-relative reinforcement learning objective for asynchronous large-language-model post-training that stabilizes learning without behavior-policy probabilities by asymmetrically downweighting negative-advantage responses with a fixed coefficient [2606.03070]. A separate usage associates “scaled” with the efficient accelerator scaling properties of “SPO: Sequential Monte Carlo Policy Optimisation,” where Sequential Monte Carlo (SMC) planning produces improved action targets that are then distilled into a parametric policy through forward-KL projection inside an Expert Iteration loop [2402.07963]. The term therefore requires careful disambiguation.

## 1. Terminology and scope

In the asynchronous post-training literature, Scaled Policy Optimization is defined as a fixed negative-scaling baseline introduced alongside ASymPO. Its purpose is to address a specific failure mode—scale imbalance—when stale responses are evaluated only under the current policy and behavior-policy probabilities are unavailable [2606.03070]. In that setting, SPO is not a generic synonym for PPO-style optimization, nor is it a trust-region method in the classical TRPO sense; it is a current-policy-only objective with asymmetric response-level scaling.

The acronym is also used for unrelated methods. “SPO” denotes “Sequential Monte Carlo Policy Optimisation” in model-based reinforcement learning [2402.07963], “Simple Policy Optimization” in first-order KL-aware policy optimization [2401.16025], “Segment Policy Optimization” in reinforcement learning for large language models [2505.23564], “Soft Policy Optimization” in off-policy sequence-model training [2503.05453], “Single-stream Policy Optimization” in group-free RLVR [2509.13232], and “Sinkhorn Policy Optimization” in metric-aware trust-region policy optimization [2306.14133]. This suggests that “Scaled Policy Optimization” is not a standardized field-wide label, but a paper-specific designation whose meaning depends on context.

## 2. Group-relative formulation in asynchronous LLM post-training

The Scaled Policy Optimization objective introduced in asynchronous post-training is defined in a group-relative setting. For a prompt $x$ and response $y_g=(a_{g,1},\dots,a_{g,m_g})$, the current-policy token probabilities and average token negative log-probability are

$$
p_{g,i}(\theta)=\pi_\theta(a_{g,i}\mid x,a_{g,<i}),\quad
S_{\theta,g}=-\frac{1}{m_g}\sum_{i=1}^{m_g}\log p_{g,i}(\theta).
$$

With group-relative rewards $r_g$ and zero-sum advantages

$$
A_g=r_g-\frac{1}{G}\sum_{j=1}^G r_j,\qquad \sum_g A_g=0,
$$

the paper considers the general scaled objective

$$
\mathcal{L}_{C}(\theta)
=-\frac{1}{G}\sum_{g=1}^G\frac{1}{m_g}\sum_{i=1}^{m_g}A_g C_g\log p_{g,i}(\theta)
=\frac{1}{G}\sum_{g=1}^G A_g C_g S_{\theta,g}.
$$

This formulation is explicitly current-policy-only. The learner requires sampled tokens and scalar rewards per response, then recomputes all token log-probabilities under $\pi_\theta$. It does not require behavior-policy log-probabilities, policy-version tags, or importance ratios. No special token alignment or numerical consistency with the rollout system is needed beyond standard deterministic tokenization and model evaluation on the learner [2606.03070].

The resulting objective is tailored to asynchronous pipelines in which rollout workers may sample responses under stale policies while the learner trains a newer current policy. In such pipelines, the relevant scale is the current-policy average token negative log-probability $S_{\theta,g}$, which becomes the quantity through which stale-response mismatch manifests.

## 3. Fixed asymmetric negative scaling

SPO instantiates the general scaled objective with a fixed asymmetric rule:

$$
C_g^{\mathrm{SPO}}=
\begin{cases}
1, & A_g\geq 0,\\
\alpha, & A_g<0,
\end{cases}
$$

where $\alpha\in(0,1)$ is a hyperparameter. Equivalently, if $\tilde{A}_g=A_g$ for $A_g\geq 0$ and $\tilde{A}_g=\alpha A_g$ for $A_g<0$, then SPO minimizes

$$
\mathcal{L}_{\mathrm{SPO}}(\theta)
=-\frac{1}{G}\sum_{g=1}^G\frac{1}{m_g}\sum_{i=1}^{m_g}\tilde{A}_g\log p_{g,i}(\theta).
$$

The asymmetry is the essential design choice. Positive-advantage terms are unscaled, whereas negative-advantage terms are downweighted by the fixed factor $\alpha<1$. The update direction is preserved—positive responses are reinforced and negative responses are suppressed—but negative updates are made weaker [2606.03070].

Because $C_g^{\mathrm{SPO}}$ does not depend on $\theta$, the gradient has the simple form

$$
\nabla_\theta \mathcal{L}_{\mathrm{SPO}}(\theta)
=\frac{1}{G}\sum_{g=1}^G A_g C_g^{\mathrm{SPO}}\nabla_\theta S_{\theta,g},
\qquad
\nabla_\theta S_{\theta,g}
=-\frac{1}{m_g}\sum_{i=1}^{m_g}\nabla_\theta \log p_{g,i}(\theta).
$$

This preserves the rollout–learner interface of current-policy-only training while introducing a deterministic response-level correction. Relative to adaptive normalization schemes, the rule is deliberately minimal: one scalar $\alpha$ governs all negative responses.

## 4. Scale imbalance and comparison with related objectives

The motivation for SPO is a failure mode identified in asynchronous training without behavior information. If stale responses are evaluated under the current policy, the naive current-policy-only loss

$$
\mathcal{L}_{\mathrm{naive}}(\theta)=\frac{1}{G}\sum_{g=1}^G A_g S_{\theta,g}
$$

can become unstable when stale negative-advantage responses have much larger current negative log-probability than positive responses. In the notation of the paper, instability appears when

$$
\sum_{g\in\mathcal{N}} |A_g| S_{\theta,g} \gg \sum_{g\in\mathcal{P}} A_g S_{\theta,g},
$$

even though $\sum_g A_g=0$ [2606.03070]. Zero-sum advantages therefore do not imply balanced loss contributions once responses are evaluated at different current-policy scales.

SPO addresses this by shrinking the negative side uniformly:

$$
\mathcal{L}_{\mathrm{SPO}}(\theta)
=\frac{1}{G}\sum_{g\in\mathcal{P}}A_g S_{\theta,g}
-\frac{1}{G}\sum_{g\in\mathcal{N}}\alpha |A_g| S_{\theta,g}.
$$

The paper’s interpretation is that fixed negative scaling reduces the tendency of stale negatives to dominate. If the naive negative-to-positive contribution ratio is $R_{\mathrm{naive}}=\bar{S}_{\mathcal{N}}/\bar{S}_{\mathcal{P}}$, SPO scales it to approximately $\alpha\,\bar{S}_{\mathcal{N}}/\bar{S}_{\mathcal{P}}$, bringing it closer to unity when negatives are over-scaled [2606.03070].

This mechanism differs from both behavior-corrected and adaptive current-policy-only approaches. Behavior-corrected methods such as PPO- or GRPO-style objectives use behavior-policy probabilities, importance ratios, and clipping, providing a common reference scale and bounding drift. ASymPO, introduced in the same paper, instead normalizes each response by its own current scale,

$$
C_g^{\mathrm{AsymPO}}=\frac{1}{\operatorname{sg}(S_{\theta,g})},
$$

so that the forward loss exactly inherits the group’s zero-sum balance, while the gradient normalizes each response’s update by its own current scale [2606.03070]. SPO does not preserve exact response-level zero-sum balance. It approximates balance by uniformly shrinking the negative side rather than matching each response’s scale individually.

The paper also states a clear limitation. Because the coefficient is fixed rather than adaptive, SPO cannot precisely counteract response-dependent scale variation, which is the root cause identified in the asynchronous setting. This is the sense in which ASymPO is presented as a more direct correction of the same failure mode.

## 5. Training procedure, hyperparameters, and empirical behavior

The algorithmic procedure is straightforward. Rollout workers sample $G$ responses per prompt and compute scalar rewards $r_g$. The learner forms group-relative advantages

$$
A_g=r_g-\frac{1}{G}\sum_{j=1}^G r_j,
$$

recomputes $p_{g,i}(\theta)$ and $S_{\theta,g}$ under the current policy, applies the SPO scaling rule, computes $\mathcal{L}_{\mathrm{SPO}}(\theta)$, and updates $\theta$ with the chosen optimizer [2606.03070].

In the reported experiments, the negative scaling factor is $\alpha=0.2$, following prior intuition from RIFT. Sensitivity is explicit: $\alpha=1$ reduces to the naive loss and collapses in this setting, whereas $\alpha=0$ removes negatives entirely and becomes RFT-style positive-only training, which underperforms because it does not suppress low-reward responses. The group size is $G=8$. The implementation uses VeRL with ppo mini-batch size $32$, train batch size $512$, staleness threshold $0.5$, learning rate $2\times 10^{-6}$, training for $3$ epochs, maximum prompt length $1024$ tokens, and maximum response length $4096$ tokens. The models are Qwen3-1.7B-Base, Qwen3-4B-Base, and LLaMA-3.2-3B-Instruct. Training uses a randomly sampled $4$k subset of MATH, with a supplementary experiment on a $4$k subset of DAPO-Math-17K [2606.03070].

The stability findings are categorical. The naive current-policy loss and GPG collapsed across all three model families, yielding no usable final checkpoints. GRPO, SPO, and ASymPO maintained stable training rewards. Final performance is competitive with behavior-corrected baselines, but not uniformly superior. On Qwen3-1.7B-Base trained on MATH 4k, GRPO reports mean@8 $37.45$ and pass@8 $54.56$, while SPO reports mean@8 $36.41$ and pass@8 $53.92$. On LLaMA-3.2-3B-Instruct, GRPO reports mean@8 $30.84$ and pass@8 $46.09$, while SPO reports mean@8 $29.61$ and pass@8 $45.25$. On Qwen3-4B-Base, GRPO reports mean@8 $45.52$ and pass@8 $60.90$, while SPO reports mean@8 $45.45$ and pass@8 $58.06$. In the supplementary Qwen3-1.7B-Base experiment on DAPO-17K 4k, SPO reports mean@8 $38.13$ and pass@8 $57.53$, exceeding the GRPO numbers of mean@8 $37.18$ and pass@8 $54.70$ [2606.03070].

The empirical takeaway stated in the paper is correspondingly narrow. SPO eliminates the collapse of naive current-policy-only training and produces competitive final performance relative to GRPO, sometimes trailing slightly and sometimes comparable, depending on model and metric. Its role is therefore best understood as a simple stabilizing baseline for behavior-free asynchronous training rather than as a universally dominant replacement for behavior-corrected objectives.

## 6. Broader usage of “scaled” in SPO and relation to Sequential Monte Carlo Policy Optimisation

A distinct usage of the phrase appears in “SPO: Sequential Monte Carlo Policy Optimisation,” a model-based reinforcement learning algorithm grounded within the Expectation Maximisation framework [2402.07963]. There, the method is a policy-iteration scheme in which SMC planning produces expert targets and these are distilled into a parametric policy. The key idea is to replace sequential tree search with a particle-based search that is trivially parallelizable on modern accelerators, scales to large budgets, and works in both discrete and continuous action spaces without modifications.

In that line of work, planning is cast through control as inference and an EM-style decomposition. The E-step approximates the posterior over high-reward trajectories via SMC, and the M-step fits the policy to the induced improved action distribution through forward-KL projection. The paper’s clarification is explicit: “SPO as presented here is the Sequential Monte Carlo Policy Optimization operator embedded in Expert Iteration: SMC planning produces an improved policy $\pi'$, and a scaled policy optimization step projects $\pi_\theta$ onto $\pi'$ via forward KL.” The “scaled” aspect refers to the ability to scale search computation efficiently—particles and horizon—on accelerators, yielding favorable wall-clock characteristics relative to sequential search methods such as MCTS [2402.07963].

That SMC-based SPO is architecturally different from the asynchronous current-policy-only SPO of ASymPO. Its search runs $N$ particle rollouts in parallel for horizon $h$, uses a soft-advantage weighting scheme, extracts the improved policy as the marginal over first actions, and trains the policy by minimizing either forward KL in discrete spaces or weighted negative log-likelihood in continuous spaces. The reported main hyperparameters are $N=16$, $h=4$, resampling period $p\in\{2,4\}$, and resampling temperature $\tau_s=0.1$. The paper emphasizes $O(Nh)$ time per decision, $O(N)$ memory for current states, weights, and initial actions, and strong TPU utilization through batched tensor operations [2402.07963].

The coexistence of these two usages is significant for terminology. In one case, Scaled Policy Optimization is an explicit current-policy-only asynchronous RL objective with fixed asymmetric negative scaling [2606.03070]. In the other, “scaled” refers to the efficient scaling properties of SMC-based planning and policy distillation inside a model-based Expert Iteration framework [2402.07963]. The shared acronym therefore masks materially different assumptions, objectives, and update rules.

Source: https://www.emergentmind.com/topics/scaled-policy-optimization-spo