Score Centering Stabilizes Off-policy Reinforcement Learning

Abstract: Reinforcement learning (RL) of LLMs is notoriously sensitive to small differences between training and inference engines, often referred to as the training-inference mismatch (TIM). However, completely eliminating TIM is impractical, as it would come at a major cost to rollout efficiency. In this paper, we show that the instability of RL under TIM is primarily caused by drift: a persistent bias between training and inference engines that accumulates with every training step. We derive an additive "score centering" correction term that stabilizes RL under TIM by canceling drift. When training models from 0.6B to 30B parameters, score centering alone matches or outperforms methods based on importance sampling under quantization, with the gap growing as the mismatch becomes more severe. Because the correction is additive, score centering also composes with importance sampling -- their composition outperforms pure importance-sampling baselines in our staleness experiments.

• The paper introduces score centering, a method that eliminates the drift term in policy-gradient estimators, stabilizing off-policy reinforcement learning during training-inference mismatch (TIM).
• Score centering significantly outperforms importance sampling and other baselines under severe quantization and quantized KB-cache settings.
• The methodology is computationally efficient, using top-k approximation to avoid storing full vocabularies while demonstrating comparable performance to full-vocabulary implementations, making it scalable for practical applications.

Problem setting and central claim

“Score Centering Stabilizes Off-policy Reinforcement Learning” (2609.20807) studies the training-inference mismatch (TIM) that arises when LLM rollouts are generated by an inference engine with policy $q_\theta$, while gradients are computed by a training engine representing policy $p_\theta$. Although the two engines are nominally instances of the same model, differences in numerical precision, kernel execution, batching, quantization, code paths, routing, and checkpoint freshness can make $q_\theta \ne p_\theta$.

The paper’s central claim is that the principal source of instability under TIM is not mismatch per se, but an accumulating drift term in the policy-gradient estimator. This drift causes the trainer to distill toward the sampler, even when the reward contains no discriminative information. Because the sampler is subsequently refreshed from the trainer, the resulting feedback loop compounds the bias over training. The authors derive an additive correction, score centering (SC), that cancels this drift exactly at each prefix without using importance ratios. Across Qwen3 models from 0.6B to 30B parameters, SC is competitive with or superior to importance-sampling corrections under severe quantization and staleness.

Training-inference mismatch and the source of instability

For an on-policy policy-gradient method, the expected update is the reward-weighted score,

$$\mathbb{E}_{p_\theta}\left[R \nabla_\theta \log p_\theta(y)\right].$$

When rollouts are sampled from $q_\theta$ but scored under $p_\theta$, this identity no longer directly applies. Exact correction through importance sampling is possible, but the likelihood ratio $p_\theta(y)/q_\theta(y)$ has high variance, especially for autoregressive sequences and rare tokens. Practical methods therefore clip, mask, or otherwise truncate the ratio, introducing bias.

The paper isolates a different failure mechanism by conditioning on a prefix and considering one next-token score. Let $s_{y_t}=\nabla_\theta\log p_\theta(y_t)$ and let $\bar{s}$ denote the expected score under the sampler. The off-policy update decomposes as

$$\mathbb{E}_q[R s_{y_t}] = \mathbb{E}_q[R]\bar{s} + \operatorname{Cov}_q(R,s_{y_t}).$$

The covariance is the reward-dependent signal. The first term is the drift. It depends on the reward only through its conditional mean and therefore does not identify which token contributed to success. Since $p_\theta$0 is the negative gradient of the cross-entropy from the sampler distribution to the trainer distribution, this term acts as distillation toward $p_\theta$1.

On-policy, $p_\theta$2 because the expected score under $p_\theta$3 vanishes. Under TIM, however, $p_\theta$4 is generally nonzero. A sampler that is quantized, stale, or numerically inconsistent therefore becomes a moving teacher. The trainer is pushed toward that biased teacher and is then copied back to the sampler, allowing the mismatch-induced error to accumulate. This explanation distinguishes online RL from ordinary offline distillation: distillation toward a fixed teacher can converge, whereas online distillation toward a periodically refreshed and systematically biased copy can create a positive feedback loop.

The experiments also separate two instability mechanisms that are often conflated. In the authors’ Countdown experiments, offline training is stable primarily with nonnegative $p_\theta$5 rewards, whereas online training under TIM is least stable with those same rewards. Group-centered rewards containing both positive and negative values are more stable online. The result supports the paper’s assertion that offline negative-reward instability and online TIM-induced drift have different causes. The paper does not further analyze the former, which it associates with unbounded negative-log-probability updates and distribution sharpening.

Score centering

Score centering replaces each token score with

$p_\theta$6

where the expectation $p_\theta$7 is taken under the sampler’s next-token distribution. Consequently,

$p_\theta$8

and the expected update becomes only the covariance term,

$p_\theta$9

This removes the drift exactly, including under a constant reward. The remaining difference from the ideal on-policy update is that the covariance is estimated under $q_\theta \ne p_\theta$0 rather than $q_\theta \ne p_\theta$1. Thus SC is not an exact replacement for importance sampling in every off-policy regime; it specifically removes the additive bias caused by the nonzero expected score.

The distinction from conventional baselines and control variates is important. Reward baselines exploit the zero-mean score identity on-policy to reduce variance. SC instead subtracts the expected score itself to correct the mean of the off-policy update. Unlike importance sampling, it is additive and deterministic conditional on the prefix. It introduces neither sampled multiplicative ratios nor clipping thresholds.

SC also composes naturally with importance sampling. If an importance method assigns token weight $q_\theta \ne p_\theta$2, SC subtracts the expected weighted score,

$q_\theta \ne p_\theta$3

This composition preserves exact cancellation of drift for the weighted estimator. The paper therefore treats SC and importance sampling as orthogonal: SC controls the additive mismatch-induced bias, while importance weighting partially corrects the distribution under which the reward-score covariance is measured.

Efficient implementation with top-$q_\theta \ne p_\theta$4 log probabilities

Exact SC would require the full sampler vocabulary distribution for every generated token, which is impractical for large vocabularies and long rollouts. The implementation stores only the sampler’s top-$q_\theta \ne p_\theta$5 log probabilities and reconstructs the tail using the trainer distribution, rescaled to match the sampler’s tail mass.

For head tokens $q_\theta \ne p_\theta$6, the approximate sampler distribution retains the sampler probabilities. For the tail, it uses $q_\theta \ne p_\theta$7, where $q_\theta \ne p_\theta$8 is the ratio between sampler and trainer tail mass. Because the trainer’s full expected score is zero, the tail contribution can be reduced to a correction over the top-$q_\theta \ne p_\theta$9 tokens:

$$\mathbb{E}_{p_\theta}\left[R \nabla_\theta \log p_\theta(y)\right].$$0

This yields a scalar loss implementable with standard autodifferentiation and stop-gradient operations. In the reported experiments, $$\mathbb{E}_{p_\theta}\left[R \nabla_\theta \log p_\theta(y)\right].$$1 adds negligible wall-clock cost—within approximately 1% of baseline methods—and $$\mathbb{E}_{p_\theta}\left[R \nabla_\theta \log p_\theta(y)\right].$$2 performs comparably to full-vocabulary centering.

Figure 1: Top-$$\mathbb{E}_{p_\theta}\left[R \nabla_\theta \log p_\theta(y)\right].$$3 score centering with $$\mathbb{E}_{p_\theta}\left[R \nabla_\theta \log p_\theta(y)\right].$$4 or $$\mathbb{E}_{p_\theta}\left[R \nabla_\theta \log p_\theta(y)\right].$$5 matches full-vocabulary score centering across the tested settings.

The top-$$\mathbb{E}_{p_\theta}\left[R \nabla_\theta \log p_\theta(y)\right].$$6 approximation remains effective even in the most severe reported 30B setting, where the sampler’s top-128 mass is less accurately concentrated because of INT4 KV-cache quantization. This result supports the practical claim that SC does not require storing prohibitively large full-vocabulary distributions. It does not, however, establish that the same tail model will remain accurate for arbitrary vocabularies, tokenizers, or substantially more pathological sampler distributions.

Experimental design

The experiments use a common REINFORCE objective with group-centered rewards and compare correction methods in isolation. This design is methodologically useful because several baselines normally bundle correction rules with other algorithmic components such as dynamic sampling, response-length penalties, or policy-version clipping. The paper instead holds the sampler, trainer, optimizer, advantage estimates, and one-step-per-batch update fixed, applying each correction to the shared objective.

The principal models and tasks are Qwen3-0.6B-Instruct on Countdown and Qwen3-30B-A3B-Base on the mathematical subset of INTELLECT-2. TIM is induced through three mechanisms: synthetic Gaussian sampler-weight perturbations, sampler quantization, and sampler staleness. The perturbations are intentionally severe to produce separation between methods within feasible compute budgets. The authors explicitly interpret these short, severe-mismatch experiments as a proxy for longer training under milder mismatch, rather than as a direct reproduction of ordinary deployment conditions.

Synthetic mismatch and drift accumulation

Under fixed Gaussian perturbations to sampler weights, larger mismatch causes earlier collapse. SC, truncated importance sampling (TIS), masked importance sampling (MIS), and their compositions are the strongest methods at lower noise levels. Under the largest perturbation, only SC and SC composed with TIS or MIS remain stable.

Figure 2: Increasing sampler-weight noise causes earlier collapse; SC alone or composed with TIS/MIS remains stable under the largest noise.

The timing of collapse is consistent with the drift hypothesis. For example, DPPO collapses at approximately steps 160, 80, and 20 as the noise scale increases, while TIS is stable at the smallest noise but collapses at approximately steps 180 and 40 at larger scales. The numerical pattern indicates that mismatch severity controls the rate at which harmful bias accumulates rather than merely adding a fixed perturbation to each update. This is a stronger diagnostic than comparing final performance alone: methods differ in how long they preserve stable dynamics before drift becomes dominant.

Quantization and staleness

The realistic TIM experiments use an independently quantized sampler with a BF16 trainer and a sampler refreshed only every 64 steps. Under sampler quantization, SC performs best either alone or when composed with TIS/MIS. Under staleness, the compositions dominate vanilla SC.

Figure 3: SC is strongest under severe sampler quantization, while SC composed with TIS or MIS is strongest under substantial update staleness.

This difference follows directly from the estimator decomposition. SC cancels the sampler-induced drift, but its residual covariance is still measured under the sampler distribution. When the trainer moves substantially during a 64-step synchronization interval, importance weighting partially corrects that distributional discrepancy, after which SC removes the remaining additive drift. The result yields a concrete operational recommendation: SC alone is particularly appropriate for numerical mismatch such as quantization, whereas SC combined with an IS correction is preferable when the dominant mismatch is checkpoint staleness.

PPO and DAPO survive the staleness condition but collapse under quantization and synthetic weight noise. The paper attributes this asymmetry to the design of their clipping regions: policy-ratio clipping addresses ratios caused by policy movement, but not necessarily systematic numerical discrepancies that do not resemble ordinary policy updates. This is a substantive claim because it challenges the assumption that standard PPO-style clipping is a general-purpose safeguard against all trainer-sampler discrepancies.

Scaling to a 30B mixture-of-experts model

The largest experiment trains Qwen3-30B-A3B-Base on INTELLECT-2 mathematics under progressively more severe sampler quantization. The sampler and trainer may also disagree in mixture-of-experts routing because router indices are not replayed, creating an additional source of TIM.

With an FP8 sampler, uncorrected policy gradient remains stable and reaches 58% training accuracy. With FP4 KV-cache quantization, vanilla policy gradient collapses within 200 steps and MIS collapses late in training, whereas SC reaches 52% and TIS reaches 51% while remaining stable. Under the more severe INT8 weight/activation and INT4 KV-cache configuration, SC reaches 30%, TIS reaches 12%, and all other methods finish below 5%.

Figure 4: Under progressively stronger sampler quantization, SC maintains stable training as vanilla policy gradient and several importance-sampling methods collapse.

These results provide the paper’s strongest empirical evidence for the proposed mechanism. SC does not merely match clipped importance sampling under mild mismatch; its advantage increases as mismatch becomes severe. The 30B result also indicates that the correction remains usable in a large MoE system with routing disagreement, although the limited seed counts in the most expensive experiments weaken the statistical strength of method-by-method comparisons.

Relationship between reward structure and online instability

The offline-versus-online comparison on Qwen3-1.7B provides an important qualification to the main narrative. With offline data, $$\mathbb{E}_{p_\theta}\left[R \nabla_\theta \log p_\theta(y)\right].$$7 rewards behave like positive-sample distillation and are stable, while reward modes containing negative values can become unstable. In online training under TIM, the ordering reverses: $$\mathbb{E}_{p_\theta}\left[R \nabla_\theta \log p_\theta(y)\right].$$8 rewards are least stable, and group-centered rewards are comparatively more robust.

Figure 5: Reward sign and online staleness interact differently: offline training favors $$\mathbb{E}_{p_\theta}\left[R \nabla_\theta \log p_\theta(y)\right].$$9 rewards, whereas online training under TIM is least stable with $q_\theta$0 rewards.

The implication is that reward centering should not be interpreted as eliminating drift. Group centering makes advantages sum to zero across rollouts for a prompt, but drift is a prefix-level quantity. A prefix can have positive or negative expected advantage even when the group mean is zero, so the conditional expected score remains nonzero. Group centering reduces the magnitude of drift but does not cancel it. SC addresses this residual directly.

Limitations and open questions

The paper’s principal limitation is that SC does not fully correct the off-policy distribution. After drift cancellation, the covariance is still taken under $q_\theta$1 rather than $q_\theta$2. The staleness experiments demonstrate the practical consequence: SC performs better when combined with TIS or MIS. Thus the method’s strongest claim is specifically about removing drift, not about producing an unbiased estimator of the ideal on-policy policy gradient under arbitrary mismatch.

The experimental regime is also deliberately nonstandard. The headline comparisons use severe quantization, fixed sampler-weight noise, or a sampler updated only every 64 steps, often with short sequences. These conditions are intended to approximate the cumulative effect of milder mismatch over longer runs, but the equivalence is not established. The paper leaves open how SC compares under realistic asynchronous systems with continuously batched inference, longer agentic trajectories, mixed sources of mismatch, and smaller per-step numerical errors.

The top-$q_\theta$3 implementation relies on a tail model derived from the trainer distribution. Although $q_\theta$4 and $q_\theta$5 match full SC in all reported settings, the approximation has not been stress-tested across broader distributions or tasks. The largest-scale comparisons also use uneven seed counts, including single-seed results for many baselines, and the MoE experiments do not replay routing decisions. These factors limit the precision with which the reported performance gaps can be attributed solely to the correction rule.

Finally, the paper does not investigate the separate instability of offline negative-reward training in depth. Its explanation in terms of unbounded negative log-probability updates is plausible within the presented setup, but the relationship between that mechanism, SC, reward normalization, and broader off-policy objectives remains unresolved.

Conclusion

The paper identifies a specific failure mode in off-policy LLM RL: trainer-sampler mismatch introduces a nonzero expected score, and the resulting drift distills the trainer toward a biased, moving sampler. Score centering removes this drift through an additive expected-score subtraction that is exact at the prefix level and practical with top-$q_\theta$6 sampler log probabilities.

Across controlled mismatch, severe quantization, sampler staleness, and a 30B MoE model, SC is consistently competitive with importance sampling and substantially more robust in the most severe quantization regimes. Its residual distributional mismatch explains why composition with TIS or MIS is advantageous under staleness. The resulting contribution is both diagnostic and algorithmic: it separates drift cancellation from importance-ratio correction and provides a low-overhead mechanism for stabilizing RL when exact trainer-inference equivalence is impractical.