Length Reweighting Error in Sequence-Level RL

• Length Reweighting Error (LRE) is a measure of the distortion in the effective length mixture due to fixed clipping in sequence-level reinforcement learning.
• LRE quantifies how the length-conditional acceptance rate varies, systematically biasing gradients by over-emphasizing short responses over long ones.
• Using a length-fair clipping band in FSPO, the approach minimizes LRE to ensure directional alignment between the clipped update and the true gradient, leading to more stable training.

Searching arXiv for papers on “Length Reweighting Error”, FSPO, and related length-bias work. In sequence-level reinforcement learning for LLMs, Length Reweighting Error (LRE) denotes a clipping-induced distortion of the effective length mixture in the update rule. In the FSPO formulation, LRE measures how much the length-conditional acceptance rate varies across response lengths under sequence-level importance-sampling clipping. When this variation is nonzero, the clipped update no longer preserves the target mixture over lengths; instead, it systematically reweights short and long responses differently. The term also has related but non-identical interpretations in adjacent areas, including translation quality estimation and temporally rescaled losses for sequence modelling, where the central issue is likewise a length- or time-dependent distortion of the error signal rather than a faithful reflection of task quality or memory structure (Mao et al., 11 Sep 2025).

1. Formal definition in sequence-level reinforcement learning

The FSPO treatment introduces LRE in a length-stratified analysis of sequence-level RL. Let $s \sim p(s)$ be a prompt or context, let $o = y_{1:L(o)}$ be a sampled response sequence of length $L(o)$, and let $\pi_\theta(o\mid s)$ denote the sequence policy. The sequence log importance-sampling ratio is defined as

$$S(o\mid s) = \log \frac{\pi_{\theta'}(o\mid s)}{\pi_{\theta}(o\mid s)} = \sum_{t=1}^{L(o)} \log \frac{\pi_{\theta'}(y_t \mid h_t(s,y_{<t}))}{\pi_{\theta}(y_t \mid h_t(s,y_{<t}))}.$$

For a length-dependent clip band $b_L>0$, the acceptance event is

$$A_L = \{\mathrm{len}(o)=L \ \text{and}\ |S(o\mid s)|\le b_L\}.$$

The corresponding length-conditional acceptance rate is

$q(L) = \Pr(A_L \mid L),$

and the global average acceptance is

$q = \mathbb{E}[q(L)].$

Under clipping, the effective gradient mixture becomes

$g(b) = \mathbb{E}_L\!\left[q(L)\, g_L\right],$

where $o = y_{1:L(o)}$0 is the per-length gradient contribution, while the unclipped target gradient is

$o = y_{1:L(o)}$1

Within this setup, Definition 2.1 gives

$o = y_{1:L(o)}$2

This quantity is zero exactly when the acceptance probability is constant across lengths, in which case clipping does not distort the length mixture. The paper also gives a weighted variant,

$o = y_{1:L(o)}$3

used for assumption-free analysis (Mao et al., 11 Sep 2025).

2. Mechanism: how fixed clipping induces length bias

The motivating problem is sequence-level RL for LLMs under verifiable, sequence-level rewards. In that regime, the reward is assigned to the whole completion, so the paper argues that it is more faithful to operate with sequence-level IS ratios rather than token-level ratios. The mismatch arises when PPO- or GRPO-style clipping is transplanted directly to the full sequence with a fixed clip range, for example

$o = y_{1:L(o)}$4

The key structural point is that the sequence log-ratio is a sum over tokens. The paper explicitly states that the drift of the log-ratio scales like $o = y_{1:L(o)}$5, while the dispersion scales like $o = y_{1:L(o)}$6. A fixed threshold therefore yields a length-dependent acceptance probability: longer sequences are more likely to be clipped, whereas shorter sequences are more likely to pass unclipped.

In this sense, LRE is not merely a diagnostic about response lengths. It quantifies a mismatch between the clipped objective and the intended sequence-level objective. The clipped update uses

$o = y_{1:L(o)}$7

instead of

$o = y_{1:L(o)}$8

A common misconception is that clipping only truncates extreme ratios while leaving the length composition of the update intact. The FSPO analysis rejects that view: if $o = y_{1:L(o)}$9 varies with $L(o)$0, then clipping changes the optimization target by reweighting responses by length (Mao et al., 11 Sep 2025).

3. Theoretical guarantees associated with small LRE

The main theoretical claim in the FSPO analysis is that small LRE implies directional alignment between the clipped and unclipped updates. The argument is built on a bounded within-length deviation condition, given informally as Assumption 2.1 (Bounded Stratification): there exists $L(o)$1 such that for all $L(o)$2,

$L(o)$3

The proof sketch additionally uses Assumption B.1 (Bounded co-variation),

$L(o)$4

to separate acceptance fluctuations from the scale of the stratified gradients.

The decomposition underlying the theorem is

$L(o)$5

Under these conditions, Theorem 2.1 states that

$L(o)$6

for some constant $L(o)$7.

The significance of this result is precise. If $L(o)$8 is nearly constant, then $L(o)$9 is close to a uniformly scaled version of $\pi_\theta(o\mid s)$0; hence the clipped update remains directionally aligned with the true gradient. LRE therefore functions as the control variable for how strongly clipping distorts optimization direction. In this formulation, length fairness is not a cosmetic property but a sufficient condition for preserving the geometry of the underlying update (Mao et al., 11 Sep 2025).

4. FSPO and length-fair clipping

FSPO, or Fair Sequence Policy Optimization, is introduced as a remedy for clipping-induced length bias. For prompts $\pi_\theta(o\mid s)$1, with $\pi_\theta(o\mid s)$2 sampled completions $\pi_\theta(o\mid s)$3, the paper optimizes

$\pi_\theta(o\mid s)$4

where

$\pi_\theta(o\mid s)$5

The central modification is the length-fair clipping band

$\pi_\theta(o\mid s)$6

where $\pi_\theta(o\mid s)$7 is a KL-corrected drift term and $\pi_\theta(o\mid s)$8 controls the width through the estimated standard deviation and a desired clip fraction. The $\pi_\theta(o\mid s)$9 scaling is motivated by the variance scaling of sums of approximately independent tokenwise log-ratio increments.

The paper further motivates this with a Gaussian approximation,

$$S(o\mid s) = \log \frac{\pi_{\theta'}(o\mid s)}{\pi_{\theta}(o\mid s)} = \sum_{t=1}^{L(o)} \log \frac{\pi_{\theta'}(y_t \mid h_t(s,y_{<t}))}{\pi_{\theta}(y_t \mid h_t(s,y_{<t}))}.$$0

with $$S(o\mid s) = \log \frac{\pi_{\theta'}(o\mid s)}{\pi_{\theta}(o\mid s)} = \sum_{t=1}^{L(o)} \log \frac{\pi_{\theta'}(y_t \mid h_t(s,y_{<t}))}{\pi_{\theta}(y_t \mid h_t(s,y_{<t}))}.$$1. To estimate the drift online, it defines the minibatch token-average KL statistic

$$S(o\mid s) = \log \frac{\pi_{\theta'}(o\mid s)}{\pi_{\theta}(o\mid s)} = \sum_{t=1}^{L(o)} \log \frac{\pi_{\theta'}(y_t \mid h_t(s,y_{<t}))}{\pi_{\theta}(y_t \mid h_t(s,y_{<t}))}.$$2

and maintains the exponential moving average

$$S(o\mid s) = \log \frac{\pi_{\theta'}(o\mid s)}{\pi_{\theta}(o\mid s)} = \sum_{t=1}^{L(o)} \log \frac{\pi_{\theta'}(y_t \mid h_t(s,y_{<t}))}{\pi_{\theta}(y_t \mid h_t(s,y_{<t}))}.$$3

This is tied to the expectation identity

$$S(o\mid s) = \log \frac{\pi_{\theta'}(o\mid s)}{\pi_{\theta}(o\mid s)} = \sum_{t=1}^{L(o)} \log \frac{\pi_{\theta'}(y_t \mid h_t(s,y_{<t}))}{\pi_{\theta}(y_t \mid h_t(s,y_{<t}))}.$$4

which justifies a linear-in-$$S(o\mid s) = \log \frac{\pi_{\theta'}(o\mid s)}{\pi_{\theta}(o\mid s)} = \sum_{t=1}^{L(o)} \log \frac{\pi_{\theta'}(y_t \mid h_t(s,y_{<t}))}{\pi_{\theta}(y_t \mid h_t(s,y_{<t}))}.$$5 drift correction. The paper also notes that dual-clip is implemented in log-space, and that the overall objective is to preserve IS semantics while removing length bias (Mao et al., 11 Sep 2025).

5. Empirical profile and practical implications

The empirical evidence reported for FSPO is organized around acceptance-by-length diagnostics, explicit LRE measurements, and downstream training behavior. Figure 1, as described in the text, plots $$S(o\mid s) = \log \frac{\pi_{\theta'}(o\mid s)}{\pi_{\theta}(o\mid s)} = \sum_{t=1}^{L(o)} \log \frac{\pi_{\theta'}(y_t \mid h_t(s,y_{<t}))}{\pi_{\theta}(y_t \mid h_t(s,y_{<t}))}.$$6 across response-length bins. RLOO and GSPO exhibit clearly length-dependent acceptance, whereas FSPO yields much flatter acceptance across bins.

The reported LRE values are:

Method	Reported LRE
RLOO	0.162
GSPO	0.264
FSPO	0.037

These values operationalize the intended fairness criterion: FSPO sharply reduces variation in acceptance across lengths. The paper remarks that slightly higher acceptance in the shortest bins can appear because of small sample size in those bins and occasional unusually short outlier sequences, but this does not alter the overall flattening trend.

The practical consequences are presented as both optimization and behavioral effects. Because fixed clipping over-accepts short responses and over-clips long ones, gradient signals become distorted: long incorrect answers may be overly suppressed, negative learning signals for overly long generations become weaker, and the model may fail to learn proper length control. In contrast, FSPO is reported to produce flatter acceptance-by-length, more stable training, and less pathological length drift. The paper further states that RLOO learns quickly at first but can later show response lengths that “explode” with filler text, while GSPO learns more slowly and struggles to increase length, especially on the 1.7B model; FSPO yields more balanced length dynamics and better performance, especially on the 8B model. The same section summarizes stronger Avg@32 results and consistency across models (Mao et al., 11 Sep 2025).

6. Related formulations and terminological boundaries

Although the formal definition above comes from sequence-level RL, closely related phenomena appear in other areas where the error signal is implicitly reweighted by sequence length or temporal position. In quality estimation for machine translation, one paper studies a systematic preference or penalty tied to output length, independent of true quality. It identifies two manifestations: bias w.r.t. source length, where scores decrease as the source passage gets longer even when translations are error-free, and bias w.r.t. translation length, where shorter candidates are preferred among equally good translations. To mitigate this, the paper proposes training on error density

$$S(o\mid s) = \log \frac{\pi_{\theta'}(o\mid s)}{\pi_{\theta}(o\mid s)} = \sum_{t=1}^{L(o)} \log \frac{\pi_{\theta'}(y_t \mid h_t(s,y_{<t}))}{\pi_{\theta}(y_t \mid h_t(s,y_{<t}))}.$$7

and reconstructing the error rating at inference by

$$S(o\mid s) = \log \frac{\pi_{\theta'}(o\mid s)}{\pi_{\theta}(o\mid s)} = \sum_{t=1}^{L(o)} \log \frac{\pi_{\theta'}(y_t \mid h_t(s,y_{<t}))}{\pi_{\theta}(y_t \mid h_t(s,y_{<t}))}.$$8

This is not presented as the FSPO definition of LRE, but it is a directly related form of length-dependent error reweighting in evaluation and reranking (Zhang et al., 24 Oct 2025).

A second adjacent line of work concerns temporally rescaled or temporally positive-weighted errors in sequence modelling. There, the weighted loss is

$$S(o\mid s) = \log \frac{\pi_{\theta'}(o\mid s)}{\pi_{\theta}(o\mid s)} = \sum_{t=1}^{L(o)} \log \frac{\pi_{\theta'}(y_t \mid h_t(s,y_{<t}))}{\pi_{\theta}(y_t \mid h_t(s,y_{<t}))}.$$9

with a continuous version

$b_L>0$0

The induced memory bias is

$b_L>0$1

Under continuity, linearity, time-homogeneity, causality, regularity, and positive integrable $b_L>0$2, the paper proves that any positive temporal weighting remains biased toward short-term memory, and that the only “memory-unbiased” error is the last-term-only loss. This provides a rigorous example of a broader principle: reweighting the loss across sequence positions changes what the model preferentially learns, even when the architecture is fixed (Wang et al., 2023).

The acronym LRE is also overloaded in several unrelated literatures. In quantum machine learning, it can mean Layerwise Richardson Extrapolation, a layerwise variant of zero-noise extrapolation in hybrid quantum neural networks (Njiki et al., 19 Apr 2026). In adaptive estimation and control, it can denote linear regression equations, as in perturbed LRE identification laws (Glushchenko et al., 2023). In combinatorics, it can mean Left Rotate, Right Rotate, Exchange, a generator set for sorting permutations in a Cayley graph (Kuppili et al., 2020). These usages are terminologically distinct from Length Reweighting Error.

For the length-fair RL usage, the essential point is that LRE is a quantitative measure of whether a clipping rule preserves or distorts the intended length mixture of the update. In related areas, the same underlying concern appears under different names: the error function, metric, or clipping rule can embed a systematic preference over sequence length or temporal position, with direct consequences for optimization, ranking, and generalization (Mao et al., 11 Sep 2025).