--- title: Length Reweighting Error in Sequence-Level RL url: https://www.emergentmind.com/topics/length-reweighting-error-lre type: topic --- # Length Reweighting Error in Sequence-Level RL Searching arXiv for recent papers on “Length Reweighting Error”, FSPO, and related length-bias work. In sequence-level reinforcement learning for large language models, **Length Reweighting Error (LRE)** denotes a clipping-induced distortion of the effective length mixture in the update rule. In the FSPO formulation, LRE measures how much the **length-conditional acceptance rate** varies across response lengths under sequence-level importance-sampling clipping. When this variation is nonzero, the clipped update no longer preserves the target mixture over lengths; instead, it systematically reweights short and long responses differently. The term also has related but non-identical interpretations in adjacent areas, including translation quality estimation and temporally rescaled losses for sequence modelling, where the central issue is likewise a length- or time-dependent distortion of the error signal rather than a faithful reflection of task quality or memory structure [2509.09177]. ## 1. Formal definition in sequence-level reinforcement learning The FSPO treatment introduces LRE in a length-stratified analysis of sequence-level RL. Let \(s \sim p(s)\) be a prompt or context, let \(o = y_{1:L(o)}\) be a sampled response sequence of length \(L(o)\), and let \(\pi_\theta(o\mid s)\) denote the sequence policy. The **sequence log importance-sampling ratio** is defined as \[ S(o\mid s) = \log \frac{\pi_{\theta'}(o\mid s)}{\pi_{\theta}(o\mid s)} = \sum_{t=1}^{L(o)} \log \frac{\pi_{\theta'}(y_t \mid h_t(s,y_{0\), the acceptance event is \[ A_L = \{\mathrm{len}(o)=L \ \text{and}\ |S(o\mid s)|\le b_L\}. \] The corresponding **length-conditional acceptance rate** is \[ q(L) = \Pr(A_L \mid L), \] and the global average acceptance is \[ q = \mathbb{E}[q(L)]. \] Under clipping, the effective gradient mixture becomes \[ g(b) = \mathbb{E}_L\!\left[q(L)\, g_L\right], \] where \(g_L\) is the per-length gradient contribution, while the unclipped target gradient is \[ g^* = \mathbb{E}_L[g_L]. \] Within this setup, **Definition 2.1** gives \[ \mathrm{LRE} = \mathbb{E}\big[(q(L)-q)^2\big]. \] This quantity is zero exactly when the acceptance probability is constant across lengths, in which case clipping does not distort the length mixture. The paper also gives a weighted variant, \[ \mathrm{LRE}_w = \mathbb{E}\big[q(L)^2 - q^2\big], \] used for assumption-free analysis [2509.09177]. ## 2. Mechanism: how fixed clipping induces length bias The motivating problem is **sequence-level RL for LLMs under verifiable, sequence-level rewards**. In that regime, the reward is assigned to the whole completion, so the paper argues that it is more faithful to operate with **sequence-level IS ratios** rather than token-level ratios. The mismatch arises when PPO- or GRPO-style clipping is transplanted directly to the full sequence with a **fixed clip range**, for example \[ \mathrm{clip}\!\left(\frac{\pi_{\theta'}(o\mid s)}{\pi_{\theta}(o\mid s)}, 1-\epsilon,1+\epsilon\right). \] The key structural point is that the sequence log-ratio is a sum over tokens. The paper explicitly states that the **drift** of the log-ratio scales like \(O(L)\), while the **dispersion** scales like \(O(\sqrt{L})\). A fixed threshold therefore yields a length-dependent acceptance probability: longer sequences are more likely to be clipped, whereas shorter sequences are more likely to pass unclipped. In this sense, LRE is not merely a diagnostic about response lengths. It quantifies a mismatch between the clipped objective and the intended sequence-level objective. The clipped update uses \[ g(b)=\mathbb{E}_L[q(L)g_L] \] instead of \[ g^*=\mathbb{E}_L[g_L]. \] A common misconception is that clipping only truncates extreme ratios while leaving the length composition of the update intact. The FSPO analysis rejects that view: if \(q(L)\) varies with \(L\), then clipping changes the optimization target by **reweighting responses by length** [2509.09177]. ## 3. Theoretical guarantees associated with small LRE The main theoretical claim in the FSPO analysis is that **small LRE implies directional alignment** between the clipped and unclipped updates. The argument is built on a bounded within-length deviation condition, given informally as **Assumption 2.1 (Bounded Stratification)**: there exists \(\eta \in [0,1)\) such that for all \(L\), \[ |g_L - g^*| \le \eta\, \|g^*\|. \] The proof sketch additionally uses **Assumption B.1 (Bounded co-variation)**, \[ \mathbb{E}\big[\,|q(L)-q|\,\|g_L\|\,\big] < \mathbb{E}|q(L)-q| \cdot \mathbb{E}\|g_L\|, \] to separate acceptance fluctuations from the scale of the stratified gradients. The decomposition underlying the theorem is \[ g(b)-qg^* = \mathbb{E}_L\!\left[(q(L)-q)g_L\right] + \mathbb{E}_L\!\left[q(L)(g_L-g^*)\right]. \] Under these conditions, **Theorem 2.1** states that \[ \cos\!\big(g(b),g^*\big) \ge 1 - \kappa\Big(\eta + (1+\eta)\,\mathrm{LRE}\Big), \] for some constant \(\kappa \ge 1\). The significance of this result is precise. If \(q(L)\) is nearly constant, then \(g(b)\) is close to a uniformly scaled version of \(g^*\); hence the clipped update remains directionally aligned with the true gradient. LRE therefore functions as the control variable for how strongly clipping distorts optimization direction. In this formulation, **length fairness** is not a cosmetic property but a sufficient condition for preserving the geometry of the underlying update [2509.09177]. ## 4. FSPO and length-fair clipping FSPO, or **Fair Sequence Policy Optimization**, is introduced as a remedy for clipping-induced length bias. For prompts \(x\sim\mathcal D\), with \(G\) sampled completions \(\{y_i\}_{i=1}^G\), the paper optimizes \[ J_{\mathrm{FSPO}(\theta') = \mathbb{E}_{x,\{y_i\} \left[ \sum_{i=1}^G \min\Big( e^{S_i(\theta')} A_i,\; e^{\mathrm{clip}(S_i(\theta'),-b_{L_i},b_{L_i})} A_i \Big) \right], \] where \[ S_i(\theta') = \log\frac{\pi_{\theta'}(y_i\mid x)}{\pi_{\theta}(y_i\mid x)} = \sum_{t=1}^{L_i} \log \frac{\pi_{\theta'}(y_{i,t}\mid h_{i,t})}{\pi_{\theta}(y_{i,t}\mid h_{i,t})}. \] The central modification is the **length-fair clipping band** \[ b_L = \mu_L + c\sqrt{L}, \] where \(\mu_L\) is a **KL-corrected drift term** and \(c\) controls the width through the estimated standard deviation and a desired clip fraction. The \(\sqrt{L}\) scaling is motivated by the variance scaling of sums of approximately independent tokenwise log-ratio increments. The paper further motivates this with a Gaussian approximation, \[ S_L - \mu_L \sqrt{L} \Rightarrow \mathcal N(0,\sigma^2), \] with \(\mu_L = O(L)\). To estimate the drift online, it defines the minibatch token-average KL statistic \[ \mathrm{KL}^{\text{token}}_k = \frac{1}{N_{\text{tok}}} \sum_{i=1}^{|B_k|} \sum_{t=1}^{L_i} \log \frac{\pi_\theta(y_{i,t}\mid h_{i,t})}{\pi_{\theta'}(y_{i,t}\mid h_{i,t})}, \] and maintains the exponential moving average \[ \hat{\mu}_k = (1-\alpha)\hat{\mu}_{k-1} + \alpha\big(-\mathrm{KL}^{\text{token}}_k\big). \] This is tied to the expectation identity \[ \mathbb{E}[S_L] = - \sum_{t=1}^L \mathbb{E}\Big[\mathrm{KL}\big(\pi_\theta(\cdot\mid h_t)\,\|\,\pi_{\theta'}(\cdot\mid h_t)\big)\Big], \] which justifies a linear-in-\(L\) drift correction. The paper also notes that dual-clip is implemented in log-space, and that the overall objective is to preserve IS semantics while removing length bias [2509.09177]. ## 5. Empirical profile and practical implications The empirical evidence reported for FSPO is organized around **acceptance-by-length diagnostics**, explicit LRE measurements, and downstream training behavior. Figure 1, as described in the text, plots \(q(L)\) across response-length bins. **RLOO** and **GSPO** exhibit clearly length-dependent acceptance, whereas **FSPO** yields much flatter acceptance across bins. The reported LRE values are: | Method | Reported LRE | |---|---:| | RLOO | 0.162 | | GSPO | 0.264 | | FSPO | 0.037 | These values operationalize the intended fairness criterion: FSPO sharply reduces variation in acceptance across lengths. The paper remarks that slightly higher acceptance in the shortest bins can appear because of small sample size in those bins and occasional unusually short outlier sequences, but this does not alter the overall flattening trend. The practical consequences are presented as both optimization and behavioral effects. Because fixed clipping over-accepts short responses and over-clips long ones, gradient signals become distorted: long incorrect answers may be overly suppressed, negative learning signals for overly long generations become weaker, and the model may fail to learn proper length control. In contrast, FSPO is reported to produce **flatter acceptance-by-length**, **more stable training**, and **less pathological length drift**. The paper further states that **RLOO** learns quickly at first but can later show response lengths that “explode” with filler text, while **GSPO** learns more slowly and struggles to increase length, especially on the **1.7B** model; **FSPO** yields more balanced length dynamics and better performance, especially on the **8B** model. The same section summarizes stronger **Avg@32** results and consistency across models [2509.09177]. ## 6. Related formulations and terminological boundaries Although the formal definition above comes from sequence-level RL, closely related phenomena appear in other areas where the error signal is implicitly reweighted by sequence length or temporal position. In **quality estimation for machine translation**, one paper studies a systematic preference or penalty tied to output length, independent of true quality. It identifies two manifestations: **bias w.r.t. source length**, where scores decrease as the source passage gets longer even when translations are error-free, and **bias w.r.t. translation length**, where shorter candidates are preferred among equally good translations. To mitigate this, the paper proposes training on **error density** \[ D(x, h) = \frac{R(x, h)}{|h|} \] and reconstructing the error rating at inference by \[ \hat{R}(x, h) = D(x, h) \cdot |h|. \] This is not presented as the FSPO definition of LRE, but it is a directly related form of length-dependent error reweighting in evaluation and reranking [2510.22028]. A second adjacent line of work concerns **temporally rescaled** or **temporally positive-weighted** errors in sequence modelling. There, the weighted loss is \[ \mathrm{Error}^{\mathrm{TPE}} = \frac{1}{T}\sum_{t=1}^T w(t)\,|y(t)-\hat{y}(t)|,\qquad w(t)>0, \] with a continuous version \[ \mathrm{Error}^{\mathrm{TPE}} = \frac{1}{T}\int_0^T w(t)\,|y(t)-\hat{y}(t)|\,dt. \] The induced memory bias is \[ b(s)=\int_s^T w(t)\,dt. \] Under continuity, linearity, time-homogeneity, causality, regularity, and positive integrable \(w\), the paper proves that **any positive temporal weighting remains biased toward short-term memory**, and that the only “memory-unbiased” error is the **last-term-only** loss. This provides a rigorous example of a broader principle: reweighting the loss across sequence positions changes what the model preferentially learns, even when the architecture is fixed [2307.11462]. The acronym **LRE** is also overloaded in several unrelated literatures. In quantum machine learning, it can mean **Layerwise Richardson Extrapolation**, a layerwise variant of zero-noise extrapolation in hybrid quantum neural networks [2604.17515]. In adaptive estimation and control, it can denote **linear regression equations**, as in perturbed LRE identification laws [2310.14073]. In combinatorics, it can mean **Left Rotate, Right Rotate, Exchange**, a generator set for sorting permutations in a Cayley graph [2002.07342]. These usages are terminologically distinct from **Length Reweighting Error**. For the length-fair RL usage, the essential point is that LRE is a quantitative measure of whether a clipping rule preserves or distorts the intended length mixture of the update. In related areas, the same underlying concern appears under different names: the error function, metric, or clipping rule can embed a systematic preference over sequence length or temporal position, with direct consequences for optimization, ranking, and generalization [2509.09177]. Source: https://www.emergentmind.com/topics/length-reweighting-error-lre