LSPO: Length-aware Sampling for Policy Optimization

• The paper introduces LSPO, a meta-RLVR algorithm that filters training prompts based on response length to improve reasoning accuracy.
• LSPO applies a batch-adaptive, post-rollout filtering process that retains both the shortest and longest responses to enhance policy learning.
• Empirical tests reveal that LSPO boosts accuracy across various benchmarks and RLVR methods by trading a higher per-step rollout cost for faster convergence.

Length-aware Sampling for Policy Optimization (LSPO) is a meta-RLVR algorithm for LLM reasoning that uses response length as a dynamic sampling signal during reinforcement learning with verifiable rewards. Rather than modifying the policy objective itself, LSPO filters training prompts at each optimization step according to the average response length of their sampled rollouts, with the aim of improving the effectiveness of policy learning across reasoning tasks. It was introduced in the context of post-Deepseek-R1 RLVR research, where much of the literature had concentrated on loss design, while dynamic sampling had been used primarily for training efficiency rather than final policy quality (Chen et al., 1 Oct 2025).

1. Conceptual basis

LSPO is motivated by two empirical observations reported for LLM reasoning under RLVR. First, recent studies of overthinking indicate a strong empirical correlation between response length and output quality: correct answers tend to be shorter, while incorrect or uncertain reasoning paths are longer. Second, response length is an easily-computable signal that may reflect both confidence and task difficulty. On that basis, LSPO treats response length as a criterion for data selection during RL training, rather than as a direct reward term or a constraint in the loss (Chen et al., 1 Oct 2025).

The method operates in the RLVR setting, where multiple responses are sampled per prompt and optimization is performed using verifiable rewards. LSPO is described as algorithm-agnostic: it can be layered on top of standard RLVR algorithms such as GRPO, DAPO, and GSPO. This makes it a meta-algorithm rather than a standalone objective. A plausible implication is that LSPO is best understood as a training-time sampling policy over prompts, not as a replacement for the underlying policy-gradient update.

A common misunderstanding is to equate length-aware policy optimization with direct encouragement of short outputs. LSPO does not do this. Its central design is to retain prompts at both ends of the observed length distribution, rather than uniformly preferring brevity. The method therefore uses length as a signal for informativeness in sampling, not as a universal target for minimization.

2. Sampling rule and optimization workflow

LSPO augments a standard RLVR iteration with a length-based filtering stage. For each prompt, multiple rollouts are generated. Before any length computation, LSPO applies accuracy-based filtering that removes prompts whose sampled outputs are either always correct or always incorrect; these are treated as zero-variance cases.

For each remaining prompt $q$, LSPO computes the average response length over its $G$ sampled outputs: $L(q) := \frac{1}{G} \sum_{i=1}^{G} |o_i|$ where $|o_i|$ is the length of the $i$-th response. The empirical CDF and percentiles of $L(q)$ are then computed within the batch. LSPO retains prompts if their average response length lies in either the lower tail or a capped upper band of the distribution: $$L(q) \le Q_{L(q)}(L_\text{low}) \;\lor\; [L(q) \ge Q_{L(q)}(L_\text{high}) \land L(q) \le Q_{L(q)}(L_\text{max})]$$ where $Q_{L(q)}(\cdot)$ is the length percentile function (Chen et al., 1 Oct 2025).

The retained prompts form a filtered minibatch on which the underlying RLVR update is executed. In simplified form, each training iteration proceeds by: generating rollouts, removing zero-variance prompts, computing average response lengths, selecting prompts from the shortest and longest regions of the batchwise length distribution, and then applying GRPO-, DAPO-, or GSPO-style optimization on the resulting minibatch.

This workflow has two notable properties. First, it is batch-adaptive: the effective thresholds are defined through percentiles rather than absolute token counts. Second, it is post-rollout: response length is measured after generation rather than predicted in advance. The paper presents both as deliberate design choices.

3. Reported empirical performance

LSPO was evaluated on multiple base models and datasets, including Qwen-2.5-Math-7B, Qwen3-4B-Base, and Llama-3.2-4B-Instruct, across three math reasoning benchmarks: AIME25, Olympiad, and Minerva. The reported outcome is consistent improvement over accuracy-only dynamic sampling across GRPO, DAPO, and GSPO backbones (Chen et al., 1 Oct 2025).

Base RLVR algorithm	Acc-only Avg@32	LSPO Avg@32
GRPO	37.5	38.7
DAPO	37.9	38.6
GSPO	38.5	39.2

For Qwen-2.5-Math-7B with GRPO, the reported benchmark values are AIME25 $21.5 \rightarrow 23.2$, Olympiad $48.5 \rightarrow 49.6$, and Minerva $G$0, moving the average from $G$1 to $G$2. Comparable gains are reported for DAPO and GSPO, and appendix results are said to show that Llama models also benefit from LSPO (Chen et al., 1 Oct 2025).

The training-efficiency picture is more nuanced. LSPO incurs higher rollout cost per step, approximately $G$3 more after accuracy filtering, because its sampling is stricter. However, the reported learning curves show faster convergence and better accuracy at any given wall-clock time, especially near convergence. This suggests that LSPO trades additional per-step collection cost for improved overall optimization efficiency.

4. Ablation findings and design rationale

The ablation study emphasizes that the exact use of length in sampling matters. The strongest result is that retaining both shortest and longest responses—the two extremes of length—is crucial. By contrast, training only on intermediate-length responses harms performance. The paper gives a representative comparison in which discarding both extremes, using $G$4, performs worse than the default LSPO configuration $G$5 and $G$6 (Chen et al., 1 Oct 2025).

Percentile-based filtering is also reported to be preferable to value-based filtering. The stated reason is that absolute thresholds are fragile because the response-length distribution changes during training, which in turn causes unstable batch sizes and excessive resampling. Percentile thresholds adapt to the batch and require fewer resampling rounds.

The study further reports that dynamic filtering based on accuracy percentiles is less effective than LSPO’s length-based filtering, although combined approaches are identified as a promising direction for future work. This is important for interpreting LSPO: the method is not merely another curriculum based on correctness statistics. Its distinctive claim is that response length supplies information not captured by reward variance or correctness alone.

A further implication drawn in the paper is that both highly confident cases and highly uncertain or difficult cases are informative for policy improvement. The shortest prompts in the retained set are associated with efficient or confident reasoning, while the longest prompts are associated with difficult or uncertain reasoning trajectories. LSPO formalizes this intuition as a dynamic sampling rule.

5. Relation to other length-aware policy-optimization methods

LSPO belongs to a broader line of work in which response length is treated as a first-class variable in post-training, but it intervenes at a different point in the RL pipeline than several related methods.

LUSPO addresses length bias in the policy objective itself. It identifies response length bias in GSPO and GRPO, attributes it to averaging and clipping effects, and proposes a length-weighted objective that multiplies each sequence loss by its own length in order to remove bias and prevent response length collapse (Liu et al., 5 Feb 2026). By contrast, LSPO leaves the underlying RL loss unchanged and instead alters which prompts are used for updates.

DDPO uses difficulty-differentiated reward shaping. It partitions questions into easy and hard regimes using rollout correctness, reduces output length for simple tasks, and expands exploration for complex tasks. Its theoretical argument is that expected accuracy is maximized when the length distribution is centered on an optimal length and is as concentrated as possible (Xia et al., 19 Mar 2026). LSPO does not estimate an optimal target length; it filters prompts by observed batchwise length percentiles.

HAPO uses per-problem history. It tracks a history state, typically the minimum length of previously generated correct responses for each problem, and defines a length reward that encourages progressively shorter correct solutions without over-penalizing shorter incorrect ones (Huang et al., 16 May 2025). LSPO, in contrast, does not maintain per-instance historical state and does not shape the reward.

FSPO and CTPO act at the importance-sampling and clipping level. FSPO enforces length-fair clipping for sequence-level RL by adapting the log-IS clipping band with a KL-corrected drift term and $G$7 scaling (Mao et al., 11 Sep 2025). CTPO proposes cumulative token IS ratios with position-adaptive clipping that scales with $G$8, seeking unbiased prefix correction with lower variance than full-sequence IS (Zhang et al., 8 May 2026). These methods modify the update geometry itself, whereas LSPO modifies the sampled data distribution presented to an otherwise standard optimizer.

This suggests a useful taxonomy of length-aware RLVR methods: sampling-based approaches such as LSPO, reward-shaping approaches such as DDPO and HAPO, and objective- or clipping-level approaches such as LUSPO, FSPO, and CTPO.

6. Significance, limits, and prospective extensions

LSPO is presented as shifting the role of dynamic sampling in RLVR from computational efficiency toward final model effectiveness. Because it is orthogonal to the underlying RLVR algorithm and can be plugged into GRPO-, DAPO-, or GSPO-style workflows, it broadens the design space of policy optimization beyond loss engineering alone (Chen et al., 1 Oct 2025).

Its main practical cost is that it remains rollout-dependent. The method first generates multiple responses and only then filters prompts using observed average response length. The reported $G$9 increase in rollout cost per step reflects this dependence. The paper therefore identifies a natural efficiency frontier: if expected response length could be estimated before generation, filtering could occur prior to expensive rollout.

Several future directions are explicitly proposed. These include building predictors to estimate expected response length pre-rollout, exploring adaptive or multi-factor filtering that combines length with signals such as entropy or confidence, and extending the method to domains in which response length is tightly linked to solution optimality, confidence, or human usability. These proposals indicate that LSPO is not framed as a complete theory of length-aware RLVR, but as a modular sampling strategy that exposes response length as a useful optimization signal.

Within the broader literature on reasoning-oriented RL for LLMs, LSPO is notable for isolating a simple claim: dynamic prompt selection based on the extremes of the response-length distribution can improve reasoning performance across multiple RLVR backbones. The reported ablations reinforce that the effect is not reducible to generic hard-example mining or to uniform pressure toward shorter outputs. Instead, LSPO treats length distribution itself as a structured signal for policy improvement.