Step-Level Length Control Policy Optimization

• SCPO is a reinforcement learning approach that controls the length of individual reasoning steps to balance efficiency and accuracy.
• It uses online importance estimation and step-level generalized advantage estimation to redistribute token allocation, ensuring key steps are preserved.
• Empirical benchmarks show SCPO enhances token efficiency and reasoning performance compared to global length regularization methods.

Searching arXiv for the SCPO-related papers and adjacent step-level RL work. Step-Level Length Control Policy Optimization (SCPO) is a reinforcement learning method for large reasoning models that applies length control at the granularity of individual reasoning steps rather than at the level of an entire chain of thought. In the literature summarized here, the clearest explicit formulation of SCPO appears in SmartThinker, where it is presented as the second-stage policy optimization component of a two-stage framework for compressing redundant reasoning while preserving critical reasoning segments (He et al., 6 Jul 2025). SCPO belongs to a broader step-level optimization trend in agentic and reasoning-oriented RL, alongside step-level MDP formulations for LLM agents and step-level process-reward methods in navigation, but it is distinguished by its explicit focus on differentiated length allocation across steps, its online step-importance estimation, and its use of step-level generalized advantage estimation for token-level policy updates (Wang et al., 20 Apr 2026).

1. Conceptual foundations and problem setting

SCPO addresses a failure mode of global length regularization in reasoning-model training. Prior approaches that penalize only total sequence length use a single scalar reward over the whole reasoning trace. In the SmartThinker formulation, this produces an undesirable trade-off: critical reasoning segments may be over-compressed, while trivial segments may remain verbose (He et al., 6 Jul 2025). The step-level perspective instead assumes that a reasoning trajectory contains heterogeneous internal segments whose informational value is not uniform.

In SCPO, a sampled response $o_i$ is decomposed into reasoning steps and a final answer,

$$o_i = (S_i, a_i), \qquad S_i = \{s_{i1}, s_{i2}, \dots, s_{ik_i}\},$$

where each $s_{i,j}$ is a contiguous chunk of tokens with token length $l_{i,j}$. The policy remains autoregressive at the token level, but rewards and advantages are computed at the step level and then assigned back to the tokens within each step (He et al., 6 Jul 2025). This makes SCPO a hybrid method: token-parameterized, but step-aligned in credit assignment.

This step alignment is consistent with a broader critique of token-centric RL for long-horizon agents. StepPO argues that in multi-turn and tool-using settings the environment evolves at the granularity of interaction rounds rather than single tokens, and therefore the proper action representation is the step, not the token (Wang et al., 20 Apr 2026). A plausible implication is that SCPO can be understood as the specialization of this step-level viewpoint to reasoning-length control: the internal sequence remains tokenized for likelihood computation, but the optimization target is the step as a semantically meaningful unit.

A related motivation appears in SeeNav-Agent, which introduces step-level reinforcement fine-tuning for vision-language navigation. There, the emphasis is on dense, verifiable process rewards for planning, not on reasoning-length allocation, but the same basic premise holds: step-level credit is often more informative than trajectory-level or token-level credit in long-horizon tasks (Wang et al., 2 Dec 2025). By contrast, LAPO operates at the trajectory level and controls the length of full reasoning chains through sequence-level rewards and prompt-level budgets rather than explicit step-level process supervision (Wu et al., 21 Jul 2025).

2. Historical trajectory and relation to adjacent methods

SCPO emerges from the convergence of two research lines: length-aware RL for reasoning models and step-level policy optimization for sequential decision making. The SmartThinker paper explicitly positions SCPO against global length penalty methods such as Alpha-0.1 and L1-Max, arguing that sequence-level compression is too coarse for preserving reasoning quality on difficult problems (He et al., 6 Jul 2025). In that sense, SCPO is not merely a shorter-output objective; it is a structured policy optimization scheme that redistributes length inside the reasoning trace.

The immediate conceptual precursors are methods that elevate the step as the basic optimization unit. StepPO formalizes a step-level MDP for LLM agents, with states $s_t$, whole-step actions $a_t$, step rewards $r_t$, and a return objective

$$J(\theta)=\mathbb{E}_{\tau\sim\pi_\theta}\left[\sum_{t=0}^{T-1}\gamma^t r_t\right].$$

Its actor update uses a clipped PPO-style surrogate on step-level importance ratios and step-level advantages derived from GAE (Wang et al., 20 Apr 2026). Although StepPO does not itself introduce a dedicated SCPO algorithm, its framework is presented as a substrate on which explicit length-aware step costs can be added directly to $r_t$. This suggests a general recipe for step-level length control in agentic RL.

SeeNav-Agent offers a second neighboring design. Its Step Reward Group Policy Optimization (SRGPO) combines episode-level and step-level normalized advantages,

$$A_{i,t}=A_E(\boldsymbol{\tau}_i)+\omega\cdot A_S(\mathbf{c}^{(i)}_t,\mathbf{a}^{(i)}_t),$$

and uses randomly grouped step rewards to obtain dense process supervision without requiring identical states (Wang et al., 2 Dec 2025). In the supplied technical reference, this is explicitly discussed as a pattern that could inform an SCPO-style method by adding explicit per-step length costs. The common thread is not identical notation but the shift from terminal-only reward assignment to per-step local credit.

LAPO represents a different point in the design space. It also targets efficiency, but at the level of full reasoning sequences. In Stage 1 it learns distributions of successful solution lengths, and in Stage 2 it inserts an explicit plan such as “I will answer the question with $$o_i = (S_i, a_i), \qquad S_i = \{s_{i1}, s_{i2}, \dots, s_{ik_i}\},$$0 tokens” into the reasoning context, with a Gaussian adherence reward centered at the target length $$o_i = (S_i, a_i), \qquad S_i = \{s_{i1}, s_{i2}, \dots, s_{ik_i}\},$$1 (Wu et al., 21 Jul 2025). LAPO therefore internalizes reasoning efficiency, but it does so without explicit step segmentation, per-step rewards, or stepwise halting decisions. Relative to LAPO, SCPO is finer-grained and more local in its control signals.

3. Formal definition of SCPO in SmartThinker

In SmartThinker, SCPO is defined as a GRPO-derived policy optimization method with four core components: an online importance estimator, a step-level length control reward function, step-level generalized advantage estimation (S-GAE), and a difficulty-adaptive clipping strategy (He et al., 6 Jul 2025). The policy remains token-autoregressive,

$$o_i = (S_i, a_i), \qquad S_i = \{s_{i1}, s_{i2}, \dots, s_{ik_i}\},$$2

but the scalar advantage attached to tokens depends on the step they belong to.

Online importance estimation

For a reasoning step $$o_i = (S_i, a_i), \qquad S_i = \{s_{i1}, s_{i2}, \dots, s_{ik_i}\},$$3, SmartThinker defines the full-trajectory answer probability

$$o_i = (S_i, a_i), \qquad S_i = \{s_{i1}, s_{i2}, \dots, s_{ik_i}\},$$4

and the answer probability after removing that step,

$$o_i = (S_i, a_i), \qquad S_i = \{s_{i1}, s_{i2}, \dots, s_{ik_i}\},$$5

The raw importance is then

$$o_i = (S_i, a_i), \qquad S_i = \{s_{i1}, s_{i2}, \dots, s_{ik_i}\},$$6

This makes a step important if its removal lowers the probability of the correct answer, while normalizing by step length to avoid rewarding verbosity for its own sake (He et al., 6 Jul 2025).

SmartThinker augments this with an additional score for “transition” steps containing keywords such as “but” or “however,” motivated by the observation that purely removal-based importance can underweight reflection and verification. Difficulty is defined by sampled group accuracy,

$$o_i = (S_i, a_i), \qquad S_i = \{s_{i1}, s_{i2}, \dots, s_{ik_i}\},$$7

where $$o_i = (S_i, a_i), \qquad S_i = \{s_{i1}, s_{i2}, \dots, s_{ik_i}\},$$8 is the number of correct responses among $$o_i = (S_i, a_i), \qquad S_i = \{s_{i1}, s_{i2}, \dots, s_{ik_i}\},$$9 sampled responses. The final importance is

$s_{i,j}$0

On difficult problems, transition-like steps therefore receive a larger bonus, which weakens their compression pressure.

Step-level reward

For correct responses, SCPO standardizes step length $s_{i,j}$1 and total step count $s_{i,j}$2, and defines a min-max normalized importance $s_{i,j}$3. With base coefficient $s_{i,j}$4, it sets

$s_{i,j}$5

The correct-response step reward is

$s_{i,j}$6

so long steps are penalized unless they are important, and large numbers of steps are penalized unless the problem is difficult (He et al., 6 Jul 2025).

For incorrect responses, SCPO uses

$s_{i,j}$7

This reward is always negative, but it does not include an explicit length penalty. The stated rationale is to preserve exploration: incorrect trajectories may still need substantial reasoning length, especially on difficult questions (He et al., 6 Jul 2025).

Step-level generalized advantage estimation

A central component of SCPO is S-GAE. SmartThinker argues that undiscounted cumulative process rewards produce a length bias because longer trajectories accumulate more reward merely by containing more steps. SCPO therefore computes

$s_{i,j}$8

where $s_{i,j}$9 is the normalized step reward and $l_{i,j}$0 is a discount factor (He et al., 6 Jul 2025). The same step-level advantage is then assigned to all tokens in the corresponding step.

This is closely analogous in spirit to the step-level GAE advocated by StepPO for agentic RL, where the one-step TD residual is

$l_{i,j}$1

and advantages are accumulated over steps rather than tokens (Wang et al., 20 Apr 2026). The difference is that SmartThinker’s SCPO is process-reward-driven for reasoning compression, whereas StepPO is framed as a more general actor-critic formalism for agent-environment interaction.

Difficulty-adaptive clipping

SCPO modifies PPO-style clipping bounds as a function of difficulty: $l_{i,j}$2 Easy problems receive tighter clipping, which suppresses unnecessary exploration; difficult problems receive a looser upper bound, which permits more exploration of low-probability continuations (He et al., 6 Jul 2025). The result is a difficulty-conditioned exploration regime integrated directly into the surrogate update.

4. Training pipeline and optimization mechanics

SCPO is used as the second stage of the two-stage SmartThinker framework. The first stage is a short-reasoning-mode warm-up based on rejection sampling and supervised fine-tuning. For each question, the base long-reasoning model generates multiple candidates, the correct answers are filtered, and the shortest correct response is selected for SFT (He et al., 6 Jul 2025). This shifts the policy distribution toward shorter but valid reasoning before RL begins.

The SCPO stage then performs step-level RL over grouped sampled responses. For each batch question, SmartThinker samples $l_{i,j}$3 responses, decomposes each into steps, computes difficulty $l_{i,j}$4, estimates step importance by step removal, constructs the step rewards, normalizes them, computes S-GAE, and finally optimizes a clipped token-level objective using the step-level advantages. The paper states that the explicit KL penalty is omitted, setting $l_{i,j}$5, because prior work indicates that an explicit KL term can overly constrain exploration (He et al., 6 Jul 2025).

The practical training configuration reported in the supplied reference uses 3,000 QA pairs from DeepScaleR-Preview, batch size 8 questions, 8 responses per question, context length 4K during training and 8K for evaluation, AdamW with learning rate $l_{i,j}$6, cosine schedule with 60-step warm-up, and 4 policy updates per batch (He et al., 6 Jul 2025). The key SCPO-specific hyperparameters are $l_{i,j}$7, $l_{i,j}$8, $l_{i,j}$9, and $s_t$0.

From a systems perspective, this workflow is consistent with the broader step-native training logic articulated in StepPO. StepPO emphasizes storing trajectories at step boundaries, preserving token-ID consistency, and aligning reward propagation with complete actions rather than sub-action tokens (Wang et al., 20 Apr 2026). A plausible implication is that SCPO inherits many of the engineering requirements of general step-level RL: reliable step segmentation, consistent replay of step-aligned token spans, and efficient recomputation for per-step diagnostics such as ablation-style importance estimates.

A related but distinct implementation lesson comes from SRGPO. There, random step grouping across trajectories reduces the need for identical-state matching and lowers variance in step-level advantage estimation (Wang et al., 2 Dec 2025). While SmartThinker does not use the same grouping mechanism, both methods show that dense step-level learning signals can be made practical when stepwise comparability is carefully defined.

5. Empirical behavior, benchmarks, and ablations

SmartThinker reports that SCPO increases the proportion of effective steps—defined as steps with normalized importance greater than 0.01—by $s_t$1 for low-difficulty questions and $s_t$2 for high-difficulty questions, while increasing the percentage of total length attributed to effective steps by approximately $s_t$3 (He et al., 6 Jul 2025). This is the paper’s direct evidence that length is being redistributed toward more consequential reasoning segments rather than simply removed globally.

On DeepSeek-R1-1.5B, SmartThinker is reported to achieve the best average Pass@k and Maj@k among the listed baselines while using the fewest tokens. Relative to Alpha-0.1, the overall average Pass@k improves by 5.4 points, and average length is reduced from 5763 to 3283, or $s_t$4 of the Alpha-0.1 baseline (He et al., 6 Jul 2025). The same source states that on AIME24, relative to Alpha-0.1, Pass@10 improves by $s_t$5 and average length decreases from 7380 to 5178. On DeepSeek-R1-7B, SmartThinker is described as outperforming Alpha-0.1 and AdaptThink in Pass@k and Maj@k while also improving token efficiency (He et al., 6 Jul 2025).

The paper also reports a substantial training-efficiency effect from the SFT warm-up. Without SFT, RL training is described as slow and unstable, taking 122 hours for 1,000 steps; with SFT warm-up, 1,300 steps are completed in 68 hours with similar or better accuracy (He et al., 6 Jul 2025). This indicates that SCPO is not only an objective design but also part of a training curriculum.

Several ablations isolate the role of SCPO’s components. For S-GAE, $s_t$6 is reported to yield overly long sequences, while too small a $s_t$7 underweights later steps excessively; $s_t$8 gives the best balance (He et al., 6 Jul 2025). For the base penalty $s_t$9, a large value such as 0.8 minimizes tokens but causes noticeable accuracy drops, whereas overly small values weaken differentiated compression; $a_t$0 is reported as the best trade-off. For clipping, larger $a_t$1 values make the lower bound too aggressive, causing entropy instability. The transition-keyword bonus is also presented as necessary: without it, the model reduces transition-word usage and performance deteriorates, especially through under-preservation of reflective steps (He et al., 6 Jul 2025).

These results align qualitatively with the broader literature’s claims about step-level signals. StepPO reports that step-aligned optimization outperforms token-level PPO on HotpotQA in a multi-step agent setting using identical base model, data, rollout pipeline, and hyperparameters except for optimization granularity (Wang et al., 20 Apr 2026). SeeNav-Agent similarly reports that SRGPO improves training stability, convergence efficiency, and generalization capability relative to GRPO and GiGPO in navigation (Wang et al., 2 Dec 2025). While these are different domains and objectives, they reinforce the general proposition that step-level credit assignment can be more effective than coarse sequence-level optimization in long-horizon settings.

6. Interpretation, limitations, and acronym ambiguity

SCPO can be interpreted as a form of structure-aware RL for reasoning. Its central claim is not merely that shorter outputs are desirable, but that reasoning efficiency depends on allocating computational budget unevenly across steps. Important steps should be protected from compression; redundant steps should absorb most of the penalty (He et al., 6 Jul 2025). This is a more granular objective than the length-aware sequence reward used in LAPO, where efficiency is learned through problem-level target lengths and adherence to a self-declared token budget (Wu et al., 21 Jul 2025).

At the same time, SCPO inherits limitations from its design. The online importance estimator requires recomputing answer probabilities with individual steps removed, which the SmartThinker reference notes is computationally non-trivial (He et al., 6 Jul 2025). Step segmentation is assumed rather than deeply theorized. Difficulty is estimated from the fraction of correct responses in a sampled group, which may be noisy when correctness is sparse or group size is small. The method is also demonstrated on mathematical reasoning tasks; extension to other domains is described as plausible rather than established.

A broader open question concerns the relationship between semantic steps and optimization units. StepPO argues for elevating the step to the status of the action in agentic RL, with step-aligned MDPs, structured traces, and whole-step PPO objectives (Wang et al., 20 Apr 2026). SmartThinker’s SCPO, by contrast, still optimizes token probabilities and assigns the same scalar advantage to all tokens in a segmented step. This suggests that present step-level length control occupies an intermediate position: semantically step-aware, but not yet fully step-native at the policy parameterization level.

The acronym “SCPO” is also overloaded in the literature. In safe RL, SCPO denotes “State-wise Constrained Policy Optimization,” a trust-region method for state-wise safety constraints in robotics and locomotion (Zhao et al., 2023). In later constrained learning work, SCPO denotes “Constrained Policy Optimization via Sampling-Based Weight-Space Projection,” a safe-by-induction parameter-space projection method under rollout-based safety constraints (Cao et al., 15 Dec 2025). These usages are unrelated to Step-Level Length Control Policy Optimization. In the reasoning-model context, the SCPO relevant here is specifically the SmartThinker method for step-level length control (He et al., 6 Jul 2025).

Taken together, the current literature suggests three complementary design directions. SmartThinker provides an explicit SCPO algorithm for importance-aware compression inside reasoning chains (He et al., 6 Jul 2025). StepPO supplies a more general step-level RL formalism in which per-step length costs can be inserted directly into the reward and advantage targets (Wang et al., 20 Apr 2026). SeeNav-Agent shows that step-level reward grouping and dense process supervision can materially improve stability in long-horizon decision tasks (Wang et al., 2 Dec 2025). A plausible implication is that future SCPO variants may combine SmartThinker’s importance-aware length allocation, StepPO’s step-native MDP and systems design, and LAPO’s problem-specific efficiency priors into a unified family of step-aligned computation-allocation methods for reasoning and agentic systems (Wu et al., 21 Jul 2025).