Self-traced Step-wise Preference Optimization (SSPO)

• The paper demonstrates SSPO as a framework that uses self-generated stepwise value estimates to address overthinking and error accumulation in chain-of-thought reasoning.
• SSPO employs Verbal Value Probing (VVP) to compute dense, step-level signals that guide the reinforcement learning process without needing auxiliary models or human annotations.
• By pruning gradient updates after the first decline in value, SSPO effectively compresses reasoning trajectories, reducing verbosity while preserving or improving accuracy.

Searching arXiv for the cited SSPO and closely related step-wise preference optimization papers. arXiv search: (Xu et al., 18 Aug 2025, Chen et al., 2024, Lai et al., 2024, Xu et al., 20 Feb 2025, Yi et al., 19 Feb 2025, Li et al., 2024, Lahlou et al., 2024) Self-traced Step-wise Preference Optimization (SSPO) is a post-training framework for reasoning-oriented LLMs that augments rule-based reinforcement learning with dense, step-wise process supervision generated by the policy model itself. In the formulation introduced in "SSPO: Self-traced Step-wise Preference Optimization for Process Supervision and Reasoning Compression," SSPO targets two linked pathologies of chain-of-thought post-training: high computational cost and overthinking, understood as verbose reasoning that fails to self-fix and allows errors to accumulate across multiple steps. The framework is described as pluggable, requiring neither auxiliary models nor stepwise manual annotations, and it is designed to improve both answer accuracy and reasoning succinctness by tracing value changes along a model’s own reasoning trajectory and pruning updates after the first detected decline (Xu et al., 18 Aug 2025).

1. Conceptual basis and problem setting

SSPO is motivated by a contrast between two familiar families of reasoning post-training methods. Outcome-level rule-based reinforcement learning methods such as GRPO use sparse final-answer rewards and therefore treat all tokens or reasoning steps in a trajectory similarly. The SSPO paper argues that this permissive signal can encourage redundant steps, overthinking, and error accumulation. Process supervision methods such as PRM-style approaches or MCTS-based schemes provide denser intermediate supervision, but they typically require auxiliary reward or value models, manual stepwise annotations, multiple rollouts or search procedures, or external API-based evaluators (Xu et al., 18 Aug 2025).

The framework’s central claim is that many incorrect chain-of-thought outputs do not fail immediately. Rather, they often begin from an initially plausible path and then continue generating after the reasoning state has already deteriorated. The paper reports that some questions are answered correctly by direct inference but become incorrect under chain-of-thought prompting, and that these failing trajectories often exhibit fluctuating or declining stepwise value estimates. This motivates a training rule that supervises reasoning at the level of individual steps and suppresses reinforcement of later low-value continuations (Xu et al., 18 Aug 2025).

In this sense, SSPO uses "preference optimization" in a broader process-supervision sense rather than in the narrow DPO sense of pairwise whole-response learning. Steps associated with increasing stepwise value are treated as preferred, while steps after the first value decline are treated as likely error propagation. This suggests a preference ordering over intermediate reasoning states or prefixes rather than over complete responses alone.

2. Formal representation and Verbal Value Probing

The paper represents a response as

$$o = [s_1, s_2, ..., s_T, s_c, y] \in \mathbb{N}^{l_r}.$$

Here, $s_n \in \mathbb{N}^{l_n}$ is the token sequence of the $n$-th reasoning step, $T$ is the number of reasoning steps, $s_c$ is a drawn conclusion format such as “Thus, the answer is {\cdot}”, and $y$ is the ground-truth string that can be validated by rule-based matching. A hidden state at step $t$ is written as

$h_t = [p, s_1, s_2, ..., s_t],$

with the action at step $t$ being the generation of the next reasoning step $s_{t+1}$ (Xu et al., 18 Aug 2025).

SSPO begins by recalling a critic-based generalized advantage formulation,

$s_n \in \mathbb{N}^{l_n}$0

Its key move is to replace an explicit critic with self-generated stepwise value estimates obtained by Verbal Value Probing (VVP). The VVP score at step $s_n \in \mathbb{N}^{l_n}$1 is defined as

$s_n \in \mathbb{N}^{l_n}$2

where the surrounding description clarifies that, after appending the conclusion format $s_n \in \mathbb{N}^{l_n}$3 to the current partial reasoning prefix, the model is probed for the probability of producing the ground-truth answer token or string $s_n \in \mathbb{N}^{l_n}$4. This probability is interpreted as a stepwise estimate of how promising the current reasoning state is (Xu et al., 18 Aug 2025).

The framework is therefore "self-traced" because the model traces its own reasoning quality using its own output probabilities, rather than an external process reward model. The resulting $s_n \in \mathbb{N}^{l_n}$5 values function as coarse human-preference proxies: larger $s_n \in \mathbb{N}^{l_n}$6 indicates that the prefix up to step $s_n \in \mathbb{N}^{l_n}$7 is more aligned with eventually producing the correct answer, while lower or declining $s_n \in \mathbb{N}^{l_n}$8 indicates that the reasoning may be going off track.

The paper also analyzes "CoT-poisonous" and "CoT-beneficial" queries. CoT-poisonous queries are those for which direct inference is correct but chain-of-thought reasoning is incorrect, while CoT-beneficial queries show the reverse pattern. The reported VVP trajectories support the claim that overthinking often coincides with fluctuating or decreasing stepwise value sequences (Xu et al., 18 Aug 2025).

3. Self-traced advantage estimation and Error Step Pruning

SSPO reformulates GAE using VVP-derived values. For rollout $s_n \in \mathbb{N}^{l_n}$9, the stepwise advantage is

$n$0

The rollout-level binary reward is group-normalized as

$n$1

where $n$2 is the correctness of rollout $n$3. The value normalization is written as

$n$4

The paper notes a minor typographical artifact in this printed equation, but the intended meaning is normalization of the VVP value using the same group statistics (Xu et al., 18 Aug 2025).

This advantage has two components. The first, $n$5, compares different sampled trajectories in the rollout group. The second, $n$6, measures whether the current step improves or worsens the trajectory. The paper explicitly frames this as a dual-advantage view: between-trajectory comparison plus within-trajectory temporal comparison.

The distinctive training rule is Error Step Pruning. SSPO assumes that preferred reasoning trajectories should exhibit a monotonically increasing value sequence. It therefore identifies the first step at which the value stops increasing and prunes subsequent steps from gradient updates. The training loss is

$n$7

so later steps after the first value decline are excluded from gradient updates. The paper states that for declining steps the advantage function “degrade[s] into the vanilla GRPO” (Xu et al., 18 Aug 2025).

This pruning rule operationalizes the framework’s preference interpretation. The model traces where its own reasoning begins to degrade, treats the pre-decline prefix as the preferred region of the trajectory, and stops reinforcing subsequent low-value continuation.

4. Training pipeline, pluggability, and reasoning compression

SSPO is presented as a pluggable RL process supervision framework that can be attached to rule-based policy optimization methods such as GRPO and DAPO. It is not a replacement for the entire RL pipeline; rather, it supplements rule-based RL with VVP-based stepwise value estimation, self-traced dense advantage computation, and error-step pruning (Xu et al., 18 Aug 2025).

The end-to-end pipeline can be summarized as follows. For each query, the model samples multiple rollouts; in the reported experiments, there are 16 responses per example. Each response is segmented into reasoning steps $n$8. A final rule-based reward $n$9 is computed by exact-match answer checking. For every prefix $T$0, VVP computes the probability of the ground-truth answer, yielding stepwise values $T$1. Rewards and values are normalized, dense advantages are computed, the first value decline is detected, and the policy is optimized with the pruned loss in Eq. (4) (Xu et al., 18 Aug 2025).

The framework’s compression effect is emergent rather than imposed by an explicit token-length penalty. If later steps lower the estimated chance of reaching the correct answer, SSPO stops reinforcing them. The paper argues that this encourages the model to avoid unnecessary exploration and to maintain coherent, efficient reasoning. This is especially important in the comparison with DAPO: in SSPO(DAPO), the authors explicitly remove DAPO’s response-length regularization term in order to isolate SSPO’s effect (Xu et al., 18 Aug 2025).

The paper therefore interprets reasoning compression as a consequence of better process supervision rather than heuristic truncation. "Succinct" is operationalized by the average response length metric, and the method is presented as adapting reasoning length to task needs rather than enforcing a universal cap.

5. Experimental evaluation

The experiments use Qwen2.5-7B-Instruct as the base model, VeRL as the training framework, a learning rate of $T$2, a global batch size of 512, and 16 responses per example for group-relative reward computation. The mathematical reasoning training data come from DAPO-Math-17k, deduplicated to 17,917 question-answer pairs, with AIME-2024 as test set. Medical reasoning uses MedQA-en and MedQA-zh. MedQA-en combines U.S. exams and Taiwan China English exams, with 21,476 train and 2,686 test examples; MedQA-zh combines Mainland China exams and Taiwan Chinese exams, with 38,698 train and 4,839 test examples (Xu et al., 18 Aug 2025).

The main metrics are accuracy and average response length. The headline result is on AIME24 under GRPO: GRPO achieves 18.12% accuracy with average response length 1439.07, while SSPO(GRPO) reaches 19.58% with response length 907.84. The paper highlights this as a 1.46-point improvement in accuracy and a 36.91% reduction in reasoning length (Xu et al., 18 Aug 2025).

The medical results are more mixed. On MedQA-en, GRPO obtains 71.10% accuracy and 425.49 response length, while SSPO(GRPO) obtains 70.40% and 410.29. DAPO obtains 72.04% and 406.82, while SSPO(DAPO) obtains 71.63% and 393.58. On MedQA-zh, GRPO obtains 86.65% and 298.73, while SSPO(GRPO) obtains 86.34% and 292.50. DAPO obtains 86.80% and 303.01, while SSPO(DAPO) obtains 87.19% and 333.41 (Xu et al., 18 Aug 2025).

Setting	Baseline ACC / Len	SSPO ACC / Len
AIME24, GRPO	18.12 / 1439.07	19.58 / 907.84
MedQA-en, GRPO	71.10 / 425.49	70.40 / 410.29
MedQA-en, DAPO	72.04 / 406.82	71.63 / 393.58
MedQA-zh, DAPO	86.80 / 303.01	87.19 / 333.41

The authors describe the overall pattern as comparable or improved accuracy together with substantial reasoning compression, especially relative to GRPO. The strongest evidence is on AIME24. The medical benchmarks show that the trade-off between brevity and correctness is task-dependent rather than uniformly favorable.

The paper also reports entropy trajectory analysis. SSPO maintains similarly high entropy early in training but lower entropy later than the baselines, which the authors interpret as preserving early exploration while reducing erroneous exploration and overthinking later in training (Xu et al., 18 Aug 2025).

6. Relation to adjacent work and acronym disambiguation

Within the 2024–2026 literature, SSPO occupies a specific place among step-wise preference and process-supervision methods. Step-DPO localizes optimization to the first erroneous step in a reasoning chain and conditions the comparison on the preceding correct prefix (Lai et al., 2024). SVPO moves preference learning from the response level to the step level for mathematical reasoning by using MCTS to generate action-value estimates over intermediate reasoning states and by training both a policy and an explicit value model (Chen et al., 2024). Full-Step-DPO extends step-wise optimization to the full reasoning chain by scoring all steps with a self-supervised process reward model and using those scores to weight a DPO-style gradient (Xu et al., 20 Feb 2025). SPPD similarly treats reasoning as an MDP over prefixes and next steps, introducing a dynamic value margin inside a step-DPO objective for self-generated reasoning trees (Yi et al., 19 Feb 2025). PORT, by contrast, applies DPO to prefix-conditioned next-step preferences on external reasoning traces with synthetic negatives rather than to self-traced process supervision (Lahlou et al., 2024). A two-stage MCTS-based self-correction framework likewise combines self-generated correction traces with step-level preference learning, but does so through search-enhanced verification rather than VVP-based pruning (Jiang et al., 2024).

SSPO differs from these approaches in two main respects. First, it does not require an auxiliary reward model, explicit value model, or external stepwise annotations; its stepwise values are generated by the policy model itself through VVP. Second, it emphasizes pruning after the first value decline as a mechanism for reasoning compression rather than only step discrimination or branch ranking. A plausible implication is that SSPO is especially oriented toward controlling overthinking in rule-based RL pipelines, whereas Step-DPO, SVPO, Full-Step-DPO, and SPPD are more directly framed as process-level preference learning for mathematical reasoning.

The acronym is also overloaded. "SSPO" has been used for Semi-Supervised Preference Optimization, a response-level method that combines a small labeled preference dataset with a larger pool of unlabeled prompt-response examples via threshold-based pseudo-labeling (Lee et al., 28 Oct 2025). It has also been used for Subsentence-level Policy Optimization, an RLVR method that applies segment-level importance ratios in PPO-style training rather than self-traced reasoning preferences (Yang et al., 6 Nov 2025). Those papers are distinct from Self-traced Step-wise Preference Optimization.

7. Limitations and significance

The paper does not present SSPO as universally dominant. Several limitations are stated or implied. Because VVP uses the policy model itself to estimate value, the quality of the stepwise signal depends on the model’s own calibration. Error Step Pruning assumes that preferred trajectories should have a monotonically increasing value sequence and that the first decline marks the onset of bad reasoning. The paper notes that real reasoning may temporarily explore, backtrack successfully, or recover after non-monotonic fluctuations, so the pruning rule may be less suitable when productive recovery requires brief value dips. The framework also depends on a rule-based final reward and exact or structured answer matching, making it naturally suited to domains such as math QA and multiple-choice medical QA rather than tasks with ambiguous correctness (Xu et al., 18 Aug 2025).

The empirical gains are also uneven. AIME24 shows both shorter and more accurate reasoning, but some medical settings show slight accuracy drops, and one MedQA-zh configuration yields longer outputs than DAPO. The paper therefore supports a narrower conclusion: self-generated stepwise supervision can preserve or improve performance while often compressing reasoning, but not in every configuration.

Its broader significance lies in the claim that process supervision need not always require external process labels or an auxiliary critic. SSPO provides a concrete recipe for extracting dense stepwise supervision from the policy model itself: probe the probability of the ground-truth answer at every prefix, normalize those values into a GAE-like advantage, identify the first deterioration point, and stop reinforcing the degraded continuation. For reasoning post-training, this reframes overthinking as a process-level optimization failure rather than only a decoding or length-control problem (Xu et al., 18 Aug 2025).