Self-Reset Policy Optimization (SRPO)

• SRPO is a reset-based reinforcement learning method that tackles credit assignment by identifying the first erroneous thought and resampling continuations.
• It employs a two-group rollout strategy to concentrate the learning signal on the suffix following a self-localized reset rather than the entire reasoning trajectory.
• Empirical results show SRPO consistently outperforms methods like GRPO and RRPO on multiple reasoning benchmarks, with modest additional compute overhead.

Searching arXiv for the target paper and closely related reset/credit-assignment work. Self-Reset Policy Optimization (SRPO) is a reset-based reinforcement learning method for language-model reasoning that addresses a central limitation of reinforcement learning with verifiable rewards: standard outcome-based training assigns a single final reward uniformly across all tokens in a reasoning trajectory, even when only a subset of steps caused success or failure. SRPO improves credit assignment by resetting a failed rollout to a meaningful intermediate state—specifically, the first erroneous thought identified by the model itself—and then sampling several alternative continuations from that state. The resulting learning signal is concentrated on the suffix after the reset rather than broadcast across the full trace. Within the paper’s framing, SRPO is analyzed in the Conservative Policy Iteration (CPI) framework, compared against Group Relative Policy Optimization (GRPO) and Random-Reset Policy Optimization (RRPO), and shown to consistently outperform both on reasoning benchmarks by using only the model itself with no external supervision (Samanta et al., 25 May 2026).

1. Motivation and problem setting

SRPO was introduced for multi-step reasoning under reinforcement learning with verifiable reward, where contemporary methods typically post-train LLMs by assigning a single outcome reward uniformly across all tokens in a trajectory. The paper identifies this as a credit-assignment problem: a final reward indicates whether the entire chain succeeded, but not which intermediate decisions were responsible. In structured reasoning tasks, this coarse supervision can cause optimization to reinforce or penalize large portions of a trajectory that are not causally decisive (Samanta et al., 25 May 2026).

The method is formulated in a thought-level Markov decision process rather than a token-level one. The initial state is the prompt, $x_0 \sim \mu$. At step $h$, the model emits a semantically coherent reasoning step,

$$\tilde y_h \sim \pi_h(\cdot \mid x_0, \tilde y_{1:h-1}),$$

where each thought is a token sequence delimited by a stop pattern. Reward is sparse: $r_h = 0$ for $h < H$, and $r_H \in \{0,1\}$ comes from oracle verification of the final answer. The paper argues that thought-level actions are a more natural unit for sparse-reward reasoning than tokens, while avoiding externally imposed segment boundaries.

Within this setting, SRPO treats failed reasoning traces not merely as negative examples but as trajectories that can be revisited at a more informative state. Its central claim is that a reset allows the learner to re-enter a previously visited state and resample a continuation, so that outcome differences can be attributed to decisions made from that state. This places SRPO in a distinct category relative to full-trajectory RLVR methods such as GRPO.

2. Reset-based credit assignment

SRPO differs from standard outcome-based RL methods by changing both where training data are collected and which tokens receive gradient. The paper presents the conceptual progression as follows.

Method	Reset point	Credit assignment
GRPO	No reset	Final reward is broadcast across all tokens
RRPO	Random point in a failed trace	Loss is applied only to continuation tokens after the random reset
SRPO	Self-localized first erroneous thought	Loss is applied only to continuation tokens after the self-localized reset

SRPO begins from a failed rollout and prompts the same policy to identify the first erroneous thought in its own chain. The prompt asks for the first step that cannot be justified from the problem statement and earlier verified steps. If the model returns index $h^*$, the reset state is

$x^* \gets (x_0, \tilde y_{1:h^*-1}),$

which is the verified-correct prefix immediately before the first error (Samanta et al., 25 May 2026).

Training uses two groups per prompt. Group 1 is a standard GRPO-like base group of $G$ iid rollouts from the prompt $x_0$. The procedure is explicit: sample rollouts from $h$0 until the first incorrect one appears; use all preceding correct rollouts as initial members of group 1; discard the seed failed rollout; and sample additional iid rollouts from $h$1 until group 1 has size $h$2. Group 2 is the shared-prefix reset group: starting from the failed seed trajectory, identify $h$3, form $h$4, and sample $h$5 iid continuations from $h$6. These continuations share the prefix and differ only in the newly sampled suffix.

For each group $h$7, rewards are normalized by group-relative statistics,

$h$8

The model then applies a GRPO-style loss over all $h$9 rollouts, but for the shared-prefix group it masks out the prefix tokens so that only suffix tokens receive gradient. The shared-prefix loss is

$$\tilde y_h \sim \pi_h(\cdot \mid x_0, \tilde y_{1:h-1}),$$0

A defining property of SRPO is that it does not sample only one counterfactual continuation from the reset state. It samples multiple iid suffix rollouts from $$\tilde y_h \sim \pi_h(\cdot \mid x_0, \tilde y_{1:h-1}),$$1, creating a shared-prefix group whose reward differences help attribute credit to the suffix actions. Under an 8-rollout budget, the best-performing split is $$\tilde y_h \sim \pi_h(\cdot \mid x_0, \tilde y_{1:h-1}),$$2: 4 base rollouts and 4 shared-prefix rollouts from one self-localized reset state.

3. CPI analysis and the oracle view of resets

The paper situates SRPO in the Conservative Policy Iteration framework in order to formalize why reset-based credit assignment can improve over random resets. Standard CPI can be written as a conservative mixture update,

$$\tilde y_h \sim \pi_h(\cdot \mid x_0, \tilde y_{1:h-1}),$$3

with the classical lower bound

$$\tilde y_h \sim \pi_h(\cdot \mid x_0, \tilde y_{1:h-1}),$$4

The random-reset variant is denoted CPI-RR (Samanta et al., 25 May 2026).

To model an ideal reset mechanism, the paper introduces a credit-assignment oracle that identifies whether a state is improvable. For threshold $$\tilde y_h \sim \pi_h(\cdot \mid x_0, \tilde y_{1:h-1}),$$5, the $$\tilde y_h \sim \pi_h(\cdot \mid x_0, \tilde y_{1:h-1}),$$6-improvable set at time $$\tilde y_h \sim \pi_h(\cdot \mid x_0, \tilde y_{1:h-1}),$$7 is

$$\tilde y_h \sim \pi_h(\cdot \mid x_0, \tilde y_{1:h-1}),$$8

with coverage

$$\tilde y_h \sim \pi_h(\cdot \mid x_0, \tilde y_{1:h-1}),$$9

The oracle is the membership test

$r_h = 0$0

This yields a credit-assignment greedy policy that behaves like the greedy policy on improvable states and like the original policy elsewhere: $r_h = 0$1 The associated decomposition is

$r_h = 0$2

and for the credit-assignment greedy policy,

$r_h = 0$3

The key structural result is the credit-aware state-distribution drift bound: $r_h = 0$4 Plugging this into the performance-difference argument gives

$r_h = 0$5

The paper’s interpretation is that if updates are concentrated on improvable states, the signal is not diluted by random sampling over all visited states.

Under a realizable finite function class $r_h = 0$6 and bounded rewards, the main theorem compares CPI-RR with an oracle-based method. CPI-RR requires

$r_h = 0$7

to achieve

$r_h = 0$8

whereas the oracle-based method requires

$r_h = 0$9

to achieve

$h < H$0

The paper summarizes this as a sample-complexity improvement by $h < H$1 and a per-iteration improvement by $h < H$2. SRPO itself does not have access to the oracle; rather, self-localization is presented as a practical approximation.

4. Empirical performance and benchmark profile

The experiments train two base models, Qwen2.5-14B-Instruct and OLMo-3-7B-Instruct, on NuminaMath-Olympiads, using 400 problems for 2 epochs. Evaluation covers a 10-benchmark suite with 500 test items per benchmark, 3 seeds, and temperature 0 at evaluation: NuminaMath-Olympiads, HMMT Nov 2025, MATH Level-5, StrategyQA, AceReason-Math, SciKnowEval Chemistry, SciKnowEval Biology, CommonsenseQA, SciKnowEval Materials, and SciKnowEval Physics. The comparison set includes GRPO, SCoRe, Cr-GRPO, SPO-Tree, RRPO, and SRPO (Samanta et al., 25 May 2026).

The headline result is that SRPO is the strongest non-baseline method on both base models. For Qwen2.5-14B-Instruct, the paper states that SRPO gets the best result on 7/10 tasks. For OLMo-3-7B-Instruct, SRPO is best on 6/10 tasks. A notable pattern is that RRPO is usually comparable to GRPO, while SRPO is consistently stronger. The paper explicitly interprets this as evidence that self-localization matters, not just resetting.

The compute-split ablation tests three schedules under a fixed 8-rollout budget: 1×4, 2×4, and 1×8. The 1×4 split performs best on most benchmarks for both models and is adopted as the default. This suggests that a single self-localized reset state with multiple suffix samples is more effective than spreading the same budget across more reset prefixes.

The paper also extends SRPO to code reasoning on LiveCodeBench v6 medium with 383 problems, using 1 epoch and a fixed 8-rollout budget. The train-time verifier uses a split of the unit tests, with the remaining tests held out for evaluation. On this benchmark, SRPO converges to a higher pass rate and reaches matching pass rates 2–3× faster than GRPO and RRPO. RRPO again tracks GRPO more closely, reinforcing the same interpretation.

From an efficiency perspective, SRPO is more expensive than GRPO, but the shared-prefix mechanism limits the additional cost because the second group reuses the prefix and only samples a suffix. On LiveCodeBench-medium, SRPO used about 1.31× total wall clock relative to GRPO, compared to 1.53× for RRPO.

5. Implementation details, failure modes, and limitations

Several implementation choices are central to SRPO’s behavior. The paper states that PPO-style clipping is not used by default in the main SRPO loss; an ablation shows clipping does not help and often hurts. For shared-prefix rollouts, prefix masking is mandatory: the gradient is masked out on the shared prefix, and this is described as central to SRPO’s credit assignment. Rollout generation is done by generating each thought as a separate vLLM call, with temperature $h < H$3 and top-$h < H$4. The training configuration uses LoRA adapters with rank 64 and $h < H$5, AdamW, learning rate $h < H$6, no warmup, weight decay 0.01, and gradient clipping at 1.0. The paper also sets max prompt length 2048, max response length 4096, max thoughts per chain 20, and max tokens per thought 256 (Samanta et al., 25 May 2026).

The main limitation is that self-localization is the bottleneck. SRPO depends on the model’s ability to find the first erroneous step, and the paper reports that clean prefixes correct nearly 2× as often as erroneous ones in the coding analysis. On this basis, SRPO is described as only an approximation to the ideal credit-assignment oracle.

A second limitation is structural: the method assumes tasks decompose into identifiable reasoning thoughts with clear reset states. The paper therefore notes that it may not transfer cleanly to tasks without such structure. A third limitation is that RRPO and SRPO are designed for RLVR and require verifiable rewards. Extending reset-based methods to non-verifiable tasks remains open. Finally, the theory is single-step or one-iteration: the CPI analysis establishes improvement guarantees for one policy-update step, while a full convergence analysis for credit-aware policy optimization is left open.

A common misconception is that any reset mechanism should suffice once suffix-only optimization is introduced. The reported RRPO results do not support that view. The empirical pattern that RRPO is usually comparable to GRPO, while SRPO is consistently stronger, suggests that the gain depends on concentrating resets near genuinely improvable states rather than resetting at arbitrary points.

6. Terminology and related uses of “SRPO”

The acronym SRPO is overloaded across recent machine learning literature, and Self-Reset Policy Optimization should be distinguished from several unrelated methods.

In language-model post-training, "Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing" uses SRPO to denote Sample-Routed Policy Optimization, an on-policy RLVR method that routes correct samples to GRPO and failed samples to SDPO, with entropy-aware weighting on the distillation branch (Li et al., 2 Apr 2026). In safety alignment, "Self-ReSET: Learning to Self-Recover from Unsafe Reasoning Trajectories" presents a pure reinforcement learning framework that stores unsafe prefixes from the model’s own on-policy reasoning in a replay buffer and trains the model to continue from those unsafe prefixes toward safe final outputs (Zhang et al., 9 May 2026). Both address trajectory-level correction, but neither is the reset-based suffix-resampling method defined in Self-Reset Policy Optimization.

Outside LLM reasoning, earlier and parallel work assigns the same acronym to different objects. "State Regularized Policy Optimization on Data with Dynamics Shift" uses SRPO for a method that regularizes a policy toward a stationary state distribution learned from other dynamics, rather than toward a behavior policy (Xue et al., 2023). "Self-Improving Robust Preference Optimization" uses SRPO for an offline RLHF framework that formulates preference learning as a self-improvement process with a min-max objective and iterative self-revision at inference time (Choi et al., 2024). "SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning" uses SRPO for Self-Reflection enhanced reasoning with Group Relative Policy Optimization, a two-stage reflection-aware framework for multimodal reasoning (Wan et al., 2 Jun 2025). "SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models" uses SRPO for a VLA-RL framework that assigns progress-wise rewards to failed trajectories by comparing them to successful in-batch trajectories in latent world space (Fei et al., 19 Nov 2025).

This terminological overlap suggests that citations should use the full expansion rather than the acronym alone. In the specific sense of (Samanta et al., 25 May 2026), Self-Reset Policy Optimization denotes a reset-based credit-assignment method for reasoning trajectories that self-localizes the first erroneous thought, resamples multiple suffixes from that state, and applies learning only to the continuation.