Step-SRPO: Diverse Approaches in Policy Optimization

• Step-SRPO is a collective term encompassing methods that replace whole-trajectory processing with staged pipelines, token-level corrections, or two-stage training approaches.
• The literature shows that 'step' can mean pipeline decomposition, decision granularity, or structured reasoning stages, each affecting credit assignment and reward construction differently.
• Empirical studies report that methods under the Step-SRPO umbrella improve performance metrics, such as PSNR, SSIM, and success rates, across super-resolution and reinforcement learning tasks.

“Step-SRPO” is not the official title of a single algorithm in the cited literature. The term is more accurately understood as a shorthand applied to several distinct methods whose published names include SRPO, or to neighboring step-level frameworks that solve similar credit-assignment problems. Across these works, “step” can mean a staged pipeline, step-level credit assignment, thought-level reset localization, token-level or role-aware weighting, progress-wise reward construction, or two-stage training. A plausible implication is that “Step-SRPO” functions primarily as a disambiguation label rather than as a canonical method name (Gu et al., 2022, Li et al., 2 Apr 2026, Samanta et al., 25 May 2026).

1. Terminological status and major usages

The literature attaches the acronym SRPO to multiple non-equivalent methods. Several papers explicitly state that they do not define a separate variant called “Step-SRPO,” even when their method is naturally stepwise or step-aware (Gu et al., 2022, Wan et al., 2 Jun 2025, Li et al., 2 Apr 2026).

Published name	Domain	Closest “step” interpretation
“Super-Resolution by Predicting Offsets” (Gu et al., 2022)	Rasterized graphics SR	Staged pipeline: edge offsets, interpolation, fusion
“Sample-Routed Policy Optimization” (Li et al., 2 Apr 2026)	RLVR for LLMs	Sample-level routing; token-level correction inside failed samples
“Structured Role-aware Policy Optimization” (Jiang et al., 8 May 2026)	Multimodal RLVR	Structured <perception> and <reasoning> steps with token-level reweighting
“Self-Reset Policy Optimization” (Samanta et al., 25 May 2026)	Reasoning RLVR	Thought-level reset localization and suffix-only optimization
“Self-Referential Policy Optimization” (Fei et al., 19 Nov 2025)	Vision-Language-Action RL	Progress-wise reward; offline variant with timestep-wise incremental progress
“SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning” (Wan et al., 2 Jun 2025)	Multimodal reasoning	Stage-structured answer → reflection → revision
“SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM” (Zhang et al., 19 Apr 2025)	Math/code RL for LLMs	Two-stage training plus history resampling
“Safety-aware Reasoning Path Optimization” (Cai et al., 15 Sep 2025)	Multimodal safety	Stepwise contrastive optimization over reasoning states

This multiplicity matters because the word “step” is not used uniformly. In some works it refers to a pipeline decomposition; in others it denotes a decision granularity for optimization; in still others it is only an informal way to describe a two-stage schedule or iterative self-revision. The result is a terminological landscape in which the same query can point to substantially different mathematical objects and training procedures.

2. The staged raster-graphics interpretation

In the graphics paper “Super-Resolution by Predicting Offsets,” the authors state that they do not define a separate method called “Step-SRPO.” The closest faithful interpretation is the paper’s own staged SRPO pipeline for real-time super-resolution of rasterized graphics images (Gu et al., 2022). The method is motivated by the claim that rasterized graphics differ fundamentally from natural-image SR: under very tight compute budgets, the perceptually dominant structures are often geometric edges, and the difference between low-resolution and high-resolution rasterized images is often not new texture but edge pixels that should switch to a neighboring region’s color.

The core representation is an offset field. Given an LR rasterized RGB image $I^{LR}\in\mathbb{R}^{3\times H\times W}$, the network predicts

$$\Omega = F(I^{LR}), \qquad \Omega \in \mathbb{R}^{2\times sH\times sW},$$

where $s$ is the upscaling factor (Gu et al., 2022). These offsets are used to form an offset-based SR image for edge regions, while flatter regions are reconstructed with Lanczos interpolation. The final image is fused by

$$I^{SR} = \alpha \odot I^{SR}_{\Omega} + (1-\alpha)\odot I^{SR}_{\uparrow},$$

with the blending mask derived from rounded nonzero offsets: $$\alpha_{[i,j]} = \omega_k \otimes \mathbb{I}\left\{\lfloor \Omega_{[i,j]}\rceil \neq 0\right\}.$$ The paper uses a $3\times 3$ Gaussian blur filter with standard deviation $1.5$ to soften this edge mask (Gu et al., 2022).

The architecture is intentionally minimal: a three-layer convolutional network with two $5\times5$ convolutions before nearest-neighbor feature upsampling and a final convolution after upsampling. It predicts only a 2D offset vector per HR pixel and has exactly 8,434 parameters (Gu et al., 2022). Training is self-supervised with an SSIM-based loss on the projected SR rasterized image rather than $L_1$ or $L_2$, and the model is trained on 300 images of random geometric objects with random gradient colors. For $$\Omega = F(I^{LR}), \qquad \Omega \in \mathbb{R}^{2\times sH\times sW},$$0 SR, SRPO reports 8.43K params, 8.50 G FLOPs, PSNR 26.97, SSIM 0.8532, and LPIPS 0.1501, with the paper explicitly noting that LPIPS is the best in its comparison table (Gu et al., 2022).

Within this interpretation, “step” means the ordered sequence offset prediction → edge reconstruction → flat-region interpolation → guided fusion. It does not denote step-level reward modeling or process supervision.

3. Sample-routed and stage-structured post-training meanings

A very different usage appears in “Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing,” where SRPO stands for Sample-Routed Policy Optimization. The paper explicitly states that the method is not step-level routing; routing is fundamentally sample-level / rollout-level, although the SDPO branch applies token-level weighting inside routed failed samples (Li et al., 2 Apr 2026). The routing indicators are

$$\Omega = F(I^{LR}), \qquad \Omega \in \mathbb{R}^{2\times sH\times sW},$$1

where $$\Omega = F(I^{LR}), \qquad \Omega \in \mathbb{R}^{2\times sH\times sW},$$2 indicates whether rollout $$\Omega = F(I^{LR}), \qquad \Omega \in \mathbb{R}^{2\times sH\times sW},$$3 is correct and $$\Omega = F(I^{LR}), \qquad \Omega \in \mathbb{R}^{2\times sH\times sW},$$4 indicates whether teacher information is available (Li et al., 2 Apr 2026). The final routed objective combines GRPO and dynamically weighted SDPO: $$\Omega = F(I^{LR}), \qquad \Omega \in \mathbb{R}^{2\times sH\times sW},$$5 Entropy-aware weights use the teacher entropy

$$\Omega = F(I^{LR}), \qquad \Omega \in \mathbb{R}^{2\times sH\times sW},$$6

with default $$\Omega = F(I^{LR}), \qquad \Omega \in \mathbb{R}^{2\times sH\times sW},$$7 (Li et al., 2 Apr 2026). On Qwen3-8B, SRPO raises the five-benchmark average to 77.4%, which the paper reports as +3.4 over GRPO and +6.3 over SDPO (Li et al., 2 Apr 2026). In this line of work, “step” is at most an informal reference to denser per-token supervision inside failed samples; the routing decision itself remains at the sample level.

Another stage-structured meaning appears in “SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning,” where SRPO denotes Self-Reflection enhanced reasoning with Group Relative Policy Optimization. That paper explicitly states that it does not define a separate “Step-SRPO” and that reflection is not optimized as a reward on every individual intermediate reasoning step (Wan et al., 2 Jun 2025). Instead, the model is trained to emit a structured sequence

$$\Omega = F(I^{LR}), \qquad \Omega \in \mathbb{R}^{2\times sH\times sW},$$8

with RL outputs following $$I^{SR} = \alpha \odot I^{SR}_{\Omega} + (1-\alpha)\odot I^{SR}_{\uparrow},$$9 (Wan et al., 2 Jun 2025). The total reward is decomposed as

$$\Omega = F(I^{LR}), \qquad \Omega \in \mathbb{R}^{2\times sH\times sW},$$9

with reflection effectiveness defined by a discrete indicator that rewards preserving a correct answer, correcting a wrong answer, or penalizes changing a right answer into a wrong one (Wan et al., 2 Jun 2025). Here “step” means a stage-structured response format, not a formal step-level verifier or per-step RL objective.

4. Step-level reasoning and agentic credit-assignment interpretations

The closest literal step-level SRPO in the reasoning literature is “Credit Assignment with Resets in LLM Reasoning,” where SRPO stands for Self-Reset Policy Optimization and the paper explicitly describes the intended concept as step-localized SRPO at the granularity of thought-level steps (Samanta et al., 25 May 2026). The setting is a Thought MDP in which the atomic action is a coherent reasoning thought. SRPO generates a failed seed trajectory, asks the model to self-localize the first erroneous thought, resets to the prefix before that thought,

$s$0

samples multiple alternative suffixes from that reset state, and applies updates only to suffix tokens. The shared-prefix loss is

$s$1

where $s$2 is the group-normalized reward within the shared-prefix rollout set (Samanta et al., 25 May 2026). The paper reports that SRPO is the best non-baseline method on 7/10 tasks for Qwen2.5-14B-Instruct and 6/10 tasks for OLMo-3-7B-Instruct, and that on LiveCodeBench v6 medium it reaches matching pass rates 2–3× faster than GRPO and RRPO (Samanta et al., 25 May 2026).

A related but distinct step-structured formulation is “Structured Role-aware Policy Optimization,” where SRPO denotes Structured Role-aware Policy Optimization for multimodal RLVR (Jiang et al., 8 May 2026). The response is explicitly decomposed into $$\alpha_{[i,j]} = \omega_k \otimes \mathbb{I}\left\{\lfloor \Omega_{[i,j]}\rceil \neq 0\right\}.$$0 and the sequence-level GRPO advantage $s$3 is refined into token-level, role-aware advantages

$s$4

with strictly positive weights $s$5 derived from self-distilled on-policy contrasts (Jiang et al., 8 May 2026). Perception tokens are weighted by visual dependency under original versus corrupted images, while reasoning tokens are weighted by consistency with the generated perception. In this interpretation, “step” refers to the structured perception step and reasoning step, together with role-aware credit redistribution rather than reset-based counterfactual repair.

The broader step-aware landscape includes methods that are not named SRPO but illuminate how the term is often used. “Let’s Reward Step-by-Step: Step-Aware Contrastive Alignment for Vision-Language Navigation in Continuous Environments” introduces SACA, whose Perception-Grounded Step-Aware auditor computes a continuous step score $s$6, a process score $s$7, a binary mask $s$8, and a divergence point $s$9, then applies prefix cloning, divergence-point contrastive correction, and suffix repair (Li et al., 10 Mar 2026). “StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning” argues that agentic RL should replace the token-level MDP with a step-level MDP, with step-level GAE

$$I^{SR} = \alpha \odot I^{SR}_{\Omega} + (1-\alpha)\odot I^{SR}_{\uparrow},$$0

and a step-level importance ratio

$$I^{SR} = \alpha \odot I^{SR}_{\Omega} + (1-\alpha)\odot I^{SR}_{\uparrow},$$1

(Wang et al., 20 Apr 2026). These works are not SRPO papers, but they sharpen the step-level sense in which “Step-SRPO” is often interpreted.

5. Progress-wise and two-stage training interpretations

In embodied control, “SRPO” can denote Self-Referential Policy Optimization rather than step-level reset or token routing. The VLA paper “SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models” states that the phrase “Step-SRPO” does not appear verbatim, but the closest concept is SRPO’s progress-wise reward construction from successful trajectories in the current batch (Fei et al., 19 Nov 2025). Successful trajectories are embedded with a world-model encoder,

$$I^{SR} = \alpha \odot I^{SR}_{\Omega} + (1-\alpha)\odot I^{SR}_{\uparrow},$$2

clustered with DBSCAN, and failed trajectories receive a distance-based reward via the nearest success prototype

$$I^{SR} = \alpha \odot I^{SR}_{\Omega} + (1-\alpha)\odot I^{SR}_{\uparrow},$$3

In the main online method this reward is trajectory-level, but the appendix real-world offline variant makes the stepwise interpretation explicit through

$$I^{SR} = \alpha \odot I^{SR}_{\Omega} + (1-\alpha)\odot I^{SR}_{\uparrow},$$4

Empirically, the paper reports that SRPO starts from a supervised baseline of 48.9% and reaches 99.2% average success on LIBERO in 200 RL steps, with a 167% performance improvement on LIBERO-Plus (Fei et al., 19 Nov 2025). Here “step” refers to progress estimation more than to explicit thought-level process supervision.

A different staged meaning appears in “SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM,” where SRPO expands to two-Staged history-Resampling Policy Optimization (Zhang et al., 19 Apr 2025). The method is built on GRPO, but introduces two-stage cross-domain training and History Resampling. Stage 1 uses solely challenging mathematical data to elicit long chain-of-thought behavior; Stage 2 introduces coding data for mixed-domain capability integration. The GRPO backbone uses

$$I^{SR} = \alpha \odot I^{SR}_{\Omega} + (1-\alpha)\odot I^{SR}_{\uparrow},$$5

while the paper reports that nearly 50% of sampled groups in vanilla GRPO have identical rewards and therefore provide little useful gradient (Zhang et al., 19 Apr 2025). SRPO removes prompts whose rollout groups are all-correct at epoch end, keeps mixed and all-incorrect prompts, runs Stage 1 for 840 steps, and reaches final reported scores of 50.0 on AIME24 and 41.6 on LiveCodeBench in 2,000 steps total, surpassing DeepSeek-R1-Zero-Qwen-32B while using 20% of DeepSeek’s training steps (Zhang et al., 19 Apr 2025). In this usage, “step” refers to a two-stage training schedule and a step-by-step algorithmic recipe, not to step-level credit assignment.

6. Related step-level preference optimization and overall conceptual pattern

The step-level sense of “Step-SRPO” also overlaps with methods outside the SRPO acronym family. “Step-level Value Preference Optimization for Mathematical Reasoning” introduces SVPO, which uses MCTS to build step-level preference pairs and trains a value model with

$$I^{SR} = \alpha \odot I^{SR}_{\Omega} + (1-\alpha)\odot I^{SR}_{\uparrow},$$6

alongside a DPO-style policy objective (Choi et al., 2024). “STEP: Success-rate-aware Trajectory-Efficient Policy Optimization” performs success-rate-guided trajectory replacement and step-level decomposition in GUI-agent RL, with adaptive replacement

$$I^{SR} = \alpha \odot I^{SR}_{\Omega} + (1-\alpha)\odot I^{SR}_{\uparrow},$$7

and step-level advantages for successful trajectories only (Chen et al., 17 Nov 2025). “Safety-aware Reasoning Path Optimization” represents reasoning as a sequence of states $$I^{SR} = \alpha \odot I^{SR}_{\Omega} + (1-\alpha)\odot I^{SR}_{\uparrow},$$8 and applies a step-level contrastive loss between positive and negative continuations from the same prefix, again using an acronym SRPO but with yet another expansion (Cai et al., 15 Sep 2025).

Across all of these works, two recurring patterns dominate. First, “step” frequently denotes a decomposition of a complex trajectory into semantically meaningful units—edge versus flat regions, perception versus reasoning, valid prefix versus divergent suffix, initial answer versus reflection versus revision, or timestep-wise progress increments. Second, the acronym SRPO itself is highly overloaded: it can mean Super-Resolution by Predicting Offsets, Sample-Routed Policy Optimization, Structured Role-aware Policy Optimization, Self-Reset Policy Optimization, Self-Referential Policy Optimization, Self-Reflection enhanced reasoning with Group Relative Policy Optimization, two-Staged history-Resampling Policy Optimization, or Safety-aware Reasoning Path Optimization (Gu et al., 2022, Li et al., 2 Apr 2026, Jiang et al., 8 May 2026, Samanta et al., 25 May 2026, Fei et al., 19 Nov 2025, Wan et al., 2 Jun 2025, Zhang et al., 19 Apr 2025, Cai et al., 15 Sep 2025).

The most defensible encyclopedia-level conclusion is therefore not that “Step-SRPO” names one settled method, but that it identifies a family resemblance across several lines of work: replacing coarse whole-trajectory treatment with some form of staged decomposition, local routing, role-aware weighting, reset-based repair, or progress-sensitive optimization. A plausible implication is that scholarly use of the term requires explicit expansion or citation of the intended paper, because the underlying algorithms, objectives, and domains are otherwise not interchangeable.