Stepwise Outcome-based Reward Models (SORMs)

• SORMs are a family of methods that integrate outcome-driven signals at each step, offering fine-grained feedback compared to terminal-only reward models.
• They employ techniques like synthetic rollout labeling, token-level attribution, and Monte Carlo methods to estimate value functions and improve decision accuracy.
• Applications in mathematical reasoning, code synthesis, diffusion models, and more demonstrate improved performance, trace correctness, and model alignment.

Stepwise Outcome-based Reward Models (SORMs) are a family of methods for integrating step-level, outcome-driven supervision into sequential decision processes, most prominently in LLMs and diffusion models. SORMs address the limitations of traditional outcome reward models that provide only a global terminal reward by decomposing the reward assignment to provide fine-grained feedback at intermediate steps. This decomposition enables improved stability, interpretability, and efficiency across a variety of reasoning, generation, and alignment tasks.

1. Principles and Motivation

SORMs are defined by the allocation of outcome-based signals (typically the probability of arriving at a correct or preferred final output) to individual intermediate steps within a sequence. Compared to outcome reward models (ORMs), which assign feedback solely at the sequence terminus, SORMs estimate, at each step (or token), the marginal likelihood that the process beginning from that prefix and continuing under the policy will ultimately yield a desirable outcome. This principle is explicit in the estimation of approximated value functions such as

$V^π(S) = \mathbb{E}_{A \sim π(S)}[R(A)] = p(\textrm{is_correct}(A)\mid Q, P_i, π)$

where $P_i$ is a reasoning prefix up to step $i$ (Havrilla et al., 13 Feb 2024), or as in discrete diffusion, the stepwise reward $r_t(x_t,c) = \mathbb{E}[r(x_0, c) \mid x_t, c]$ (Han et al., 7 Jul 2025). The stepwise signal provides the foundation for better credit assignment, efficient exploration of reasoning paths, and stability in optimization with sparse supervised signals.

SORMs are motivated by observed limitations in both outcome- and process-based reward modeling. Pure outcome models provide label efficiency but lack granularity for error localization. Pure process-based rewards require expensive, often impractical step-level human annotation, and are difficult to generalize across tasks (Uesato et al., 2022, Havrilla et al., 13 Feb 2024). SORMs aim to synthesize the advantages of both: label-efficient, outcome-driven supervision distributed to steps, often using synthetic or model-generated rollouts as supervision.

2. Training and Label Generation Methodologies

The various SORMs employ algorithmic and statistical approaches for generating stepwise supervision without fully relying on manual process labels:

• Synthetic Rollout Labeling: For each prefix $P_i$ in a reasoning trajectory, SORMs estimate the chance of eventually reaching a correct answer by continuing policy sampling from $P_i$ and labeling as "positive" if any continuation produces a successful outcome and "negative" otherwise (Havrilla et al., 13 Feb 2024). This operationalizes the value function $V^*(Q,P_i)$.
• Token-level Attribution: In style transfer (Liu et al., 2022), attention maps from a pretrained classifier are used to estimate the salience of each token towards the final style, allowing the attribution of global sequence rewards to individual generation steps.
• Process Reward Models Learning from ORMs: Some works construct process-level reward models by distilling knowledge from outcome models (by aligning score and preference consistency of prefix-level scores to final outcome preferences), establishing consistency between intermediate and full-sequence evaluations (Xie et al., 14 Jun 2025).
• Monte Carlo and Q-value Methods: In agentic multi-step decision scenarios, SORMs estimate per-step Q-values using BeLLMan equations, with synthetic sparse terminal rewards propagated backward (e.g., QLASS (Lin et al., 4 Feb 2025)).

Training losses may be binary cross-entropy on step correctness (Wu et al., 21 Jun 2025), Bradley-Terry pairwise preference losses (Xie et al., 14 Jun 2025), or KL regularization for per-step posterior alignment (as in diffusion preference optimization (Han et al., 7 Jul 2025)).

3. Applications and Empirical Results

SORMs are now central to a variety of domains:

Application Domain	SORM Instantiation / Approach	Key Outcome
Mathematical Reasoning	Step-KTO (Lin et al., 18 Jan 2025), DuaShepherd (Wu et al., 21 Jun 2025)	Up to +10% Pass@1 on MATH500
Code and Program Synthesis	Step-level PRMs, code mutation (Ma et al., 2023)	+4.9% pass@1 on HumanEval
Text Style Transfer	Stepwise reward attribution on token-level (Liu et al., 2022)	SOTA accuracy with 10% data
Diffusion Models	Stepwise alignment and dense trajectory rewards (Zhang et al., 18 Nov 2024, Han et al., 7 Jul 2025)	Robust step generalization
Language Agents/Planning	QLASS Q-guided tree search (Lin et al., 4 Feb 2025)	+5%+ overall reward, improved efficiency
Multimodal Reasoning (Vision)	Multi-dimensional CoT step reward (TriAtt-CoT) (Gao et al., 9 Apr 2025)	+6.3% accuracy on stepwise benchmarks
Medical Reasoning	RAG-augmented step-verification (Yun et al., 13 Jun 2025)	>80% accuracy on MedQA (8B params)

SORMs consistently outperform pure outcome-based RLHF and process-only baselines, with improvements observed in both task accuracy (Pass@1, Test Suite score, etc.) and stepwise trace correctness. In diffusion and sequence generation, SORM-based stepwise alignment is shown to improve both sample quality and generalization to a variable number of steps at inference.

4. Algorithmic and Theoretical Innovations

Multiple works elaborate precise mathematical formulations that underpin modern SORMs:

• BeLLMan Operator and Value Function: SORMs connect with the BeLLMan contraction, showing that back-propagating outcome rewards into stepwise values yields $\epsilon$‐approximate value functions with $O(\log T)$ regret under $(\Delta, \epsilon)$-gap conditions (Chitra, 14 Apr 2025).
• Additive Factorization: In discrete diffusion, the additive decomposition of trajectory-level rewards into stepwise surrogates ensures global optimality if per-step alignment is optimal (Han et al., 7 Jul 2025).
• Pareto dominance for Multi-dimensional Rewards: When reward criteria are multidimensional, as in dynamic generalizable PRMs, stepwise reward assignment employs Pareto-dominance to construct clear positive-negative training pairs (Yin et al., 23 Jul 2025).
• Bidirectional Evaluation: BiPRM (Zhang et al., 3 Aug 2025) uses both left-to-right and right-to-left scoring streams to incorporate global context into step evaluation, reducing error propagation and enhancing trace consistency, achieving up to +31.9% stepwise reward evaluation improvement.

Representative formulas include:

$$\nabla_{\theta_G} J = \mathbb{E} \left[ \frac{1}{n}\sum_{t=1}^n R'_t \nabla_{\theta_G} \log P(y'_t\mid y'_{1:t-1}, x, c; \theta_G) \right]$$

(Liu et al., 2022) and

$$Q(s_t, a_t) = r_t + \gamma \max_{a_{t+1}} Q(s_{t+1}, a_{t+1})$$

(Lin et al., 4 Feb 2025)

5. Design Considerations, Tradeoffs, and Limitations

• Supervision Tradeoff: Pure stepwise process reward models typically require extensive human annotation. By contrast, SORMs use synthetic rollouts for step labeling, improving scalability but potentially propagating errors or model biases from imperfect policies/models (Havrilla et al., 13 Feb 2024).
• Label Efficiency and Generalization: Outcome-only reward models are sample-efficient and easy to construct, but their alignment to intermediate process quality is often insufficient, especially in high-stakes or education settings (Uesato et al., 2022). Empirical studies show that, while SORMs trained from outcome-based signals effectively reduce reasoning errors (differences between correct answers and correct processes), low process/trace error is only guaranteed with explicit process feedback or reward models emulating such feedback.
• Global Context and Robustness: Unidirectional stepwise models may propagate error or ignore corrections suggested by later steps. Bidirectional constructs (BiPRM) and local refinement models attempt to address these limits, but further improvements in handling non-monotonic dependencies and global consistency remain open challenges (Zhang et al., 3 Aug 2025).
• Dynamic Objective Adaptation: Incorporating dynamic and context-sensitive reward criteria, as in DG-PRM (Yin et al., 23 Jul 2025), can improve cross-domain robustness and adaptivity, but may introduce computational complexity due to the need for online reward criterion selection and hierarchical matching.

6. Extensions and Future Research Directions

Emerging directions for SORMs include:

• Hybridization of Process and Outcome Rewards: Approaches such as DuaShepherd (Wu et al., 21 Jun 2025) and LeTS (Learning to Think-and-Search) aim to combine correctness and potential-based signals in a compound, multi-head reward architecture, leveraging both error identification and projected success.
• Automated Dataset Generation: Frameworks increasingly leverage automated programmatic or retrieval-augmented processes (see SVIP (Gao et al., 9 Apr 2025), Med-PRM (Yun et al., 13 Jun 2025)) to generate large-scale, step-annotated reward data in non-text domains (vision, medicine, code) without manual annotation.
• Process Reward Learning from ORMs: Dual-consistency frameworks such as SP-PRM (Xie et al., 14 Jun 2025) show that process reward models can be learned from outcome-based models with synthetic truncations, optimizing for score and preference consistency to improve alignment in reward-guided search at inference.
• Efficient and Reliable RL in Generative Models: In diffusion models, stepwise decomposition provides a tractable alternative to trajectory-level RL objectives, improving tractability and alignment under arbitrary reward functions (Zhang et al., 18 Nov 2024, Han et al., 7 Jul 2025).
• Suppressing Overthinking: Verifiable Stepwise Reward Mechanisms (VSRM) utilize step difference signals for both compressing output and discouraging excessive, redundant computation in over-parameterized reasoning models (Yue et al., 14 Aug 2025).

7. Impact and Broader Implications

SORMs now underpin state-of-the-art performance in categories such as mathematical reasoning, SQL generation, code synthesis, and medical diagnostics across model sizes, with notable data-efficiency and robustness to limited annotation. By bridging outcome-driven supervision with stepwise process modeling, SORMs advance both practical deployment (requiring limited annotation and exhibiting superior stability) and theoretical understanding (formal value approximation and regret bounds).

Stepwise outcome-based reward modeling is likely to remain a central paradigm for model alignment, self-refining reasoning, and verification across both monomodal and multimodal generative systems, enabling a principled balance between efficiency, correctness, and interpretability.