---
title: Stepwise Outcome-based Reward Models (SORMs)
url: https://www.emergentmind.com/topics/stepwise-outcome-based-reward-models-sorms
type: topic
---

# Stepwise Outcome-based Reward Models (SORMs)

Stepwise Outcome-based Reward Models (SORMs) are a family of methods for integrating step-level, outcome-driven supervision into sequential decision processes, most prominently in large language models and diffusion models. SORMs address the limitations of traditional outcome reward models that provide only a global terminal reward by decomposing the reward assignment to provide fine-grained feedback at intermediate steps. This decomposition enables improved stability, interpretability, and efficiency across a variety of reasoning, generation, and alignment tasks.

## 1. Principles and Motivation

SORMs are defined by the allocation of outcome-based signals (typically the probability of arriving at a correct or preferred final output) to individual intermediate steps within a sequence. Compared to outcome reward models (ORMs), which assign feedback solely at the sequence terminus, SORMs estimate, at each step (or token), the marginal likelihood that the process beginning from that prefix and continuing under the policy will ultimately yield a desirable outcome. This principle is explicit in the estimation of approximated value functions such as
$$
V^π(S) = \mathbb{E}_{A \sim π(S)}[R(A)] = p(\textrm{is_correct}(A)\mid Q, P_i, π)
$$
where $P_i$ is a reasoning prefix up to step $i$ [2402.10963], or as in discrete diffusion, the stepwise reward $r_t(x_t,c) = \mathbb{E}[r(x_0, c) \mid x_t, c]$ [2507.04832]. The stepwise signal provides the foundation for better credit assignment, efficient exploration of reasoning paths, and stability in optimization with sparse supervised signals.

SORMs are motivated by observed limitations in both outcome- and process-based reward modeling. Pure outcome models provide label efficiency but lack granularity for error localization. Pure process-based rewards require expensive, often impractical step-level human annotation, and are difficult to generalize across tasks [2211.14275, 2402.10963]. SORMs aim to synthesize the advantages of both: label-efficient, outcome-driven supervision distributed to steps, often using synthetic or model-generated rollouts as supervision.

## 2. Training and Label Generation Methodologies

The various SORMs employ algorithmic and statistical approaches for generating stepwise supervision without fully relying on manual process labels:
- **Synthetic Rollout Labeling:** For each prefix $P_i$ in a reasoning trajectory, SORMs estimate the chance of eventually reaching a correct answer by continuing policy sampling from $P_i$ and labeling as "positive" if any continuation produces a successful outcome and "negative" otherwise [2402.10963]. This operationalizes the value function $V^*(Q,P_i)$.
- **Token-level Attribution:** In style transfer [2205.09324], attention maps from a pretrained classifier are used to estimate the salience of each token towards the final style, allowing the attribution of global sequence rewards to individual generation steps.
- **Process Reward Models Learning from ORMs:** Some works construct process-level reward models by distilling knowledge from outcome models (by aligning score and preference consistency of prefix-level scores to final outcome preferences), establishing consistency between intermediate and full-sequence evaluations [2506.12446].
- **Monte Carlo and Q-value Methods:** In agentic multi-step decision scenarios, SORMs estimate per-step Q-values using Bellman equations, with synthetic sparse terminal rewards propagated backward (e.g., QLASS [2502.02584]).

Training losses may be binary cross-entropy on step correctness [2506.17533], Bradley-Terry pairwise preference losses [2506.12446], or KL regularization for per-step posterior alignment (as in diffusion preference optimization [2507.04832]).

## 3. Applications and Empirical Results

SORMs are now central to a variety of domains:

| Application Domain             | SORM Instantiation / Approach                       | Key Outcome                          |
|-------------------------------|-----------------------------------------------------|--------------------------------------|
| Mathematical Reasoning        | Step-KTO [2501.10799], DuaShepherd [2506.17533]     | Up to +10% Pass@1 on MATH500         |
| Code and Program Synthesis    | Step-level PRMs, code mutation [2310.10080]         | +4.9% pass@1 on HumanEval            |
| Text Style Transfer           | Stepwise reward attribution on token-level [2205.09324] | SOTA accuracy with 10% data      |
| Diffusion Models              | Stepwise alignment and dense trajectory rewards [2411.11727, 2507.04832] | Robust step generalization          |
| Language Agents/Planning      | QLASS Q-guided tree search [2502.02584]             | +5%+ overall reward, improved efficiency |
| Multimodal Reasoning (Vision) | Multi-dimensional CoT step reward (TriAtt-CoT) [2504.06606] | +6.3% accuracy on stepwise benchmarks|
| Medical Reasoning             | RAG-augmented step-verification [2506.11474]        | >80% accuracy on MedQA (8B params)   |

SORMs consistently outperform pure outcome-based RLHF and process-only baselines, with improvements observed in both task accuracy (Pass@1, Test Suite score, etc.) and stepwise trace correctness. In diffusion and sequence generation, SORM-based stepwise alignment is shown to improve both sample quality and generalization to a variable number of steps at inference.

## 4. Algorithmic and Theoretical Innovations

Multiple works elaborate precise mathematical formulations that underpin modern SORMs:
- **Bellman Operator and Value Function:** SORMs connect with the Bellman contraction, showing that back-propagating outcome rewards into stepwise values yields $\epsilon$‐approximate value functions with $O(\log T)$ regret under $(\Delta, \epsilon)$-gap conditions [2504.09777].
- **Additive Factorization:** In discrete diffusion, the additive decomposition of trajectory-level rewards into stepwise surrogates ensures global optimality if per-step alignment is optimal [2507.04832].
- **Pareto dominance for Multi-dimensional Rewards:** When reward criteria are multidimensional, as in dynamic generalizable PRMs, stepwise reward assignment employs Pareto-dominance to construct clear positive-negative training pairs [2507.17849].
- **Bidirectional Evaluation:** BiPRM [2508.01682] uses both left-to-right and right-to-left scoring streams to incorporate global context into step evaluation, reducing error propagation and enhancing trace consistency, achieving up to +31.9% stepwise reward evaluation improvement.

Representative formulas include:
$$
\nabla_{\theta_G} J = \mathbb{E} \left[ \frac{1}{n}\sum_{t=1}^n R'_t \nabla_{\theta_G} \log P(y'_t\mid y'_{1:t-1}, x, c; \theta_G) \right]
$$
[2205.09324]
and
$$
Q(s_t, a_t) = r_t + \gamma \max_{a_{t+1}} Q(s_{t+1}, a_{t+1})
$$
[2502.02584]

## 5. Design Considerations, Tradeoffs, and Limitations

- **Supervision Tradeoff:** Pure stepwise process reward models typically require extensive human annotation. By contrast, SORMs use synthetic rollouts for step labeling, improving scalability but potentially propagating errors or model biases from imperfect policies/models [2402.10963].
- **Label Efficiency and Generalization:** Outcome-only reward models are sample-efficient and easy to construct, but their alignment to intermediate process quality is often insufficient, especially in high-stakes or education settings [2211.14275]. Empirical studies show that, while SORMs trained from outcome-based signals effectively reduce reasoning errors (differences between correct answers and correct processes), low process/trace error is only guaranteed with explicit process feedback or reward models emulating such feedback.
- **Global Context and Robustness:** Unidirectional stepwise models may propagate error or ignore corrections suggested by later steps. Bidirectional constructs (BiPRM) and local refinement models attempt to address these limits, but further improvements in handling non-monotonic dependencies and global consistency remain open challenges [2508.01682].
- **Dynamic Objective Adaptation:** Incorporating dynamic and context-sensitive reward criteria, as in DG-PRM [2507.17849], can improve cross-domain robustness and adaptivity, but may introduce computational complexity due to the need for online reward criterion selection and hierarchical matching.

## 6. Extensions and Future Research Directions

Emerging directions for SORMs include:
- **Hybridization of Process and Outcome Rewards:** Approaches such as DuaShepherd [2506.17533] and LeTS (Learning to Think-and-Search) aim to combine correctness and potential-based signals in a compound, multi-head reward architecture, leveraging both error identification and projected success.
- **Automated Dataset Generation:** Frameworks increasingly leverage automated programmatic or retrieval-augmented processes (see SVIP [2504.06606], Med-PRM [2506.11474]) to generate large-scale, step-annotated reward data in non-text domains (vision, medicine, code) without manual annotation.
- **Process Reward Learning from ORMs:** Dual-consistency frameworks such as SP-PRM [2506.12446] show that process reward models can be learned from outcome-based models with synthetic truncations, optimizing for score and preference consistency to improve alignment in reward-guided search at inference.
- **Efficient and Reliable RL in Generative Models:** In diffusion models, stepwise decomposition provides a tractable alternative to trajectory-level RL objectives, improving tractability and alignment under arbitrary reward functions [2411.11727, 2507.04832].
- **Suppressing Overthinking:** Verifiable Stepwise Reward Mechanisms (VSRM) utilize step difference signals for both compressing output and discouraging excessive, redundant computation in over-parameterized reasoning models [2508.10293].

## 7. Impact and Broader Implications

SORMs now underpin state-of-the-art performance in categories such as mathematical reasoning, SQL generation, code synthesis, and medical diagnostics across model sizes, with notable data-efficiency and robustness to limited annotation. By bridging outcome-driven supervision with stepwise process modeling, SORMs advance both practical deployment (requiring limited annotation and exhibiting superior stability) and theoretical understanding (formal value approximation and regret bounds).

Stepwise outcome-based reward modeling is likely to remain a central paradigm for model alignment, self-refining reasoning, and verification across both monomodal and multimodal generative systems, enabling a principled balance between efficiency, correctness, and interpretability.

Source: https://www.emergentmind.com/topics/stepwise-outcome-based-reward-models-sorms