Outcome Reward Model (ORM): Apps & Challenges

• Outcome Reward Models (ORMs) are defined as models that assess only the final output’s correctness, streamlining supervision for tasks like math, code, and SQL.
• They are employed in reinforcement learning and inference-time reranking, offering label efficiency and robustness across diverse applications.
• Despite their efficiency, ORMs lack process-level feedback, prompting research into hybrid models for improved reasoning transparency and safety.

An Outcome Reward Model (ORM) is a class of reward model that assigns supervision based solely on the correctness of the final output of a sequence, ignoring the intermediate steps taken to reach that outcome. ORMs have become a foundational component in the alignment and training of LLMs for complex domains such as mathematical reasoning, code generation, SQL query synthesis, and tool-use tasks. Their efficiency in annotation, robustness to certain pathologies, and algorithmic simplicity have led to wide adoption both in reinforcement learning pipelines and as inference-time verifiers across modalities and domains. However, limitations in providing fine-grained, process-level feedback have motivated ongoing theoretical and empirical investigation into both novel ORM designs and their interplay with process reward models (PRMs).

1. Core Definition and Methodology

An ORM evaluates the correctness or utility of a fully completed sequence (e.g., a math solution, code submission, SQL query, or tool-call) via a scalar reward signal. In formal terms, the reward $R$ is defined as: $$R(x, y) = \begin{cases} 1 & \text{if the final answer is correct} \ 0 & \text{otherwise} \end{cases}$$ where $x$ is the input and $y$ is the candidate output. This approach is utilized in both supervised fine-tuning and reinforcement learning regimes, typically via policy gradient or Proximal Policy Optimization (PPO) where the ORM serves as the external reward function.

In inference-time reranking, the ORM is used as a post hoc verifier: multiple candidate sequences $Y = \{y_1, \ldots, y_K\}$ are generated, then scored and ranked by the ORM. A canonical selection algorithm is: $$f^* = \underset{f}{\arg\max} \sum_{\{y_i : \text{final\_ans}(y_i) = f\}} \text{rm\_prob}(y_i)$$

$$y^* = \underset{\{y : \text{final\_ans}(y) = f^*\}}{\arg\max} \text{rm\_prob}(y)$$

where $\text{rm\_prob}(y)$ is the reward model’s estimated class probability (Uesato et al., 2022). This protocol is used not only in mathematics but also in Text-to-SQL, code, tool-use, and logical reasoning domains (Tritto et al., 1 Sep 2025, Orlanski et al., 11 Jun 2025, Agarwal et al., 15 Sep 2025, Thatikonda et al., 27 Aug 2025).

ORMs are often trained as discriminative binary or scalar ranking models. Given input–response pairs labeled as correct or incorrect, the most common training objectives are:

• Binary cross-entropy over the predicted reward,
• Pairwise Bradley-Terry or energy-based ranking losses, e.g.,

$$\mathcal{L}(\theta; \mathcal{Y}_n) = \frac{1}{|\mathcal{Y}_+||\mathcal{Y}_-|} \sum_{y_+ \in \mathcal{Y}_+} \sum_{y_- \in \mathcal{Y}_-} \log\left[1 + \exp(E_\theta(y_+) - E_\theta(y_-))\right]$$

where $E_\theta(y)$ is the energy assigned to candidate $y$ and $\mathcal{Y}_+, \mathcal{Y}_-$ are the correct and incorrect solution sets (Jiang et al., 21 May 2025).

2. Efficacy, Strengths, and Limitations

ORMs are highly label-efficient: only the final answer requires annotation, minimizing data collection effort (as few as 1–4 tokens of supervision per instance in math tasks) (Uesato et al., 2022). This efficiency, coupled with trivial automation of supervision (string/arithmetic matching for math, execution for code, and database result matching for SQL), has enabled wide usage at scale.

However, standard ORM training and use exhibit two important limitations:

• Lack of process-level supervision: As only the final state is rewarded, the learning signal ignores intermediate errors or flawed logic—causing models to sometimes reach correct answers via incorrect or “deceptive” reasoning. This leads to high trace error: the proportion of solutions with at least one flawed intermediate step, even among those yielding correct outcomes, and diminished interpretability (Uesato et al., 2022).
• Granularity mismatch in RL or guided decoding: When used as a reward at each token or chunk during sequence generation, ORMs, which are only well-calibrated on complete solutions, yield inconsistent or suboptimal policy gradients (Xie et al., 14 Jun 2025). This inconsistency can hinder alignment and the model’s ability to follow user preferences at every generation step.

Despite these issues, ORMs have empirically driven strong final accuracy in various domains, including mathematics (Uesato et al., 2022), code (Orlanski et al., 11 Jun 2025), Text-to-SQL (Tritto et al., 1 Sep 2025), and tool use (Agarwal et al., 15 Sep 2025), but alone cannot guarantee reasoning soundness or stepwise verification.

3. Enhancements, Extensions, and Recent Developments

Several advancements have addressed the deficiencies of vanilla ORMs:

Reward Model Architecture and Inference

• Energy-Based ORM (EORM): The EORM treats candidate ranking as energy minimization. Typically implemented over a Transformer encoder, a scalar energy (negative preference) is predicted for each candidate, with lower energy indicating a more desirable output. Training uses outcome-only binary labels, optimizing smooth pairwise losses (Jiang et al., 21 May 2025).

Data Synthesis and Error Diversification

• Echo Generation for Error Expansion: When building datasets for logical or mathematical reasoning tasks, echo generation (EcCoT) is used to prompt LLMs into producing plausible but incorrect reasoning sequences by presupposing a false conclusion, then filtering for non-trivial errors not easily flagged (Thatikonda et al., 27 Aug 2025). This produces a more diverse and challenging set of negative examples, increasing ORM robustness to subtle reasoning flaws.

Indirect Process Emulation and Implicit PRMs

• Implicit Process Reward Extraction: By parameterizing the ORM reward as a log-likelihood ratio between a policy and a reference model,

$$r_\theta(\mathbf{y}) = \beta \log \left[\frac{\pi_\theta(\mathbf{y})}{\pi_{\mathrm{ref}}(\mathbf{y})}\right]$$

and computing stepwise Q-values,

$$q_\theta^t(\mathbf{y}_{<t}, y_t) = \sum_{i=1}^{t} \beta \log \left[\frac{\pi_\theta(y_i|\mathbf{y}_{<i})}{\pi_{\mathrm{ref}}(y_i|\mathbf{y}_{<i})}\right]$$

one can extract process-level (token-wise) rewards as differences $r_\theta^t = q_\theta^t - q_\theta^{t-1}$, making the ORM an implicit PRM—yielding fine-grained feedback without explicit stepwise labels (Yuan et al., 2 Dec 2024).

Test-time Applications and Cross-domain Usage

• ORMs power Best-of-N selection, inference-time ranking, and reward-pruned search (Son et al., 24 Feb 2025, Tritto et al., 1 Sep 2025, Thatikonda et al., 27 Aug 2025), enabling performance gains in non-mathematical domains such as clinical note generation (Wang et al., 17 Dec 2024), SQL synthesis (Tritto et al., 1 Sep 2025), logical deduction (Thatikonda et al., 27 Aug 2025), and tool use (Agarwal et al., 15 Sep 2025).

4. Comparison with Process Reward Models (PRMs) and Hierarchical Models

While ORMs are efficient and robust against certain forms of reward hacking (there is little incentive to inflate intermediate step count), they are fundamentally coarse. PRMs, by contrast, provide dense, fine-grained stepwise signals—offering superior explainability, the ability to pinpoint intermediate mistakes, and better support for safe or educational deployment (Zheng et al., 9 Oct 2025, Uesato et al., 2022, Wang et al., 17 Dec 2024). However, PRMs are more expensive to supervise, requiring annotation at every reasoning step, and are more vulnerable to reward exploitation where the model “inflates” reward by repeating correct steps without new content (Gao et al., 19 Oct 2024, Wang et al., 16 Mar 2025).

Hybrid and hierarchical models—such as the Hierarchical Reward Model (HRM), which evaluates both stepwise and multi-step coherence, enabling error correction across the chain—combine elements of outcome and process supervision for higher stability and generalization (Wang et al., 16 Mar 2025). Data composition and augmentation methods, such as Hierarchical Node Compression, further help bridge the gap by enabling more robust and cost-effective process supervision.

Recent work has shown that ORMs trained on outcome labels can sometimes implicitly emulate process-level signals: when used for reranking or as a reward in RL with fine-grained output, the ORM biases the model toward more coherent reasoning, reducing trace error even without explicit process annotation (Uesato et al., 2022, Yuan et al., 2 Dec 2024).

5. Applications in RL, Code Verification, SQL, and Tool Use

ORMs are central to two major classes of applications:

Reinforcement Learning (RL)

• In policy optimization (e.g., PPO, REINFORCE), ORMs supply the external reward, either as a binary (success/failure) signal or as a ranking function over candidate outputs. They are effective in stabilizing RL for large-scale pretraining and fine-tuning (Gao et al., 19 Oct 2024, Jiang et al., 21 May 2025).
• However, using pure ORM rewards does not necessarily provide additional training signal beyond the sparse success reward, and can therefore plateau performance. The development of more informative, robust reward functions—such as refined (clip/delta) process rewards for RL, or token-level discriminative reward models (Q-RM)—has proven more effective for credit assignment and sample efficiency (Gao et al., 19 Oct 2024, Chen et al., 29 May 2025).

Inference-time Verification and Scaling

• ORMs enable practical test-time scaling via candidate reranking and selection in mathematics, reasoning, code, and SQL tasks (Son et al., 24 Feb 2025, Orlanski et al., 11 Jun 2025, Tritto et al., 1 Sep 2025).
• In code, using an ORM for generate-prune-rank achieves a 11.65× speedup relative to full test-suite verification, with only 8.33% reduction in accuracy—a favorable tradeoff for high-throughput settings (Orlanski et al., 11 Jun 2025).
• In tool-calling LLMs, ORMs trained on synthesized tool call data efficiently filter and select correct actions, providing up to 25% improvement in downstream task performance over baselines (Agarwal et al., 15 Sep 2025).
• In SQL, ORM-based scoring outperforms both execution-based and majority-voting rerankers, especially on more complex or ambiguous queries (Tritto et al., 1 Sep 2025).

6. Current Challenges and Prospects

Reward Hacking and Robustness

Although ORMs are less susceptible to reward hacking than PRMs, they can still be gamed when there is a discrepancy between the process and the outcome. Models may learn “shortcuts” that bypass genuine reasoning if the dataset is not sufficiently diverse or if negative samples are not well controlled (Uesato et al., 2022, Ye et al., 3 Sep 2025). Blending outcome and process rewards—using sample filtering, hierarchical evaluation, or process-consistency criteria—yields more reliable guidance and helps bridge robustness gaps (Ye et al., 3 Sep 2025).

Evaluation and Benchmarking

The evaluation of ORMs remains an open problem. Recent work leverages overoptimization metrics (normalized area between RM proxy and gold reward curves) and emphasizes minimizing distributional mismatches, increasing evaluation set diversity, and using multi-pairwise comparisons for robust assessment (Kim et al., 19 May 2025).

Extensions and Integration

Emergent directions include:

• Extending ORM applicability via energy-based, cross-domain, or generative-verification methods (i.e., integrating ORM with chain-of-thought verifiers) (Jiang et al., 21 May 2025, Liu et al., 5 Aug 2025).
• Unifying outcome and process reward modeling for improved inference-time alignment (e.g., SP-PRM) (Xie et al., 14 Jun 2025).
• Developing lightweight, modular, and scalable ORM architectures suitable for new domains (multimodal reasoning, tool-calling, robotics).

A plausible implication is that ORM evolution will continue to reflect the balance between cost-effective, robust final-signal supervision and the increasing demand for transparent, process-aware, detailed reasoning feedback, especially as next-generation models tackle more open-ended, safety-critical, and physically grounded tasks.

7. Summary Table: ORM vs. PRM vs. Hybrid

Model Class	Supervision Granularity	Key Strengths
ORM	Final outcome only	Label efficiency; robust; scalable
PRM	Stepwise/process-level	Fine-grained feedback; higher interpretability
Hybrid/HRM	Multi-level	Robustness, self-correction, balance of cost/feedback

Outcome Reward Models are foundational for efficient sequence-level supervision and test-time verification. While they excel at maximizing final-task accuracy and resource efficiency, ongoing and future research will focus on refining their design and leveraging hybrid strategies for fine-grained, robust, and interpretable reasoning supervision across diverse tasks and modalities.