Outcome-Supervised Reward Models

• Outcome-supervised reward models are a class of algorithms that assign a single scalar reward to an entire response, focusing solely on the final output rather than intermediate reasoning steps.
• They are applied across domains such as code generation, logical reasoning, and text-to-SQL, serving as rerankers, offline critics, and reinforcement learning signal providers.
• Despite their strengths in final answer verification, challenges like sparse feedback, reward hacking, and ambiguous credit assignment motivate ongoing research into hybrid and denser reward models.

Searching arXiv for the provided topic and key papers to ground the article in current literature. Outcome-supervised reward models are reward models that judge only the final output of a LLM response, assigning a single scalar signal to a complete trajectory or answer rather than to intermediate reasoning steps. In the terminology of recent surveys, an outcome reward model implements a reward function $r_{\text{outcome}}(x,y)\in\mathbb{R}$ over an input $x$ and full response $y$, and early alignment pipelines “relied on outcome reward models (ORMs) that judge only final answers, providing a single coarse signal for long chains of thought” (Zheng et al., 9 Oct 2025). Across contemporary work, this paradigm appears in RLHF, reinforcement learning with verifiable rewards, verifier-based best-of-$N$ selection, and task-specific critics for code, logic, tool use, and text-to-SQL (Gehrmann, 3 Oct 2025). Recent literature treats outcome-supervised reward modeling both as a foundational alignment primitive and as a limiting case whose sparsity motivates process-level extensions and hybrid methods (Ding et al., 12 Jan 2026).

1. Definition and formal role

Outcome reward models operate at trajectory granularity. The standard abstraction is a scalar scoring function over a completed response, such as $r_{\text{outcome}}(x,y)\in\mathbb{R}$ (Zheng et al., 9 Oct 2025), $R_\theta(x,y)\in\mathbb{R}$ for deductive reasoning traces (Thatikonda et al., 27 Aug 2025), or $r_\theta(x,y)\in\mathbb{R}$ for tool-calling outputs (Agarwal et al., 15 Sep 2025). In all of these formulations, the supervision unit is the final answer, full message, complete program, or full tool-call sequence rather than any intermediate state.

This regime is distinguished from process supervision by what is labeled. In the survey formulation, an ORM provides “1 reward per trajectory,” whereas a process reward model defines stepwise rewards $r_t=r_{\text{process}}(x,s_{1:t})$ and aggregates them over a trajectory (Zheng et al., 9 Oct 2025). The conceptual contrast is therefore one of granularity and credit assignment: outcome supervision asks whether the final result is correct or preferred, while process supervision asks which intermediate steps are correct or useful.

Recent work also emphasizes that reward models and evaluation metrics are mathematically similar objects. “Reward Models are Metrics in a Trench Coat” argues that both are learned scoring functions mapping an input, output, and possibly context or references to a scalar score, differing mainly in whether the score is used to drive optimization or to assess performance (Gehrmann, 3 Oct 2025). This unification is especially natural for outcome-supervised reward models, because their targets are often directly defined by end-task outcomes such as exact match, test success, or human preference.

2. Supervision signals and training objectives

Outcome supervision is instantiated through several label sources. One common source is human preference data over full responses, often modeled with a Bradley–Terry or logistic preference objective (Zheng et al., 9 Oct 2025). Another is verifiable correctness: in logic, positive reward is assigned if the boxed final label matches the gold label and negative reward otherwise (Thatikonda et al., 27 Aug 2025); in code, the judge $\mathcal{J}:\Sigma^*\rightarrow\{0,1\}$ returns 1 if generated code passes the APPS tests and 0 otherwise, yielding a classic outcome reward on the final sequence (Groeneveld et al., 27 Oct 2025); in text-to-SQL, correctness is defined by result-set equivalence between candidate and gold queries, and candidates with execution errors are discarded (Tritto et al., 1 Sep 2025). ToolRM uses exact structural correctness of a completed tool-call sequence to build pairwise preferences between a ground-truth outcome and an incorrect generated one (Agarwal et al., 15 Sep 2025).

Architecturally, decoder-only LLMs are commonly converted into reward models by replacing or extending the language modeling head with a scalar or binary prediction head. In the APPS code setting, Phi-4-mini-instruct and Phi-4 are turned into reward models by replacing the final classifier layer with a linear regression layer, applying a sigmoid, and training with binary cross-entropy at every token using the rollout-level outcome label (Groeneveld et al., 27 Oct 2025). ToolRM replaces the LM head of Qwen2.5-Instruct models with a linear scalar head and trains a pairwise preference model with reward-centering regularization (Agarwal et al., 15 Sep 2025). GradeSQL instead treats correctness prediction as autoregressive generation of “Yes” or “No,” using $P(y_{\text{yes}}\mid x;\theta)$ as the reward score for a candidate SQL query (Tritto et al., 1 Sep 2025).

Training objectives follow the label type. Pointwise binary supervision is used when outcomes are verifiable, such as code pass/fail or logical correctness (Groeneveld et al., 27 Oct 2025). Pairwise objectives are used when supervision is expressed as a preferred vs rejected full output, as in standard ORM training and ToolRM’s Bradley–Terry formulation (Agarwal et al., 15 Sep 2025). In RLHF-style pipelines, the trained ORM is then used as the reward function in policy optimization, often with KL regularization to a reference model (Zheng et al., 9 Oct 2025).

3. Typical uses: reranking, critics, and RL

A principal use of outcome-supervised reward models is inference-time selection. In the survey formulation, best-of-$x$0 sampling draws $x$1 candidate responses and selects $x$2 (Zheng et al., 9 Oct 2025). This usage appears across multiple domains. In logical reasoning, candidate chain-of-thought traces are generated, scored by an ORM logit $x$3, and reranked in a flat best-of-$x$4 procedure (Thatikonda et al., 27 Aug 2025). In text-to-SQL, GradeSQL scores each candidate $x$5 with $x$6 and selects $x$7, outperforming execution-based Best-of-$x$8 and majority voting on BIRD and SPIDER (Tritto et al., 1 Sep 2025). ToolRM applies the same pattern to tool-calling outputs and reports strong downstream improvements when used as a best-of-32 selector (Agarwal et al., 15 Sep 2025).

A second use is as an offline critic. “Smaller Models, Smarter Rewards” trains small Phi-4 family models as critics for Python code generation, using binary outcome labels from APPS and reading the final token’s predicted probability as an outcome score (Groeneveld et al., 27 Oct 2025). The 14B model achieves approximately 72–74% accuracy on imbalanced test data and improves pass@1 from 45% baseline to 54–55% in best $x$9 selection, with pass@3 rising from 65% to 77–78% in best $y$0 selection (Groeneveld et al., 27 Oct 2025). The paper summarizes this as over a 20% relative improvement in search capability. This suggests that outcome-supervised reward models can function as strong reranking modules even without policy optimization.

A third use is as the reward signal in RL. The survey describes the standard pipeline in which a policy $y$1 is optimized with an objective of the form $y$2 (Zheng et al., 9 Oct 2025). In practice, this includes both classical RLHF and RL with verifiable rewards. However, several later papers argue that sparse outcome-only rewards can be inadequate for long-horizon reasoning and agentic tasks, motivating hybrid or process-aware extensions (Ding et al., 12 Jan 2026).

4. Domain-specific instantiations

Outcome-supervised reward modeling has been specialized to multiple task families, often using task-native correctness criteria.

For deductive logical reasoning, ORMs are trained on full chain-of-thought traces ending in a boxed answer over FOLIO, ProverQA, and JustLogic (Thatikonda et al., 27 Aug 2025). Training data is generated by Qwen2.5-7B-Instruct or GPT-4o, and the paper introduces echo generation, which exploits the tendency of LLMs to justify a provided but incorrect answer in order to synthesize hard negative traces. ORMs trained on CoT and echo-augmented data consistently outperform majority vote across reasoners and benchmarks, and the benefits are especially pronounced for weaker models such as Gemma3-1B (Thatikonda et al., 27 Aug 2025).

For code generation, the APPS-based work on Phi-4 critics uses outcome labels derived from unit tests and interprets token-level value-head outputs as success probabilities of partial prefixes, while remaining outcome-supervised in its label regime (Groeneveld et al., 27 Oct 2025). This blurs the interface boundary between ORMs and PRMs without introducing explicit process labels. A related implication is that outcome supervision can induce useful state-value estimates when rollout-level correctness is abundant.

For tool-calling LLMs, ToolRM defines the outcome as the completed serialized tool-call sequence itself, not the natural-language explanation or execution trace (Agarwal et al., 15 Sep 2025). It introduces FC-RewardBench, a benchmark with 1,500 unique user inputs, each paired with one correct and one incorrect tool-call sequence, and shows that generic reward models often miss subtle schema-level failures such as incorrect parameter values, incorrect function names, or incorrect numbers of functions. ToolRM models from 1.5B to 14B parameters improve downstream tool-calling performance by up to approximately 25% on average and enable reward-guided filtering for data-efficient fine-tuning (Agarwal et al., 15 Sep 2025).

For text-to-SQL, GradeSQL treats semantic correctness as equality of execution results between a candidate query and the gold query, then trains ORMs to answer whether a candidate SQL is correct (Tritto et al., 1 Sep 2025). On BIRD and SPIDER, ORM-based best-of-$y$3 selection outperforms execution-based heuristics and majority voting, with reported gains of +4.33% execution accuracy on BIRD and +2.10% on Spider over ex-BoN, and +2.91% and +0.93% over majority voting (Tritto et al., 1 Sep 2025). This establishes outcome-supervised reward models as a practical verifier layer for structured program synthesis.

For long-form, knowledge-intensive evaluation, OpenReward introduces OpenRM, a tool-augmented reward model that judges answer pairs by invoking search tools such as Wikipedia and arXiv (Hu et al., 28 Oct 2025). Its terminal outcome reward is exact-match correctness of which answer is better, while intermediate tool rewards are gated by final correctness: $y$4 (Hu et al., 28 Oct 2025). On newly collected long-form datasets, OpenRM reaches average accuracy 91.33 and substantially outperforms non-agentic baselines, illustrating a path from scalar ORMs toward outcome-grounded agentic judges (Hu et al., 28 Oct 2025).

5. Limitations and failure modes

The most widely discussed limitation of outcome-supervised reward models is sparse, delayed feedback. The PRM survey identifies sparse signals and poor credit assignment as central weaknesses: one scalar reward must summarize the entire reasoning trajectory, making it difficult to diagnose intermediate errors or reward partial progress (Zheng et al., 9 Oct 2025). This becomes especially problematic in long-horizon reasoning, retrieval-augmented generation, and agentic tool use, where early decisions strongly affect later success.

A second limitation is that outcome-only supervision can reinforce flawed internal reasoning. The survey explicitly notes that ORMs cannot distinguish a lucky correct answer from a sound reasoning process, nor can they reward mostly correct trajectories that fail only at the last step (Zheng et al., 9 Oct 2025). “Linking Process to Outcome” sharpens this criticism by arguing that standard PRM baselines and ORMs alike can suffer from ambiguous credit assignment and reward–accuracy mismatch when process rewards are not explicitly linked to final outcomes (Zhang et al., 30 Sep 2025). This suggests that coarse final-answer supervision may be insufficient whenever the objective includes faithful reasoning rather than final correctness alone.

A third concern is reward hacking and shortcut learning. “Reward Models are Metrics in a Trench Coat” emphasizes that outcome-supervised reward models inherit well-known pathologies from evaluation metrics, including susceptibility to spurious correlations, verbosity bias, style-over-substance effects, and downstream reward hacking under optimization pressure (Gehrmann, 3 Oct 2025). The same paper argues that task-specific metrics can outperform general reward models on specialized tasks such as translation quality and summarization attribution, indicating that a generic ORM may be a poor proxy when the true outcome is narrow and well defined (Gehrmann, 3 Oct 2025).

A fourth limitation is evaluation mismatch. “How to Evaluate Reward Models for RLHF” shows that static reward model benchmarks need not predict post-RLHF policy quality; in that study, granular pairwise accuracy on real human preference data is more predictive of downstream Arena performance than aggregate ranking metrics, while RewardBench is negatively correlated with RLHF outcomes for high-quality models (Frick et al., 2024). This suggests that evaluating outcome-supervised reward models solely by offline pairwise accuracy or benchmark leaderboards can be misleading if the actual deployment objective is post-training human preference.

6. Extensions, hybrids, and the current research direction

Much recent work treats outcome supervision not as an endpoint but as a base signal to be refined. A broad pattern is to derive denser or more faithful process signals from final-answer labels. “Intra-Trajectory Consistency for Reward Modeling” keeps outcome-level preference supervision but introduces a regularizer based on next-token generation probabilities to propagate response-level signals across prefixes, improving RewardBench performance and downstream DPO and best-of-$y$5 results without additional process labels (Zhou et al., 10 Jun 2025). “The Weakest Link Tells It All” formulates outcome-supervised PRM training as a multiple-instance learning problem, introduces Softmax-Weighted-Sum pooling, and proves Bayes consistency under mild assumptions, thereby learning step-level error identification from final-answer correctness alone (Jia et al., 26 Jun 2026).

Another trajectory is hybrid reward design. The LeTS framework combines stepwise process reward with outcome-based reward for think-and-search in retrieval-augmented generation, motivated by the observation that outcome-level supervision neglects intermediate think-and-search steps (Zhang et al., 23 May 2025). PRPO aligns token-level process rewards with outcome rewards in critic-free policy optimization through a location-parameter shift, improving MATH500 accuracy from 61.2% to 64.4% over GRPO on Qwen2.5-Math-1.5B with eight rollouts and no value network (Ding et al., 12 Jan 2026). Hybrid Reward Normalization for non-verifiable agentic tasks proposes ReNorm, $y$6, to calibrate process and outcome rewards and reports state-of-the-art performance across open-domain QA benchmarks (Xu et al., 29 Sep 2025).

A related line emphasizes that final outcomes must remain the anchor even when denser signals are introduced. PRPO explicitly argues that dense process rewards should be aligned to outcome rewards rather than simply averaged in (Ding et al., 12 Jan 2026). RLVP makes a parallel point for real-world agents: “penalize the path, reward the outcome” because outcome-only training ignores deployability constraints, while verifiable path penalties can add within-group reward variance and encode outcome-neutral constraints that final rewards miss (Li et al., 8 Jul 2026). This suggests a broader synthesis in which outcome-supervised reward models remain the authoritative source of task success, while process and path signals are added to improve credit assignment, safety, and sample efficiency.

From a survey perspective, the emerging consensus is complementary rather than replacement-oriented. ORMs retain key advantages—cheap labels, strong generalization, and high anti-hacking robustness when correctness is objectively verifiable—while PRMs and hybrids address granularity, interpretability, and trajectory control (Zheng et al., 9 Oct 2025). A plausible implication is that outcome-supervised reward models will remain central wherever final correctness is easy to verify, but increasingly as one component in systems that explicitly relate process quality to end-task outcomes.