Outcome-Reward Reinforcement Learning

• Outcome-reward RL is defined by providing a single scalar reward at trajectory end, requiring agents to infer credit assignment across long sequences.
• The paradigm increases sample complexity compared to per-step reward RL, making learning more challenging in high-dimensional, delayed feedback settings.
• Surrogate methods like reward shaping, hybridization, and curriculum generation improve training efficiency in applications such as LLM fine-tuning, robotics, and forecasting.

Outcome-reward reinforcement learning (RL) refers to a class of RL approaches in which the reward signal is provided only at the end of a trajectory—based on the observed outcome—rather than at each intermediate step. This paradigm is increasingly central to both LLM fine-tuning (particularly in domains like reasoning and code generation) and online RL for long-horizon tasks where explicit per-step rewards are unavailable or impractical. Outcome-reward RL addresses the sparse and delayed feedback setting, in which the agent must infer which actions in an often lengthy sequence contributed to the final outcome, making credit assignment particularly challenging. The following sections offer a comprehensive account of outcome-reward RL, covering problem formulations, theoretical properties, algorithms, the interplay with process rewards, and practical considerations.

1. Formalism and Core Principles

In outcome-reward RL, the only feedback available to the agent after executing a trajectory $\tau$ is a scalar, trajectory-level reward $r(\tau)$, typically reflecting the success or failure of the final outcome. More formally, the agent’s environment is a Markov Decision Process (MDP) $M=(\mathcal{S},\mathcal{A},T,p,H)$ with (possibly) a long horizon $H$, but with the following critical distinction:

• Process-reward RL: Feedback is provided at each step $h$ via $r_h(s_h,a_h)$.
• Outcome-reward RL: Only the aggregate or endpoint signal $r(\tau)$ is observed, where $\tau = (s_1,a_1,\dots,s_H,a_H)$.

The RL objective is to find a policy $\pi_\theta$ maximizing expected outcome reward:

$$J_{outcome}(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[r(\tau)]$$

where $r(\tau)$ is generally non-decomposable into per-step rewards. Typical reward forms include binary correctness, proper scoring rules (e.g., Brier score for forecasting), preferences over trajectory pairs, or utility assigned by external evaluators (Turtel et al., 23 May 2025, Ju et al., 2024).

The statistical challenge is that the agent receives no explicit learning signal about which specific actions in the sequence contribute to the observed outcome, heightening the variance and delay of policy gradient estimates (Chen et al., 26 May 2025).

2. Theoretical Properties and Statistical Efficiency

Outcome-reward RL is strictly harder than per-step reward RL in terms of sample complexity. In the context of general function approximation, outcome-only feedback increases the required number of episodes for near-optimal policy learning by a factor linear in trajectory length $H$ compared to per-step feedback. In some settings (notably, high-dimensional or low-observability MDPs), the separation can be exponential (Chen et al., 26 May 2025):

• Sample Complexity: With process feedback: $\widetilde{O}(C_{\text{cov}} H^2/\epsilon^2)$. With outcome-only feedback: $\widetilde{O}(C_{\text{cov}} H^3/\epsilon^2)$, where $C_{\text{cov}}$ is the coverability coefficient measuring exploration difficulty.
• Lower Bound: There exists a class of MDPs where outcome-only RL requires $\exp(\Omega(d))$ episodes to achieve constant error, compared to $O(d^2/\epsilon^2)$ for process-reward RL, with $d$ the problem’s intrinsic dimensionality.

Both deterministic and stochastic MDPs are covered, with simplifications (e.g., Bellman-residual fitting) available for the deterministic case (Chen et al., 26 May 2025).

3. Methodologies for Credit Assignment

Because direct assignment of outcome reward to all steps yields uninformative gradients, a rich array of surrogate methods, regularizations, and algorithmic frameworks have been developed to improve credit assignment and stability:

Method/Framework	Credit Assignment Approach	Typical Use Case or Domain
GRPO/ReMax, PPO variants	Shared outcome reward broadcast to tokens	Reasoning LLMs, forecasting (Turtel et al., 23 May 2025, Ye et al., 3 Sep 2025)
Behaviour Cloning on BoN	Policy trained on best-of-N outcome-positive rollouts	LLM mathematical reasoning (Lyu et al., 10 Feb 2025)
Reward Model/Preference	Surrogate reward via classifier, expert preference	Robotics, language tasks (Ju et al., 2024, Eysenbach et al., 2021)
Reward Shaping	Token-/segment-level auxiliary model	Long-horizon reasoning, code generation (Ding et al., 12 Jan 2026)
Variational/Bayesian RL	Dense reward from inferred outcome likelihood	Goal-directed RL (Rudner et al., 2021)

Key algorithmic principles include the use of baseline subtraction (to reduce variance in policy gradients), group-wise normalization (e.g., in GRPO or RLOO), and the incorporation of KL regularization towards a reference policy for stability (Lyu et al., 10 Feb 2025, Turtel et al., 23 May 2025).

Recent work augments purely outcome-conditioned optimization with auxiliary signals, such as token-level reward models or process reward models (see section 4), to mitigate sparsity (Ye et al., 3 Sep 2025, Ding et al., 12 Jan 2026, Lyu et al., 10 Feb 2025).

4. Process-Outcome Reward Hybridization

Outcomes alone provide insufficient granularity for reasoning-intensive tasks, since only the final answer is scored, and flawed trajectories that guess correctly are indistinguishable from valid ones. To address this, researchers have introduced process reward models (PRMs) to provide fine-grained, intermediate supervision, and have developed hybridization strategies:

• Dual Reward Systems: SAIL-RL combines a binary outcome reward with (i) a thinking reward, which checks logical, factual, and answer consistency of reasoning steps, and (ii) a judging reward denoting when deep reasoning is warranted (Shu et al., 4 Nov 2025). This reduces overthinking on simple tasks and hallucination on hard tasks.
• Filtering and Alignment: The PROF method does not blend process and outcome rewards linearly; instead, it filters rollouts by the consistency of process and outcome signals, retaining trajectories where both agree in sign and magnitude. This approach significantly improves both accuracy and process fidelity in math reasoning LLMs (Ye et al., 3 Sep 2025).
• Critic-Free Hybridization: PRPO aligns the normalized process reward distribution (computed over semantically segmented chains) with the mean of outcome reward, providing token-level advantages while preserving scalability and avoiding a critic network (Ding et al., 12 Jan 2026).

These strategies prevent reward hacking—a common issue when dense, automated rewards are mixed naively—and have demonstrated empirical gains of up to 4–7.7 percentage points on mathematical and multimodal reasoning benchmarks (Ye et al., 3 Sep 2025, Ding et al., 12 Jan 2026, Wang et al., 13 Nov 2025).

5. Curriculum and Goal-Example-Based Outcome RL

Outcome-reward RL is highly effective in settings where the only supervision is a set of target states or successful trajectories—eliminating the need for hand-crafted rewards:

• Example-Based RL: Approaches such as Recursive Classification of Examples (RCE) and meta-learning-based uncertainty-aware rewards (MURAL) directly estimate the future probability of reaching outcome examples by leveraging classifier-based reward shaping, bypassing explicit reward modeling (Eysenbach et al., 2021, Li et al., 2021). These frameworks provide a unified value function satisfying a data-driven Bellman equation or utilize conditional normalized maximum likelihood (CNML) estimation for calibrated shaping and exploration. /// /// Editor’s term: “Example-based RL”
• Curriculum Generation: Algorithms like D2C and OUTPACE leverage uncertainty-aware or classifier-driven metrics to automatically generate a curriculum of intermediate goals interpolating between the agent’s current frontier and outcome examples. Bipartite matching ensures both coverage and progression without explicit knowledge of environment geometry (Cho et al., 2023, Cho et al., 2023).

These methods have demonstrated superior sample efficiency and robustness in high-dimensional, long-horizon navigation and manipulation tasks.

6. Extensions: Preference and Sequence Rewards

For domains where outcome is specified via preference between pairs of trajectories rather than explicit reward, ordinal-to-cardinal conversion frameworks (e.g., ELO-Rating Based RL) are used:

• Ordinal Preferences to Cardinal Rewards: Preference-based RL algorithms infer a utility (e.g., ELO rating) for each trajectory using expert comparisons, which is then redistributed to per-step transitions to permit standard RL optimization (Ju et al., 2024).
• Reward Redistribution: ERRL ensures training stability in long-horizon settings by uniformly distributing outcome-based utility increments to all transitions in a trajectory, with theoretical optimality of the “equal split” under logistic noise assumptions.

This extends naturally to outcome-only settings, as accurate per-step reward assignment is not available, but trajectory-level preferences are consistent and can be efficiently harnessed for accelerated learning.

7. Practical Considerations and Applications

Outcome-reward RL has become central in modern LLM alignment, mathematical and code reasoning, multimodal reasoning, online probabilistic forecasting, and robotics:

• LLMs: Outcome-only RL tunes LLMs for mathematical, coding, and retrieval-augmented tasks, often combined with process-level diagnostics for chain-of-thought supervision and reward hacking prevention (Turtel et al., 23 May 2025, Weng et al., 18 May 2025, Wang et al., 13 Nov 2025).
• Forecasting: Outcome-only RL with proper scoring rules (e.g., Brier score) and adapted on-policy optimization achieves frontier calibration and economic value in event prediction (Turtel et al., 23 May 2025).
• Robotics/Navigation: Example- and classifier-based outcome RL delivers sample-efficient exploration in environments with sparse or no shaped reward (Eysenbach et al., 2021, Li et al., 2021, Cho et al., 2023, Cho et al., 2023).

Best practices include curriculum design for improved sample efficiency, non-linear reward hybridization and filtering for robustness, and calibrated baseline subtraction or actor–critic schemes for algorithmic stability. Asymptotic performance, sample complexity, and sensitivity to initial policy and data remain active research topics (Chen et al., 26 May 2025, Lyu et al., 10 Feb 2025).

• (Eysenbach et al., 2021) Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification
• (Rudner et al., 2021) Outcome-Driven Reinforcement Learning via Variational Inference
• (Li et al., 2021) MURAL: Meta-Learning Uncertainty-Aware Rewards for Outcome-Driven Reinforcement Learning
• (Cho et al., 2023) Outcome-directed Reinforcement Learning by Uncertainty & Temporal Distance-Aware Curriculum Goal Generation
• (Cho et al., 2023) Diversify & Conquer: Outcome-directed Curriculum RL via Out-of-Distribution Disagreement
• (Ju et al., 2024) ELO-Rated Sequence Rewards: Advancing Reinforcement Learning Models
• (Lyu et al., 10 Feb 2025) Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning
• (Weng et al., 18 May 2025) Graph-Reward-SQL: Execution-Free Reinforcement Learning for Text-to-SQL via Graph Matching and Stepwise Reward
• (Zhang et al., 23 May 2025) LeTS: Learning to Think-and-Search via Process-and-Outcome Reward Hybridization
• (Turtel et al., 23 May 2025) Outcome-based Reinforcement Learning to Predict the Future
• (Chen et al., 26 May 2025) Outcome-Based Online Reinforcement Learning: Algorithms and Fundamental Limits
• (Ye et al., 3 Sep 2025) Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Training
• (Shu et al., 4 Nov 2025) SAIL-RL: Guiding MLLMs in When and How to Think via Dual-Reward RL Tuning
• (Wang et al., 13 Nov 2025) Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Sampling
• (Ding et al., 12 Jan 2026) PRPO: Aligning Process Reward with Outcome Reward in Policy Optimization