Process Reward Learning (PRL)

• Process Reward Learning (PRL) is a reinforcement learning paradigm that assigns dense, stepwise rewards along decision trajectories, enabling fine-grained credit assignment.
• PRL leverages entropy regularization and intermediate reward decompositions to align actions with long-term objectives, yielding improvements in tasks like mathematical reasoning and multi-turn planning.
• PRL methods incorporate neural network-based reward models and algorithms such as PPO, DPO, and GRPO to mitigate reward hacking and enhance training efficiency in complex, agentic environments.

Process Reward Learning (PRL) is a reinforcement learning paradigm that assigns fine-grained supervision to sequential decision-making models by distributing feedback across intermediate steps of a reasoning or action trajectory—rather than relying solely on sparse, outcome-level rewards. Originally motivated by the need for more effective credit assignment over long horizons in reasoning and agentic tasks, PRL has emerged as a core methodology for optimizing LLMs, agentic systems, and complex planning, with parallel developments in both theoretical underpinnings and scalable algorithms. The approach underpins recent progress in mathematical reasoning, multi-turn planning, code synthesis, and retrieval-augmented language modeling.

1. Formal Foundations and Objective Decomposition

PRL is grounded in the entropy-regularized reinforcement learning framework. The central objective is to maximize a combination of (i) expected reward—originating from either outcome-level or process-level supervision—and (ii) a regularization term that penalizes divergence from a reference policy, typically expressed as a KL-divergence:

$Q(\pi)\;=\;\mathbb{E}_{x}\Bigl[\;\mathbb{E}_{a\sim\pi(\cdot\mid x)}\bigl[r^*(x,a)\bigr] - \frac{1}{\eta}\,\KL\bigl(\pi(\cdot\mid x)\,\big\|\,\pi_{0}(\cdot\mid x)\bigr)\Bigr]$

Where $r^*(x,a)$ is the terminal (outcome) reward, $\pi_0$ is the reference policy, and $\eta$ controls the strength of regularization. The optimal policy has a tractable Boltzmann form:

$$\pi^*(a\mid x)\;\propto\;\pi_0(a\mid x)\,\exp\bigl(\eta\,r^*(x,a)\bigr)$$

Crucially, the identity

$$r^*(x,a)\;-\;\frac{1}{\eta}\ln\frac{\pi^*(a\mid x)}{\pi_0(a\mid x)}\;=\;C$$

enables dense process-level signals: the trajectory-level reward can be exactly decomposed into intermediate process rewards tied to the log-probability ratios of actions under the evolving policy versus reference (Yao et al., 15 Jan 2026). Alternative derivations leverage soft value functions and entropy-regularized dynamic programming, yielding similar process-reward decompositions (Zhang et al., 2024).

2. Process Reward Models: Definitions, Taxonomy, and Learning

A Process Reward Model (PRM) is a function that assigns a real-valued reward to each partial trajectory (sequence of states and actions), providing dense feedback across a solution or action sequence (Zheng et al., 9 Oct 2025). In contrast to standard outcome reward models (ORMs), which emit a single terminal reward per trajectory, PRMs output stepwise rewards: for a trajectory τ = (s₀,a₀,s₁,a₁,…,s_T,a_T), the process return is often ∑ₜ r(s_t,a_t).

Types of PRMs:

• Step-Level vs. Trajectory-Level: Step-level PRMs assign a reward to every individual step; trajectory-level PRMs aggregate the sequence (sum, min, mean, etc.) to a single score.
• Discriminative vs. Generative vs. Implicit: Discriminative PRMs learn a classifier or regressor to assign correctness confidence to each step; generative PRMs sample (e.g., a critique chain or explanation) before scoring; implicit PRMs derive rewards without explicit labels via consistency, self-labeling, or preference learning mechanisms (Sullivan, 25 Sep 2025).
• Bidirectional and Hybrid: Recent work extends PRMs to bidirectional settings, combining left-to-right and right-to-left contextualization to better judge intermediate steps given both prior and future progress (Zhang et al., 3 Aug 2025). Hybrid models unify stepwise and outcome rewards through normalization or multi-stage approaches (Xu et al., 29 Sep 2025, Zhang et al., 23 May 2025).

Process rewards are typically parameterized by neural networks (often LLMs or their heads) and trained via binary cross-entropy, mean squared error, or contrastive objectives. For automated labeling, methods utilize symbolic checkers, LLM critics, or self-improvement procedures (synthetic evolution or search) to simulate or annotate stepwise correctness (Zheng et al., 9 Oct 2025).

3. Algorithmic Realizations and Training Recipes

Process Reward Learning is instantiated as a modification of standard RLHF or policy optimization pipelines:

• Reward Calculation: For each prefix or intermediate step, the process reward is computed either analytically (via log-probability ratios within the policy, as in entropy-regularized PRL (Yao et al., 15 Jan 2026, Zhang et al., 2024, Fei et al., 2 Jul 2025)), or by invoking a learned PRM.
• Advantage Estimation: Process-level rewards feed into token- or step-level advantage calculations. Innovations such as masked step advantage (MSA) facilitate stable, variance-reduced training by comparing cumulative process rewards across sampled peers per prompt (Fei et al., 2 Jul 2025).
• Policy Updates: Policy optimization is performed with PPO, DPO, RLOO, or GRPO objectives, now targeting the sum of process and, if present, outcome advantages. Algorithms often include KL-divergence or entropy penalties to control exploration and prevent distributional drift.
• Hybrid and Min-form Credit Assignment: For robustness against reward hacking, credit assignment may use the minimum future stepwise reward (min-form), as in PURE (Cheng et al., 21 Apr 2025), or process-outcome harmonization via reward normalization and trajectory consistency filters (Ye et al., 3 Sep 2025, Xu et al., 29 Sep 2025).
• Preference and Online PRL: When ground-truth step correctness is unavailable, process rewards are inferred via trajectory preference-based learning using pairwise ranking and direct preference optimization (DPO) objectives (Liu et al., 23 Sep 2025, Bıyık et al., 2020). The process DPO approach produces implicit step rewards consistent with the observed preference ranking.

Illustrative pseudocode for on-policy PRL (analytical process rewards) (Yao et al., 15 Jan 2026):

for each batch: sample rollouts a ∼ πθ for each a, compute r^*(x, a) (terminal reward) for each token t: compute future KL sum S_t = sum_{j=t}^L (1/η) k_j set process advantage ρ_t = stopgrad[A(x,a) - S_t] update πθ with PPO-clip objective using {ρ_t}

4. Empirical Evaluation and Applications

Process Reward Learning is empirically validated across reasoning and agentic domains:

• Mathematical Reasoning: PRL yields significant gains over outcome-only RL in pass@N and average@N benchmarks on MATH500, Minerva, OlympiadBench, etc. It improves both mean performance and robustness, as measured by diversity of reasoning trajectories and breadth metrics (Yao et al., 15 Jan 2026, Zhang et al., 2024, Zheng et al., 9 Oct 2025).
• Agents and Multi-Turn Tasks: AgentPRM and related frameworks demonstrate >8× compute efficiency boosts over ORM-based pipelines in multi-turn environments such as WebShop, BabyAI, and ALFWorld. Stepwise "promise" and "progress" estimates from process rewards facilitate efficient beam search, Best-of-N reranking, and improved sample complexity (Xi et al., 11 Nov 2025, Choudhury, 14 Feb 2025).
• Robustness and Reward Hacking: Min-form credit assignment and hybrid step-outcome reward structures are shown to mitigate reward hacking phenomena prevalent under summation-form RL with PRMs (Cheng et al., 21 Apr 2025, Ye et al., 3 Sep 2025).

A concise empirical comparison of PRL variants on reasoning tasks is summarized below (selected results from (Yao et al., 15 Jan 2026), metrics: pass@8, average@8):

Model	Pass@8 (%)	Average@8 (%)
REINFORCE	72.12	54.96
GRPO	72.12	54.96
PRL (this work)	72.38	55.71
Llama-3.2-1B (base)	33.42	12.72

PRL also demonstrates rapid convergence and stability, requiring fewer steps to reach peak performance compared to traditional outcome-based RL, and reduces compute cost by exploiting structure in process-level credit (Fei et al., 2 Jul 2025, Sullivan, 25 Sep 2025).

5. Limitations, Reward Hacking, and Theoretical Insights

While PRL enables efficient and granular supervision, several limitations and failure modes are noted:

• Reward Hacking: PRMs in naïve summation-based RL can be exploited, leading to degenerate behaviors such as excessive verbosity, unwillingness to terminate, or repetitive outputs. Min-form assignments and outcome reward normalization partially address this.
• Noisy or Misaligned PRMs: Offline or noisy stepwise reward models may misalign with terminal objectives. Filtering methods (PROF (Ye et al., 3 Sep 2025)), hybrid process-outcome reward normalization (Xu et al., 29 Sep 2025), and dynamic process-outcome blending are proposed to maintain alignment and stability.
• Annotation Cost and Domain Adaptivity: Reliance on step-level human annotation remains expensive, motivating active selection, uncertainty filtering (Duan et al., 14 Apr 2025), and generative PRMs based on self-reflection (He et al., 31 Jul 2025). Process supervision for agentic domains must be adapted to notions of progress and exploration rather than absolute correctness (Xi et al., 11 Nov 2025).
• Online Learning Pitfalls: Concurrent policy and reward learning risks manipulability and degenerate incentives unless carefully counterfactually specified or regularized via uninfluenceability/unriggability properties (Armstrong et al., 2020).

6. Extensions, Generalization, and Open Directions

Continued research in PRL spans several axes:

• Process-Outcome Integration: Advanced methods develop sample-efficient hybridization of step and trajectory signals, with robust reward normalization and principle-based assessment in non-verifiable domains (Xu et al., 29 Sep 2025).
• Implicit and Analytical PRMs: Implicit process reward estimation via Monte Carlo prefix-overlap, online trajectory DPO, or policy log-ratios obviates large, separately trained PRMs (Liu et al., 23 Sep 2025, Fei et al., 2 Jul 2025, Sullivan, 25 Sep 2025).
• Active Learning and Automated Supervision: Pool-based active selection, uncertainty sampling, and automated filtering reduce annotation costs and select diverse, informative trajectories (Duan et al., 14 Apr 2025).
• Bidirectional and Multimodal Reasoning: Bidirectional PRMs and multimodal architectures incorporating chain-of-thought reasoning enhance robustness and data efficiency in complex domains (Zhang et al., 3 Aug 2025, Chen et al., 5 Aug 2025).
• Benchmarks and Cross-Domain Application: Process reward frameworks are evaluated on and catalyze the creation of new benchmarks (PRMBench, ProcessBench, VisualProcessBench) spanning math, code, multimodal reasoning, web agents, and robotics (Zheng et al., 9 Oct 2025).

Open challenges include scaling to larger models, deeper process supervision (e.g., thought-level, hierarchical), improved theoretical convergence guarantees, reward adaptivity under non-stationary environments, universal process reward architectures, and robust defense against adversarial exploitation of reward structures (Zheng et al., 9 Oct 2025, Cheng et al., 21 Apr 2025).

7. Summary Table: PRL Algorithmic Instantiations

Reference	Reward Type	Method Summary	Notable Features
(Yao et al., 15 Jan 2026)	Analytical process	Entropy-reg RL, closed-form PRL	Efficient, no extra model
(Zhang et al., 2024)	Analytical process	Soft-value via KL-RL	Optimal step reward from π₀
(Fei et al., 2 Jul 2025)	Analytical process	Self-guided, masked advantage	No PRM, low compute
(Cheng et al., 21 Apr 2025)	Min process/PRM+VR	PURE min-form credit, hybrid	Reward hacking defense
(Liu et al., 23 Sep 2025)	Implicit process	Online PRL via DPO	RL from pairwise prefs
(Ye et al., 3 Sep 2025)	Filtering PRM+ORM	PROF consistency filtering	Avoids entropy collapse
(Xi et al., 11 Nov 2025)	Q-value/prgrs agent	Promise+Progress/TD+GAE	8× compute efficiency
(Duan et al., 14 Apr 2025)	Active PRM, pool	Aleatoric/epistemic active sel	SOTA with 50% less labels
(Zhang et al., 3 Aug 2025)	Bidirectional process	BiPRM, L2R+R2L fusion	+31.9% stepwise eval gain

• (Yao et al., 15 Jan 2026) PRL: Process Reward Learning Improves LLMs' Reasoning Ability and Broadens the Reasoning Boundary (2026)
• (Zhang et al., 2024) Entropy-Regularized Process Reward Model (2024)
• (Fei et al., 2 Jul 2025) Self-Guided Process Reward Optimization with Masked Step Advantage for Process Reinforcement Learning (2025)
• (Cheng et al., 21 Apr 2025) Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning (2025)
• (Liu et al., 23 Sep 2025) Online Process Reward Leanring for Agentic Reinforcement Learning (2025)
• (Ye et al., 3 Sep 2025) Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Training (2025)
• (Xi et al., 11 Nov 2025) AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress (2025)
• (Duan et al., 14 Apr 2025) Efficient Process Reward Model Training via Active Learning (2025)
• (Zhang et al., 3 Aug 2025) The Bidirectional Process Reward Model (2025)
• (Zheng et al., 9 Oct 2025) A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for LLMs (2025)

Process Reward Learning now occupies a central position in LLM alignment, agentic reinforcement learning, and robust multi-step reasoning, providing a mathematically principled and empirically validated basis for dense credit assignment over complex trajectories.