Process-supervised Reward Models (PRMs)

Updated 29 November 2025

Process-supervised Reward Models (PRMs) are learned evaluation functions that provide granular, step-level feedback across multi-step reasoning trajectories.
They integrate diverse annotation methods—human, automated, and LLM-based—to achieve precise credit assignment and robust training in complex tasks.
Recent advancements in PRMs improve error detection and guide reinforcement learning policies, with applications spanning math, code synthesis, and multimodal reasoning.

A Process-supervised Reward Model (PRM) is a learned function that evaluates the correctness or utility of each intermediate step in a multi-step reasoning trajectory produced by a model, in contrast to traditional outcome reward models (ORMs) that score only the final output. By providing dense, granular supervision at the process level, PRMs enable more precise credit assignment, early error detection, and fine-grained guidance in both supervised and reinforcement learning pipelines. This paradigm has seen rapid advances across tasks such as mathematical reasoning, code synthesis, text generation, multimodal and domain-specific applications, and has evolved alongside sophisticated methodologies for scalable reward signal acquisition and robust model training.

1. Definition, Formalism, and Distinction from Outcome Reward Models

Process-supervised Reward Models supplant the outcome-only signal of traditional reward modeling with stepwise supervision over each partial state or action in a chain-of-thought (CoT) trajectory. Formally, for a reasoning task input $x$ , and a candidate sequence of intermediate states/steps/actions $s = (s_1,\dots,s_n)$ , a PRM defines per-step rewards: $r_t = R_\theta(x, s_{1:t}) \in (0,1)$ where $R_\theta$ typically outputs the probability that step $t$ is correct given its context, with discriminative architectures employing sigmoid heads, and generative approaches outputting stepwise CoT critiques or binary verdicts (Zheng et al., 9 Oct 2025, Duan et al., 14 Apr 2025, Khalifa et al., 23 Apr 2025).

Distinction from ORMs:

Granularity: PRMs deliver dense, local feedback; ORMs provide only a single reward for the completed trajectory.
Policy Guidance: PRMs enable incremental rejection or correction at the first sign of error, supporting reward-guided search, beam selection, and step-aware RL objectives.
Aggregation: Final solution scores are commonly aggregated via stepwise products, sums, or structure-aware reductions such as minimums for robustness (Zhang et al., 13 Jan 2025).

2. Process-Level Data Generation: Human, Automated, and Hybrid Supervision

A central challenge in PRM development is the acquisition of high-fidelity step-level supervision. Data construction strategies span a fidelity-scalability spectrum (Zheng et al., 9 Oct 2025):

Human Annotation: PRM800K and similar corpora employ domain experts for stepwise labeling, yielding gold-standard but costly labels (~10⁴–10⁵ samples) (Duan et al., 14 Apr 2025).
Automated Verification: Methods such as MathShepherd, OmegaPRM, URSA apply symbolic solvers, MCTS, or runtime tests (for code) to pinpoint the first error or verify step correctness at scale. These methods introduce label noise due to misattribution and artifacts of non-human rationale (Zhang et al., 16 Oct 2025, Zhang et al., 7 May 2025).
LLM-as-Judge: Annotation is bootstrapped from strong LLMs prompted to provide step-level verification, offering higher fidelity than MC estimation but suffering potential hallucination and high compute cost (Zhang et al., 13 Jan 2025).
Active Learning / Filtering: Pool-based active learning approaches (e.g., ActPRM) select only high-uncertainty or disagreement cases for gold annotation, reducing overall labeling budget by over 50% while matching accuracy of full data regimes (Duan et al., 14 Apr 2025).
Weakly-Supervised Pseudo-labeling: FreePRM dispenses with step labels entirely by attributing all steps to the outcome and mitigating noise with a buffer probability mechanism (Sun et al., 4 Jun 2025).
Hierarchical, Error-typed, or Domain-informed Labeling: In complex settings, labels include error types (e.g., math vs. consistency errors in PathFinder-PRM (Pala et al., 26 May 2025); taxonomy-based feedback in SWE (Gandhi et al., 2 Sep 2025); template alignment in ReasonFlux-PRM (Zou et al., 23 Jun 2025)).

Data Source	Fidelity	Scale	Notable Models
Human annotation	High	Limited	PRM800K, PathFinder-PRM
Automated MC	Medium	High	MathShepherd, OmegaPRM
LLM-as-Judge	High	Medium	ActPRM, PathFinder-PRM
Active learning/filter	High	High	ActPRM, ReasonFlux-PRM
Weak/outcome labeling	Low/Var.	Very High	FreePRM

3. Architectures, Training Objectives, and Error Typing

Architectures:

Discriminative PRMs: Transformer encoder with scalar head, trained with binary cross-entropy/MSE or pairwise ranking. Variants include multi-head (aleatoric/epistemic uncertainty) ensembles, error-type specific branches, and token-wise (as in Q-RM, MT-PRM) critics (Duan et al., 14 Apr 2025, Feng et al., 15 Mar 2025, Pala et al., 26 May 2025).
Generative PRMs: Language-model-based verifiers output stepwise CoT critiques and explicit verdicts (e.g., ThinkPRM, GroundedPRM), trained with next-token prediction over rationales plus judgments, allowing verbalized error explanations and scalable compute (Khalifa et al., 23 Apr 2025, Zhang et al., 16 Oct 2025).
Hierarchical and Multi-dimensional: PathFinder-PRM models math and consistency errors via separate heads and composes a higher-level reward, improving fine-grained error detection and data efficiency (Pala et al., 26 May 2025).
Domain-informed PRMs: Architectures incorporate domain logic, e.g., step/trajectory dual branches in finance (Fin-PRM), taxonomy-conditioned feedback in SWE, SQL-chain-of-CTEs for Text-to-SQL (Zhou et al., 21 Aug 2025, Gandhi et al., 2 Sep 2025, Zhang et al., 7 May 2025).
Implicit and Weakly-Supervised: Buffer probability heads to absorb pseudo-label noise (FreePRM) (Sun et al., 4 Jun 2025), stepwise reward distillation via ORM (SP-PRM) (Xie et al., 14 Jun 2025).

Training Objectives:

Pointwise BCE:

$L_\text{BCE} = -\frac{1}{|s|} \sum_i [y_i\log p_\theta(s_i) + (1-y_i)\log(1-p_\theta(s_i))]$

Multi-task loss: Parallel error heads + reward estimation, as in PathFinder-PRM (Pala et al., 26 May 2025).
Preference Optimization: Direct Preference Optimization (DPO), margin-ranking, and group-relative policy optimization (GRPO) scale process preferences to reinforcement learning (Zhang et al., 7 May 2025, Yin et al., 23 Jul 2025, Zhou et al., 21 Aug 2025, Zou et al., 23 Jun 2025).
Process-aware aggregation: Hybrid rewards blending local tool-verified and global outcome signals (GroundedPRM), trajectory-level aggregation for template alignment (ReasonFlux-PRM) (Zhang et al., 16 Oct 2025, Zou et al., 23 Jun 2025).

4. Data Efficiency, Annotation Bottlenecks, and Active/Uncertainty Sampling

A key problem in PRM development is annotation cost. PRMs trained naïvely on large auto-labeled sets suffer from label noise and diminishing returns (Zhang et al., 13 Jan 2025). Addressing this, modern methods introduce:

Active Learning (ActPRM): An ensemble of PRMs estimates epistemic and aleatoric uncertainty on candidate trajectories; only high-uncertainty cases are labeled by an LLM judge. This sacrifices less than 2% performance but cuts annotation cost by ≳50% compared to full-data fine-tuning (Duan et al., 14 Apr 2025).
Entropy-Driven Partition (EDU-PRM): Dynamic step segmentation based on logit entropy, focusing labels where the model is most indecisive, achieves 98% reduction in query cost while matching SOTA (Cao et al., 28 Mar 2025).
Coarse-to-Fine Curriculum: CFPRM merges trivial steps at coarse granularity, then refines, demonstrating that both redundancy reduction and fine-to-coarse curriculum increase BoN accuracy by 1–3 points and mitigate error masking (Hu et al., 23 Jan 2025).
Weak Supervision (FreePRM): Step pseudo-labels inherited from the outcome signal, combined with a buffer-head to absorb noisy gradients, yield F1 scores surpassing some fully supervised PRMs (e.g., +10.9 pp over Skywork-PRM-7B), at zero step annotation cost (Sun et al., 4 Jun 2025).

5. Integration with RL and Inference-Time Guidance

PRMs are integrated in both online (on-policy) and offline (post-hoc) settings:

Test-Time Scaling: PRMs guide beam search, best-of-N selection, or greedy search, enabling rejection or early stopping at the first flagged error. HGS-PRM and reward-guided selection on math/code benchmarks show gains of 1–5+ points over CoT ranking and vanilla outcome models (Ma et al., 2023, Khalifa et al., 23 Apr 2025, Wang et al., 13 Mar 2025).
Reward-Guided RL: PRMs replace terminal RL rewards with dense r_t in PPO or GRPO, stabilizing policy updates and improving sample efficiency. Group normalization (GRPO) and hybrid local/global rewards mitigate reward hacking and optimize exploration, yielding consistent gains in code, math, SQL, and finance (Zhang et al., 7 May 2025, Zhou et al., 21 Aug 2025, Zou et al., 23 Jun 2025).
Domain- and Modality-Extensions: VisualPRM extends stepwise rewards to multimodal reasoning (e.g., VQA, scientific charts), outperforming outcome RMs and boosting reasoning accuracy by 6–9 points under BoN decoding (Wang et al., 13 Mar 2025). PRMs now guide clinical note verification (Wang et al., 17 Dec 2024), finance (Zhou et al., 21 Aug 2025), machine translation (Feng et al., 15 Mar 2025), and agentic planning (Gandhi et al., 2 Sep 2025).
Reward Consistency and Alignment: SP-PRM distills PRM supervision from ORMs by enforcing score and preference consistency on prefixes, resolving the granularity mismatch for reward-guided search in summarization, dialogue, and reasoning (Xie et al., 14 Jun 2025).

6. Evaluation: Metrics, Benchmarks, and Data Curation

Reliably evaluating PRMs requires stepwise and trajectory-level assessments:

ProcessBench & PRMBench: Standard metrics include macro-F1 for stepwise error identification, earliest error recall, and composite scores over soundness, simplicity, and sensitivity (Duan et al., 14 Apr 2025, Pala et al., 26 May 2025).
Best-of-N (BoN) Accuracy: For N candidate completions, the process score is aggregated (product or minimum) and used to select the output, with PRMs regularly outperforming outcome-only models and LLM-as-Judge critics (Duan et al., 14 Apr 2025, Khalifa et al., 23 Apr 2025).
Domain-Specific Benchmarks: VisualProcessBench assigns human labels for multimodal reasoning; AIME, GPQA-Diamond, and MATH500 test trajectory-response and template-aware PRMs (Zou et al., 23 Jun 2025, Wang et al., 13 Mar 2025).
Evaluation Pathologies: MC-estimation PRMs, if left unchecked, conflate future potential with current correctness, leading to misattributed credit and process-to-outcome bias. Consensus filtering and tool verification reduce these artifacts (Zhang et al., 13 Jan 2025, Zhang et al., 16 Oct 2025).
Empirical Advances: SOTA PRMs, including ActPRM, PathFinder-PRM, DG-PRM, and ReasonFlux-PRM, deliver improvements of 1–6.3+ points on standard math and science benchmarks with reduced annotation or compute (Duan et al., 14 Apr 2025, Pala et al., 26 May 2025, Yin et al., 23 Jul 2025, Zou et al., 23 Jun 2025).

7. Open Challenges, Innovations, and Future Directions

Annotation and Robustness: There is an inherent challenge in scaling annotation with high fidelity; hybrid (human+MC+LLM) and self-supervised protocols (EDU-PRM, ActPRM) are major research foci (Duan et al., 14 Apr 2025, Cao et al., 28 Mar 2025).
Hierarchical and Typed Rewards: Decoupling error types and hierarchical, multi-granular feedback (as in PathFinder-PRM, HRM, DG-PRM) enhances diagnostic accuracy and cross-domain generalization (Pala et al., 26 May 2025, Yin et al., 23 Jul 2025).
Generative and Interpretable Verification: Generative PRMs (GroundedPRM, ThinkPRM) support rationale-enhanced and scalable inference, surpassing discriminative verifiers and LLM judges with minimal supervision (Zhang et al., 16 Oct 2025, Khalifa et al., 23 Apr 2025).
Domain Extension: Rapid growth in domain-adapted and multi-modal PRMs (code, finance, clinical, translation, multimodal reasoning) highlights the flexibility and transfer potential of process-level supervision (Zhou et al., 21 Aug 2025, Wang et al., 17 Dec 2024, Wang et al., 13 Mar 2025).
Open Problems: Key areas for exploration include: reward signal calibration, OOD robustness, efficiency in very long chains, integration with symbolic reasoning, memory/planning, and adaptation to dynamic, safety-critical, or agentic workflows (Zheng et al., 9 Oct 2025, Gandhi et al., 2 Sep 2025).