Generative Process Reward Model (GenPRM)

• GenPRM is a process reward model that employs autoregressive language generation to produce detailed verification traces like rationales, step labels, and critiques.
• It integrates generated evaluation sequences directly into reinforcement learning and test-time search, using synthetic supervision and iterative decoding techniques.
• Empirical results demonstrate that GenPRM improves accuracy in mathematical reasoning, operations research, and multimodal reasoning by aligning process incentives with task progress.

Generative Process Reward Model (GenPRM) denotes a class of process reward models implemented as autoregressive generators that evaluate multi-step trajectories by producing verification traces—such as natural-language rationales, step labels, critiques, verdicts, code-based checks, or corrections—rather than only scalar classifier outputs. In recent work, GenPRMs have been formalized as conditional LLMs over a problem, a candidate solution, and a generated evaluation sequence; they have been used as dense reward sources in reinforcement learning, as rerankers and critics for test-time scaling, and as process verifiers in domains including mathematical reasoning, operations research, multimodal reasoning, table question answering, and generative recommendation (Rahman et al., 2 Dec 2025, Zhao et al., 1 Apr 2025, Zhou et al., 26 Sep 2025, Guo et al., 8 Jan 2026).

1. Formal definition and distinguishing characteristics

Process reward models, in the general sense, assign rewards to sub-trajectories rather than only to final outcomes. One formalization defines a PRM as a function

$$f_\phi: \Sigma^* \to (\Sigma^* \times \mathbb{R})^*$$

that segments a trajectory into steps and attaches a scalar reward to each step (Sullivan, 25 Sep 2025). GenPRM specializes this idea by making the reward model itself a generative LLM.

In SPARK, a GenPRM is explicitly defined as a conditional LLM

$$p_\theta\bigl((\tau_1,v_1),\ldots,(\tau_n,v_n),y \mid q,s\bigr),$$

where $q$ is a problem, $s=(s_1,\ldots,s_n)$ is a candidate solution, $v_i \in \{\text{correct},\text{incorrect}\}$ is a step-level verification label, and $y \in \{\text{Yes},\text{No}\}$ is the overall verdict. The model autoregressively emits, for each step, a rationale $\tau_i$ and label $v_i$, followed by the final verdict (Rahman et al., 2 Dec 2025). In StepORLM, the GenPRM $\rho_\phi$ reads an entire reasoning trajectory $\tau=(s_1,\ldots,s_T)$ and generates both a chain-of-thought evaluation $$p_\theta\bigl((\tau_1,v_1),\ldots,(\tau_n,v_n),y \mid q,s\bigr),$$0 and step-level correctness signals $$p_\theta\bigl((\tau_1,v_1),\ldots,(\tau_n,v_n),y \mid q,s\bigr),$$1, with process reward defined as

$$p_\theta\bigl((\tau_1,v_1),\ldots,(\tau_n,v_n),y \mid q,s\bigr),$$2

(Zhou et al., 26 Sep 2025). In the mathematical-reasoning GenPRM of Liu et al., the model first generates a rationale $$p_\theta\bigl((\tau_1,v_1),\ldots,(\tau_n,v_n),y \mid q,s\bigr),$$3 and verification code for each step, executes the code to obtain feedback $$p_\theta\bigl((\tau_1,v_1),\ldots,(\tau_n,v_n),y \mid q,s\bigr),$$4, and then emits “Yes” or “No,” with predicted reward

$$p_\theta\bigl((\tau_1,v_1),\ldots,(\tau_n,v_n),y \mid q,s\bigr),$$5

(Zhao et al., 1 Apr 2025).

These formulations differ in surface form, but they share a common structure: reward estimation is mediated by an explicit generated verification trace rather than a purely discriminative score. This suggests that “generative” in GenPRM refers not only to the model family, but to the operational role of generated reasoning in reward construction.

Work	Generated output	Reward form
SPARK (Rahman et al., 2 Dec 2025)	$$p_\theta\bigl((\tau_1,v_1),\ldots,(\tau_n,v_n),y \mid q,s\bigr),$$6 for each step, then $$p_\theta\bigl((\tau_1,v_1),\ldots,(\tau_n,v_n),y \mid q,s\bigr),$$7	$$p_\theta\bigl((\tau_1,v_1),\ldots,(\tau_n,v_n),y \mid q,s\bigr),$$8
StepORLM (Zhou et al., 26 Sep 2025)	Holistic critique $$p_\theta\bigl((\tau_1,v_1),\ldots,(\tau_n,v_n),y \mid q,s\bigr),$$9 and $q$0	$q$1
GenPRM via generative reasoning (Zhao et al., 1 Apr 2025)	Rationale, code, feedback, “Yes/No”	$q$2
GM-PRM (Zhang et al., 6 Aug 2025)	Critiques/scores plus corrected first error	$q$3
PROMISE path-level PRM (Guo et al., 8 Jan 2026)	Scalar reward for a SID prefix	$q$4

2. Supervision, synthetic labels, and training objectives

A central issue in GenPRM research is how to obtain process supervision without incurring the full cost of manual step annotation. Several papers replace gold process labels with synthetic or solver-backed supervision.

SPARK adopts a three-stage framework. In Stage I, a generator model samples diverse solutions, and a verifier produces multiple independent verifications. Parallel scaling uses self-consistency, either at the outcome level or at the step level, while sequential scaling uses meta-critique through an initial verification, a critique, and a merge step. Stage II converts these synthetic verifications into datasets $q$5, $q$6, and $q$7, and fine-tunes a base LLM with the next-token objective

$q$8

(Rahman et al., 2 Dec 2025).

The GenPRM of Liu et al. constructs supervision through Relative Progress Estimation (RPE). For a step $q$9, it defines

$s=(s_1,\ldots,s_n)$0

and binarizes with threshold $s=(s_1,\ldots,s_n)$1. The training corpus is built from 23K math problems in three stages: large-scale solution generation, RPE labeling, and rationale-plus-code annotation with consensus filtering. The model is then trained by supervised fine-tuning on tuples containing state, step, generated rationale, execution feedback, and reward label (Zhao et al., 1 Apr 2025).

StepORLM obtains step labels from solver-verified trajectories. At each self-evolution iteration, the framework collects trajectories, parses gold step labels $s=(s_1,\ldots,s_n)$2, and continues fine-tuning the GenPRM with

$s=(s_1,\ldots,s_n)$3

(Zhou et al., 26 Sep 2025).

ThinkPRM pushes data efficiency further by fine-tuning a decoder-only transformer on 1,000 synthetic verification chains, approximately 8,000 step labels, or about 1% of PRM800K, using standard next-token cross-entropy over generated verification CoTs (Khalifa et al., 23 Apr 2025). This is not the same as synthetic-verification training in SPARK or RPE-based labeling, but it reflects the same shift away from direct manual process supervision.

Taken together, these methods show that GenPRM training often depends less on hand-written reward labels than on a pipeline for generating, filtering, and aligning verification traces. A plausible implication is that the data-engineering problem is as central as the model architecture.

3. Reward extraction and integration into learning algorithms

GenPRMs are not used only as offline verifiers; they are integrated directly into policy optimization, reward shaping, and inference-time search.

In SPARK, the fine-tuned GenPRM is queried on each rollout. After parsing the output, the process-aware reward is

$s=(s_1,\ldots,s_n)$4

and group-wise normalized advantages are computed as in GRPO. SPARK also defines hybrid variants: Step-Augmented reward,

$s=(s_1,\ldots,s_n)$5

Selective Advantage, and Global Step-Reward (Rahman et al., 2 Dec 2025).

StepORLM integrates GenPRM into Weighted Direct Preference Optimization. When two trajectories have identical solver outcome, the pair weight is

$s=(s_1,\ldots,s_n)$6

whereas $s=(s_1,\ldots,s_n)$7 if one passes and the other fails. This makes process reward a soft weighting signal over preference pairs rather than a direct PPO-style token reward (Zhou et al., 26 Sep 2025).

CAPO uses an off-the-shelf LLM as a GenPRM without further fine-tuning. The model judges a candidate rollout in one pass, returns incorrect step indices, maps them to erroneous tokens, and defines token reward

$s=(s_1,\ldots,s_n)$8

followed by group-relative normalization and a clipped PPO-style objective (Xie et al., 4 Aug 2025). This is a particularly minimal instantiation of GenPRM: the “reward model” is an prompted judge rather than a separately trained verifier.

TP-GRPO moves from step rewards to thought-level rewards. It groups contiguous correct or incorrect steps into coherent thought units and assigns rewards only at the last token of each thought and at the final token of the solution. The token-level advantage is then

$s=(s_1,\ldots,s_n)$9

and the policy is trained with an off-policy GRPO-style loss (He et al., 31 Jul 2025). The stated motivation is to reduce ambiguity in step segmentation and alleviate reward hacking.

At inference time, GenPRMs also support search and reranking. Liu et al. use total process score $v_i \in \{\text{correct},\text{incorrect}\}$0 for Best-of-$v_i \in \{\text{correct},\text{incorrect}\}$1, majority voting on step-wise labels, and critic-style sequential refinement (Zhao et al., 1 Apr 2025). ThinkPRM uses a generated verification CoT and a final yes/no probability to rank candidate solutions or guide beam search (Khalifa et al., 23 Apr 2025). GM-PRM adds a correction mechanism: it identifies the first erroneous step, generates a corrected version, and uses it in Refined Best-of-$v_i \in \{\text{correct},\text{incorrect}\}$2 to continue generation from a repaired prefix (Zhang et al., 6 Aug 2025). PROMISE uses a path-level PRM to score prefixes in hierarchical Semantic ID generation and prune beams during decoding (Guo et al., 8 Jan 2026).

4. Domain-specific variants

Although GenPRM emerged most visibly in mathematical reasoning, the literature already contains several domain-specific reinterpretations.

In operations research, StepORLM defines each step as an emitted component of OR modeling, such as a variable definition, an objective, a constraint, or solver code. Its GenPRM is explicitly holistic: because the model must generate a single coherent critique over all steps, it is intended to capture inter-step dependencies, including cases where an early variable definition is invalidated only by a later constraint (Zhou et al., 26 Sep 2025).

In multimodal mathematical reasoning, GM-PRM decomposes each step into three interpretable sub-scores—intent, visual alignment, and logic—and trains the model to generate a corrected version of the first erroneous step it identifies. The combined reward is formed from a weighted sum of the log sub-scores, and the same generative head can produce both critiques and corrections (Zhang et al., 6 Aug 2025). This extends GenPRM from scalar step correctness to multimodal diagnostic structure.

In table question answering, GenPRM combines textual verification with code execution against the table. A candidate reasoning path $v_i \in \{\text{correct},\text{incorrect}\}$3 is scored by averaging per-step rewards,

$v_i \in \{\text{correct},\text{incorrect}\}$4

and the chain with highest average is selected. The study also evaluates alternative process-verification schemes such as Multi-Dimensional Analysis and Rephrase–React–Analysis, reflecting the difficulty of transferring math-oriented PRMs to semi-structured reasoning (Tang et al., 23 Oct 2025).

In generative recommendation, PROMISE defines a path-level PRM over prefixes of hierarchical Semantic IDs. Instead of verifying natural-language reasoning steps, the PRM scores partial recommendation paths and is trained jointly with the decoder using next-token prediction plus a path-level InfoNCE loss over positive and negative prefixes (Guo et al., 8 Jan 2026). This is still process reward modeling, but the “process” is the incremental construction of a semantic identifier rather than textual chain-of-thought.

These variants indicate that GenPRM is better understood as a modeling pattern than as a single architecture. The shared element is generative evaluation of intermediate structure; the object being evaluated can be a reasoning step, a thought, a multimodal deduction, a table-operation chain, or a path in a recommender decoder.

5. Empirical results

In mathematical reasoning, SPARK reports that step-level consistency produces ProcessBench performance of 67.5 F1, compared to 66.4 for reference-guided training and 61.9 for GPT-4o. In RL experiments with Qwen2.5-Math-7B, PRM-CoT reaches 47.4% average accuracy across six mathematical reasoning benchmarks, compared to 43.9% for ground-truth-based RLVR, and achieves 60.9% Average@16 versus 56.4% for RLVR (Rahman et al., 2 Dec 2025).

The GenPRM of Liu et al. reports that a 1.5B GenPRM with Maj@8 reaches 63.4 on ProcessBench, exceeding GPT-4o critic at 61.9, while a 7B GenPRM reaches 80.5 and outperforms Qwen2.5-Math-PRM-72B at 78.3. The same paper also reports critic refinement gains on AMC23 for Gemma-3-12B: 66.9% as policy alone, 74.0% with one turn of GenPRM critic, and 76.9% after three turns (Zhao et al., 1 Apr 2025).

ThinkPRM reports 87.3% F1 on ProcessBench for ThinkPRM-14B, versus 72.8% for LLM-as-judge and approximately 80% for DiscPRM-14B trained on $v_i \in \{\text{correct},\text{incorrect}\}$5 more labels. It also reports out-of-domain gains of 8% on GPQA-Physics and 4.5% on LiveCodeBench over discriminative verifiers trained on the full PRM800K, and 7.2% higher F1 than LLM-as-a-Judge under the same token budget (Khalifa et al., 23 Apr 2025).

In operations research, StepORLM reports that freezing GenPRM evolution drops average Pass@1 from 81.4% to 77.7%. Using the final GenPRM as a reranker boosts StepORLM’s overall accuracy from 81.4% to 85.6%, and lifts another specialized model, ORLM, by 10 points (Zhou et al., 26 Sep 2025).

For multimodal mathematical reasoning, GM-PRM with Refined-BoN improves average accuracy across several policy models. The reported gains include 28.1% to 34.0% for MiniCPM-V2.6-8B, 43.3% to 47.8% for Qwen2.5-VL-7B, and 49.5% to 52.4% for InternVL3-38B; on WeMath, the largest single-model gain is +12.4 for MiniCPM-V2.6 (Zhang et al., 6 Aug 2025).

For semi-structured table reasoning, GenPRM achieves answer-level exact match of 68.92 on WTQ, 92.42 on TabFact, 56.00 on CRT, and 63.78 on SciTab in the 32B setting. At the step level, GenPRM reaches 93.71 on WTQ, 94.72 on TabFact, 83.22 on CRT, and 85.71 on SciTab. The paper states that adding code verification boosts both answer selection and step-level accuracy by approximately 10 points (Tang et al., 23 Oct 2025).

In recommendation, PROMISE reports that PRM-guided scaling raises $v_i \in \{\text{correct},\text{incorrect}\}$6 from 22.98% for traditional beam search to 30.58% at $v_i \in \{\text{correct},\text{incorrect}\}$7 and 36.37% at $v_i \in \{\text{correct},\text{incorrect}\}$8, and gives industrial retrieval gains from Recall@100 0.1010 to 0.1609 at $v_i \in \{\text{correct},\text{incorrect}\}$9. Online A/B tests report total app usage time +0.121% and total video watch time +0.431% (Guo et al., 8 Jan 2026).

6. Limitations, misconceptions, and adjacent debates

A recurring misconception is that GenPRM necessarily requires expensive human step annotations. Several papers explicitly contest this. SPARK constructs synthetic process supervision through self-consistency and meta-critique without ground truth (Rahman et al., 2 Dec 2025). Liu et al. use RPE, rationale synthesis, and code verification on 23K math problems (Zhao et al., 1 Apr 2025). ThinkPRM uses 1,000 synthetic chains and approximately 8,000 step labels, or about 1% of PRM800K (Khalifa et al., 23 Apr 2025). CAPO avoids fine-tuning the reward model altogether by prompting an off-the-shelf LLM to produce step-wise critiques in one pass (Xie et al., 4 Aug 2025).

A second misconception is that better step verification automatically yields better end-task accuracy. The table-QA study reports Pearson correlation near zero between a chain’s average step score and its exact-match correctness, and attributes this to locally correct but irrelevant steps, single local errors that do not affect the final answer, and loosely chained reasoning in TQA (Tang et al., 23 Oct 2025). This suggests that reward aggregation is task-dependent: uniform averaging may be appropriate in some domains and misleading in others.

A third issue is cost. The math GenPRM with CoT and code verification states that inference overhead is non-negligible due to CoT+code generation and execution (Zhao et al., 1 Apr 2025). GM-PRM and PROMISE show that test-time scaling can be beneficial, but they do so by adding extra verification or reranking compute (Zhang et al., 6 Aug 2025, Guo et al., 8 Jan 2026). By contrast, the GRPO analysis argues that standard GRPO already induces a non-trivial Monte-Carlo PRM under specific assumptions and that explicitly defined PRMs may be unnecessary for some GRPO-based training regimes. That paper identifies a flaw in standard GRPO due to non-uniformly distributed process steps, proposes $y \in \{\text{Yes},\text{No}\}$0-GRPO, and reports higher validation accuracy, better downstream reasoning performance, faster peak performance, and essentially unchanged wall-clock time because the method only requires an $y \in \{\text{Yes},\text{No}\}$1 lookup of process-set size (Sullivan, 25 Sep 2025).

The resulting picture is not that GenPRM has replaced other reward models, but that it has made the design space of reward modeling more explicit. Some works use explicit generated verification traces to improve supervision, some use them to scale test-time search, and some argue that related process structure is already latent in existing RL objectives. A plausible implication is that future work will depend less on whether a system is “generative” or “discriminative” in name, and more on how faithfully its reward signal tracks causal progress through a task.