---
title: Generative Process Reward Model (GenPRM)
url: https://www.emergentmind.com/topics/generative-process-reward-model-genprm
type: topic
---

# Generative Process Reward Model (GenPRM)

Generative Process Reward Model (GenPRM) denotes a class of process reward models implemented as autoregressive generators that evaluate multi-step trajectories by producing verification traces—such as natural-language rationales, step labels, critiques, verdicts, code-based checks, or corrections—rather than only scalar classifier outputs. In recent work, GenPRMs have been formalized as conditional language models over a problem, a candidate solution, and a generated evaluation sequence; they have been used as dense reward sources in reinforcement learning, as rerankers and critics for test-time scaling, and as process verifiers in domains including mathematical reasoning, operations research, multimodal reasoning, table question answering, and generative recommendation [2512.03244][2504.00891][2509.22558][2601.04674].

## 1. Formal definition and distinguishing characteristics

Process reward models, in the general sense, assign rewards to sub-trajectories rather than only to final outcomes. One formalization defines a PRM as a function
$$
f_\phi: \Sigma^* \to (\Sigma^* \times \mathbb{R})^*
$$
that segments a trajectory into steps and attaches a scalar reward to each step [2509.21154]. GenPRM specializes this idea by making the reward model itself a generative language model.

In SPARK, a GenPRM is explicitly defined as a conditional language model
$$
p_\theta\bigl((\tau_1,v_1),\ldots,(\tau_n,v_n),y \mid q,s\bigr),
$$
where $q$ is a problem, $s=(s_1,\ldots,s_n)$ is a candidate solution, $v_i \in \{\text{correct},\text{incorrect}\}$ is a step-level verification label, and $y \in \{\text{Yes},\text{No}\}$ is the overall verdict. The model autoregressively emits, for each step, a rationale $\tau_i$ and label $v_i$, followed by the final verdict [2512.03244]. In StepORLM, the GenPRM $\rho_\phi$ reads an entire reasoning trajectory $\tau=(s_1,\ldots,s_T)$ and generates both a chain-of-thought evaluation $C(\tau)$ and step-level correctness signals $\{\hat y_t\}_{t=1}^T$, with process reward defined as
$$
r_{\text{process}}(\tau)=\frac{1}{T}\sum_{t=1}^T \hat y_t
$$
[2509.22558]. In the mathematical-reasoning GenPRM of Liu et al., the model first generates a rationale $v_t$ and verification code for each step, executes the code to obtain feedback $f_t$, and then emits “Yes” or “No,” with predicted reward
$$
\hat r_t=P_\psi(\text{“Yes”}\mid s_t,a_t,v_{1:t-1},f_{1:t-1},v_t,f_t)
$$
[2504.00891].

These formulations differ in surface form, but they share a common structure: reward estimation is mediated by an explicit generated verification trace rather than a purely discriminative score. This suggests that “generative” in GenPRM refers not only to the model family, but to the operational role of generated reasoning in reward construction.

| Work | Generated output | Reward form |
|---|---|---|
| SPARK [2512.03244] | $(\tau_i,v_i)$ for each step, then $y$ | $r_{\text{process}}(o_i)=\mathbf{1}[\text{format\_ok}] \cdot (y==\text{"Yes"}?1:0)$ |
| StepORLM [2509.22558] | Holistic critique $C(\tau)$ and $\hat y_t$ | $r_{\text{process}}(\tau)=\frac{1}{T}\sum_t \hat y_t$ |
| GenPRM via generative reasoning [2504.00891] | Rationale, code, feedback, “Yes/No” | $R(\tau)=\sum_{t=1}^T r_t$ |
| GM-PRM [2508.04088] | Critiques/scores plus corrected first error | $S(\tau)=\frac{1}{|\tau|}\sum_i f_\theta(s_i,\tau_{<i})$ |
| PROMISE path-level PRM [2601.04674] | Scalar reward for a SID prefix | $R_b(u,c,\mathcal{S}) \equiv y_{\mathcal{S}}$ |

## 2. Supervision, synthetic labels, and training objectives

A central issue in GenPRM research is how to obtain process supervision without incurring the full cost of manual step annotation. Several papers replace gold process labels with synthetic or solver-backed supervision.

SPARK adopts a three-stage framework. In Stage I, a generator model samples diverse solutions, and a verifier produces multiple independent verifications. Parallel scaling uses self-consistency, either at the outcome level or at the step level, while sequential scaling uses meta-critique through an initial verification, a critique, and a merge step. Stage II converts these synthetic verifications into datasets $D_{\mathrm{ORM}}$, $D_{\mathrm{PRM}}$, and $D_{\mathrm{PRM\mbox{-}CoT}}$, and fine-tunes a base LLM with the next-token objective
$$
L(\theta) = - \sum_{(q,s,v)\in D} \sum_{t=1}^{L(v)} \log p_\theta(v_t \mid q,s,v_{<t})
$$
[2512.03244].

The GenPRM of Liu et al. constructs supervision through Relative Progress Estimation (RPE). For a step $(s_t,a_t)$, it defines
$$
P_t = \frac{MC(s_t,a_t)}{MC(s_t)}
$$
and binarizes with threshold $\epsilon=0.8$. The training corpus is built from 23K math problems in three stages: large-scale solution generation, RPE labeling, and rationale-plus-code annotation with consensus filtering. The model is then trained by supervised fine-tuning on tuples containing state, step, generated rationale, execution feedback, and reward label [2504.00891].

StepORLM obtains step labels from solver-verified trajectories. At each self-evolution iteration, the framework collects trajectories, parses gold step labels $y_t \in \{0,1\}$, and continues fine-tuning the GenPRM with
$$
L_{\mathrm{PRM}}(\phi) = - \sum_{\tau \in D} \sum_{t=1}^{T(\tau)}
\Bigl[
y_t \log p_\phi(\text{“Step t: Correct”}\mid \tau)
+
(1-y_t)\log p_\phi(\text{“Step t: Error”}\mid \tau)
\Bigr]
$$
[2509.22558].

ThinkPRM pushes data efficiency further by fine-tuning a decoder-only transformer on 1,000 synthetic verification chains, approximately 8,000 step labels, or about 1% of PRM800K, using standard next-token cross-entropy over generated verification CoTs [2504.16828]. This is not the same as synthetic-verification training in SPARK or RPE-based labeling, but it reflects the same shift away from direct manual process supervision.

Taken together, these methods show that GenPRM training often depends less on hand-written reward labels than on a pipeline for generating, filtering, and aligning verification traces. A plausible implication is that the data-engineering problem is as central as the model architecture.

## 3. Reward extraction and integration into learning algorithms

GenPRMs are not used only as offline verifiers; they are integrated directly into policy optimization, reward shaping, and inference-time search.

In SPARK, the fine-tuned GenPRM is queried on each rollout. After parsing the output, the process-aware reward is
$$
r_{\text{process}}(o_i)=\mathbf{1}[\text{format\_ok}] \cdot (y==\text{"Yes"}?1:0),
$$
and group-wise normalized advantages are computed as in GRPO. SPARK also defines hybrid variants: Step-Augmented reward,
$$
r_{\text{step-aug}}(o_i)=\mathbf{1}[\text{format\_ok}] \cdot \Bigl[0.4\cdot \frac{\sum_k \mathbf{1}[v_k=\text{correct}]}{K}+0.6\cdot y\Bigr],
$$
Selective Advantage, and Global Step-Reward [2512.03244].

StepORLM integrates GenPRM into Weighted Direct Preference Optimization. When two trajectories have identical solver outcome, the pair weight is
$$
w_{(\tau_w,\tau_l)} = |r_{\text{process}}(\tau_w)-r_{\text{process}}(\tau_l)|,
$$
whereas $w=1$ if one passes and the other fails. This makes process reward a soft weighting signal over preference pairs rather than a direct PPO-style token reward [2509.22558].

CAPO uses an off-the-shelf LLM as a GenPRM without further fine-tuning. The model judges a candidate rollout in one pass, returns incorrect step indices, maps them to erroneous tokens, and defines token reward
$$
R_t^i = r_v \cdot W_{\text{whole}} - \mathbf{1}(t \in T_{\text{err}}^i)\cdot W_{\text{process}},
$$
followed by group-relative normalization and a clipped PPO-style objective [2508.02298]. This is a particularly minimal instantiation of GenPRM: the “reward model” is an prompted judge rather than a separately trained verifier.

TP-GRPO moves from step rewards to thought-level rewards. It groups contiguous correct or incorrect steps into coherent thought units and assigns rewards only at the last token of each thought and at the final token of the solution. The token-level advantage is then
$$
A_{i,t}=\sum_{t'=t}^{|o_i|} r_{i,t'}
$$
and the policy is trained with an off-policy GRPO-style loss [2507.23317]. The stated motivation is to reduce ambiguity in step segmentation and alleviate reward hacking.

At inference time, GenPRMs also support search and reranking. Liu et al. use total process score $R(\tau^i)=\sum_t \hat r_t^i$ for Best-of-$N$, majority voting on step-wise labels, and critic-style sequential refinement [2504.00891]. ThinkPRM uses a generated verification CoT and a final yes/no probability to rank candidate solutions or guide beam search [2504.16828]. GM-PRM adds a correction mechanism: it identifies the first erroneous step, generates a corrected version, and uses it in Refined Best-of-$N$ to continue generation from a repaired prefix [2508.04088]. PROMISE uses a path-level PRM to score prefixes in hierarchical Semantic ID generation and prune beams during decoding [2601.04674].

## 4. Domain-specific variants

Although GenPRM emerged most visibly in mathematical reasoning, the literature already contains several domain-specific reinterpretations.

In operations research, StepORLM defines each step as an emitted component of OR modeling, such as a variable definition, an objective, a constraint, or solver code. Its GenPRM is explicitly holistic: because the model must generate a single coherent critique over all steps, it is intended to capture inter-step dependencies, including cases where an early variable definition is invalidated only by a later constraint [2509.22558].

In multimodal mathematical reasoning, GM-PRM decomposes each step into three interpretable sub-scores—intent, visual alignment, and logic—and trains the model to generate a corrected version of the first erroneous step it identifies. The combined reward is formed from a weighted sum of the log sub-scores, and the same generative head can produce both critiques and corrections [2508.04088]. This extends GenPRM from scalar step correctness to multimodal diagnostic structure.

In table question answering, GenPRM combines textual verification with code execution against the table. A candidate reasoning path $S=(s_1,\ldots,s_K)$ is scored by averaging per-step rewards,
$$
\mathrm{score}(S)=\frac{1}{K}\sum_{i=1}^K r_i,
$$
and the chain with highest average is selected. The study also evaluates alternative process-verification schemes such as Multi-Dimensional Analysis and Rephrase–React–Analysis, reflecting the difficulty of transferring math-oriented PRMs to semi-structured reasoning [2510.20304].

In generative recommendation, PROMISE defines a path-level PRM over prefixes of hierarchical Semantic IDs. Instead of verifying natural-language reasoning steps, the PRM scores partial recommendation paths and is trained jointly with the decoder using next-token prediction plus a path-level InfoNCE loss over positive and negative prefixes [2601.04674]. This is still process reward modeling, but the “process” is the incremental construction of a semantic identifier rather than textual chain-of-thought.

These variants indicate that GenPRM is better understood as a modeling pattern than as a single architecture. The shared element is generative evaluation of intermediate structure; the object being evaluated can be a reasoning step, a thought, a multimodal deduction, a table-operation chain, or a path in a recommender decoder.

## 5. Empirical results

In mathematical reasoning, SPARK reports that step-level consistency produces ProcessBench performance of 67.5 F1, compared to 66.4 for reference-guided training and 61.9 for GPT-4o. In RL experiments with Qwen2.5-Math-7B, PRM-CoT reaches 47.4% average accuracy across six mathematical reasoning benchmarks, compared to 43.9% for ground-truth-based RLVR, and achieves 60.9% Average@16 versus 56.4% for RLVR [2512.03244].

The GenPRM of Liu et al. reports that a 1.5B GenPRM with Maj@8 reaches 63.4 on ProcessBench, exceeding GPT-4o critic at 61.9, while a 7B GenPRM reaches 80.5 and outperforms Qwen2.5-Math-PRM-72B at 78.3. The same paper also reports critic refinement gains on AMC23 for Gemma-3-12B: 66.9% as policy alone, 74.0% with one turn of GenPRM critic, and 76.9% after three turns [2504.00891].

ThinkPRM reports 87.3% F1 on ProcessBench for ThinkPRM-14B, versus 72.8% for LLM-as-judge and approximately 80% for DiscPRM-14B trained on $\times 100$ more labels. It also reports out-of-domain gains of 8% on GPQA-Physics and 4.5% on LiveCodeBench over discriminative verifiers trained on the full PRM800K, and 7.2% higher F1 than LLM-as-a-Judge under the same token budget [2504.16828].

In operations research, StepORLM reports that freezing GenPRM evolution drops average Pass@1 from 81.4% to 77.7%. Using the final GenPRM as a reranker boosts StepORLM’s overall accuracy from 81.4% to 85.6%, and lifts another specialized model, ORLM, by 10 points [2509.22558].

For multimodal mathematical reasoning, GM-PRM with Refined-BoN improves average accuracy across several policy models. The reported gains include 28.1% to 34.0% for MiniCPM-V2.6-8B, 43.3% to 47.8% for Qwen2.5-VL-7B, and 49.5% to 52.4% for InternVL3-38B; on WeMath, the largest single-model gain is +12.4 for MiniCPM-V2.6 [2508.04088].

For semi-structured table reasoning, GenPRM achieves answer-level exact match of 68.92 on WTQ, 92.42 on TabFact, 56.00 on CRT, and 63.78 on SciTab in the 32B setting. At the step level, GenPRM reaches 93.71 on WTQ, 94.72 on TabFact, 83.22 on CRT, and 85.71 on SciTab. The paper states that adding code verification boosts both answer selection and step-level accuracy by approximately 10 points [2510.20304].

In recommendation, PROMISE reports that PRM-guided scaling raises $\mathrm{HRecall}@3@1000$ from 22.98% for traditional beam search to 30.58% at $K^+=4000$ and 36.37% at $K^+=6000$, and gives industrial retrieval gains from Recall@100 0.1010 to 0.1609 at $K^+=6000$. Online A/B tests report total app usage time +0.121% and total video watch time +0.431% [2601.04674].

## 6. Limitations, misconceptions, and adjacent debates

A recurring misconception is that GenPRM necessarily requires expensive human step annotations. Several papers explicitly contest this. SPARK constructs synthetic process supervision through self-consistency and meta-critique without ground truth [2512.03244]. Liu et al. use RPE, rationale synthesis, and code verification on 23K math problems [2504.00891]. ThinkPRM uses 1,000 synthetic chains and approximately 8,000 step labels, or about 1% of PRM800K [2504.16828]. CAPO avoids fine-tuning the reward model altogether by prompting an off-the-shelf LLM to produce step-wise critiques in one pass [2508.02298].

A second misconception is that better step verification automatically yields better end-task accuracy. The table-QA study reports Pearson correlation near zero between a chain’s average step score and its exact-match correctness, and attributes this to locally correct but irrelevant steps, single local errors that do not affect the final answer, and loosely chained reasoning in TQA [2510.20304]. This suggests that reward aggregation is task-dependent: uniform averaging may be appropriate in some domains and misleading in others.

A third issue is cost. The math GenPRM with CoT and code verification states that inference overhead is non-negligible due to CoT+code generation and execution [2504.00891]. GM-PRM and PROMISE show that test-time scaling can be beneficial, but they do so by adding extra verification or reranking compute [2508.04088][2601.04674]. By contrast, the GRPO analysis argues that standard GRPO already induces a non-trivial Monte-Carlo PRM under specific assumptions and that explicitly defined PRMs may be unnecessary for some GRPO-based training regimes. That paper identifies a flaw in standard GRPO due to non-uniformly distributed process steps, proposes $\lambda$-GRPO, and reports higher validation accuracy, better downstream reasoning performance, faster peak performance, and essentially unchanged wall-clock time because the method only requires an $O(1)$ lookup of process-set size [2509.21154].

The resulting picture is not that GenPRM has replaced other reward models, but that it has made the design space of reward modeling more explicit. Some works use explicit generated verification traces to improve supervision, some use them to scale test-time search, and some argue that related process structure is already latent in existing RL objectives. A plausible implication is that future work will depend less on whether a system is “generative” or “discriminative” in name, and more on how faithfully its reward signal tracks causal progress through a task.

Source: https://www.emergentmind.com/topics/generative-process-reward-model-genprm