Latent Process Reward Model (LPRM)

• LPRM is a family of reward modeling approaches that computes rewards directly in learned latent state spaces, bypassing noisy high-dimensional observations.
• It integrates with reinforcement learning, diffusion models, and LLMs using methods like latent MPC planning and noise-calibrated pairwise preference learning.
• Empirical findings show that LPRMs enhance computational efficiency, sample effectiveness, and robustness against irrelevant features in complex tasks.

A Latent Process Reward Model (LPRM) is a family of reward modeling approaches in which rewards are computed directly in a learned latent or process state space, as opposed to observation or output space. LPRMs are widely used in reinforcement learning, sequence modeling, and diffusion-based generative modeling, reflecting a paradigm where preference alignment, reward estimation, or policy optimization operates "where the process lives"—namely, on internal or noisy latent representations across trajectories or processes. LPRMs emphasize computational efficiency, robustness to irrelevant or noisy observation features, and principled handling of process/step-level information.

1. Formal Definitions and Core Frameworks

LPRMs abstract away from raw observation spaces and construct a latent state space $\mathcal{Z}$, with reward functions and learning objectives defined directly in $\mathcal{Z}$ or over latent process trajectories. This general framework emerges across several domains:

• Model-based RL latent reward prediction: Observed high-dimensional states $x_t$ are encoded as $z_t = e_\psi(x_t)$; transitions are predicted by $f_\theta(z_t, a_t)$, and the reward predictor $r_\phi(z_t, a_t)$ outputs estimates for immediate reward. Training exclusively on multi-step reward prediction (e.g., by minimizing $$\mathbb{E}[\sum_{k=0}^{K-1} \| r_{t+k} - r_\phi(\hat z_{t+k}, a_{t+k}) \|^2 ]$$) leads to a concise "world-model" in latent space, sufficient for high-performance latent MPC planning (Havens et al., 2019).
• Latent reward models for diffusion: Preference modeling is performed on noisy latent diffusion states $x_t$, with reward heads $r_\theta(x_t, t, c)$ conditioned on both timestep and prompt/context (Zhang et al., 3 Feb 2025, Liu et al., 11 Feb 2026). The reward’s predictive distribution is often parameterized via noise-aware Thurstone or Bradley–Terry pairwise likelihoods, explicitly tying human or preference comparison variance to diffusion noise.
• LLMs and process reward models: Token or process-step-level rewards can be associated with "latent thoughts" (hidden states) $z_{1:T}$ in a generative trajectory. A latent classifier $\mathcal{Z}$0 predicts answer correctness from the sequence of hidden representations, enabling reward-guided optimization of the latent thinking process in LLMs (Du et al., 30 Sep 2025).
• Process reward modeling for RLHF: Certain RL algorithms, e.g., Group Relative Policy Optimization (GRPO), induce a "secret" latent process reward model by partitioning generated sequences into shared-prefix steps and assigning each such step a scalar reward, derived from aggregate outcome-level signals (Sullivan, 25 Sep 2025).

2. Architectures and Noise/Process Conditioning

LPRMs are instantiated with architectures closely coupled to their generative or policy model:

• Diffusion reward architectures reuse pretrained VAEs/UNets, with multi-layer features $\mathcal{Z}$1, timestep conditioning (e.g., sinusoidal embeddings modulating token features), and lightweight scoring heads (e.g., query-former or MLP) to yield scalar latent rewards. Inference-time noise ensembling (scoring and aggregating over multiple $\mathcal{Z}$2) increases robustness (Liu et al., 11 Feb 2026).
• LLM latent reward models employ mean-pooling over per-token hidden states at each thinking step, followed by transformers and MLPs to predict the probability of correct reasoning—operating over $\mathcal{Z}$3 (Du et al., 30 Sep 2025).
• Latent world-models for RL use MLP-based encoders $\mathcal{Z}$4, dynamics $\mathcal{Z}$5, and reward predictors $\mathcal{Z}$6 with low-dimensional $\mathcal{Z}$7 (e.g., $\mathcal{Z}$8 for pendulum tasks, $\mathcal{Z}$9 for multi-cheetah) to ensure inductive bias towards minimal, task-relevant state representations (Havens et al., 2019).

3. Learning Objectives and Preference Likelihoods

Learning in LPRMs centers on reward prediction or preference modeling losses:

• Reward prediction (RL): The sole objective is multi-step reward prediction, training $x_t$0 only for accurate prediction of $x_t$1, with no reconstruction or explicit KL regularization, ensuring that irrelevant observations are ignored (Havens et al., 2019).
• Noise-calibrated preference likelihoods (diffusion): Given noisy states $x_t$2 and context $x_t$3, DiNa-LRM extends the Thurstone model:

$x_t$4

with $x_t$5. This explicitly accounts for comparison uncertainty as diffusion noise increases. Fidelity loss is used for optimization:

$x_t$6

where $x_t$7 is the model’s pairwise preference probability (Liu et al., 11 Feb 2026).

• Pairwise/Bradley–Terry (diffusion LRM): Preference-probability over pairs $x_t$8 at arbitrary diffusion steps $x_t$9, optimized via the pairwise logistic loss as in Bradley–Terry models (Zhang et al., 3 Feb 2025).
• Latent classifier (LLM LRM): Binary cross-entropy on the output $z_t = e_\psi(x_t)$0 versus correctness labels; the reward is then $z_t = e_\psi(x_t)$1 (Du et al., 30 Sep 2025).

4. Process-Level and Step-Level Policy Optimization

LPRMs enable fine-grained alignment and optimization mechanisms:

• Latent MPC (RL): Planning operates exclusively in latent space, e.g., CEM-MPC optimizes action sequences to maximize cumulative latent predicted reward, never reconstructing observations (Havens et al., 2019).
• Latent Preference Optimization (diffusion): LPO uses the LPRM as a critic at every denoising step; for each $z_t = e_\psi(x_t)$2, multiple latent candidates are sampled, scored, and the best-performing branch is followed recursively. Final gradient updates (e.g., DPO-style) regularize against the reference model (Zhang et al., 3 Feb 2025).
• Process reward in LLM training (GRPO): The group-level optimization in GRPO is mathematically equivalent to a latent process reward model, implicitly assigning stepwise rewards to shared-prefix token segments, with per-token advantages derived from average outcome rewards over all completions traversing the same step (Sullivan, 25 Sep 2025).
• Latent Thinking Optimization (LLM reasoning): Optimization distributes probability over sampled latent trajectories $z_t = e_\psi(x_t)$3 according to their latent reward, using closed-form KL-regularized blending with the base distribution, or via acceptance-rejection sampling (Du et al., 30 Sep 2025).

5. Empirical Performance and Comparative Efficiency

Empirical evaluation consistently demonstrates that LPRMs can match or exceed the performance of observation/output-space reward models, with marked efficiency gains:

Setting	Baseline (obs/pixel)	LPRM (latent/process)	% Speedup / RAM/FLOP
Diffusion preference RM	VLM/CLIP	DiNa-LRM, LRM (SDXL)	51.4% RAM, 71.1% reward FLOPs
Step-level diffusion align	VLM+decode+score	LPO (LPRM-based)	2.5–28x faster training
RL model-based planning	State-pred.	LPRM (reward-only)	$z_t = e_\psi(x_t)$4x sample efficiency
LLM process reward (GRPO)	Standard GRPO	$z_t = e_\psi(x_t)$5-GRPO (process norm)	Up to 12% higher EM, 2x faster

LPRMs yield faster preference optimization dynamics, better sample efficiency, and often more stable or faithful optimization signals due to direct alignment with the latent space employed by generative or policy models (Liu et al., 11 Feb 2026, Zhang et al., 3 Feb 2025, Havens et al., 2019, Du et al., 30 Sep 2025, Sullivan, 25 Sep 2025).

6. Handling Irrelevant Features and Robustness

A distinguishing feature of LPRMs is their insensitivity to irrelevant or distracting input features, particularly in high-dimensional observation domains:

• In RL tasks with multi-object inputs (e.g., control of multiple agents where only a subset produce reward), LPRMs trained solely on reward prediction spontaneously learn compact latent embeddings that discard distractors. Empirically, state-reconstruction-based models fail as the number of distractors increases, while LPRMs maintain performance (Havens et al., 2019).
• In diffusion-based image generation, reward aggregation and ensembling across multiple noise levels improve reward stability and dataset robustness. Noise-dependent uncertainty calibration (e.g., $z_t = e_\psi(x_t)$6) further modulates the likelihood to reflect human comparison noise at high-diffusion timesteps (Liu et al., 11 Feb 2026).

7. Limitations, Open Directions, and Theoretical Insights

LPRMs, despite their advantages, exhibit domain-specific limitations:

• LTO for LLMs cannot improve base policies that never reach correct latent trajectories; integrating reward learning into the latent policy’s reinforcement training is an open area (Du et al., 30 Sep 2025).
• Most LPRMs operationalize scalar or binary reward—extensions to multidimensional (e.g., safety, helpfulness) or multi-objective process rewards remain an open challenge.
• In group-based step-level RLHF (e.g., GRPO), latent PRMs can skew exploration/exploitation as a function of shared trajectory structure, motivating normalization (as in $z_t = e_\psi(x_t)$7-GRPO) or explicit design of step-level reward modeling and attribution (Sullivan, 25 Sep 2025).
• Theoretical bounds exist, such as error propagation in reward-guided latent optimization, and characterizations of induced process reward distributions under group sampling (Du et al., 30 Sep 2025, Sullivan, 25 Sep 2025).

---

In sum, Latent Process Reward Models systematically advance the state of preference alignment, planning, and optimization by rooting reward modeling in latent or process step space, yielding efficient, scalable, and more robust optimization across RL, generative modeling, and LLM reasoning (Havens et al., 2019, Zhang et al., 3 Feb 2025, Liu et al., 11 Feb 2026, Du et al., 30 Sep 2025, Sullivan, 25 Sep 2025).