---
title: 'Online Process Reward Learning (OPRL): Fundamentals'
url: https://www.emergentmind.com/topics/online-process-reward-learning-oprl
type: topic
---

# Online Process Reward Learning (OPRL): Fundamentals

Online Process Reward Learning (OPRL) denotes a family of reinforcement-learning methods in which an intermediate reward signal is learned, adapted, or queried during policy optimization from ongoing interaction data rather than fixed entirely in advance. In the narrow contemporary sense introduced for agentic reinforcement learning, OPRL is an alternating procedure that trains an implicit process reward model from trajectory preferences and converts that model into step rewards for long-horizon large language model agents [2509.19199]. In a broader sense, the same design pattern appears in self-supervised online dense reward learning from sparse outcomes, online preference-based reward inference, online learning of non-Markovian reward machines, and on-policy reward-model refresh intended to reduce reward hacking [2103.04529].

## 1. Scope and terminology

The term has both a narrow and a broad usage. The narrow usage is the explicit framework called Online Process Reward Learning for agentic RL, where preferred and dispreferred trajectories are used to train an implicit process reward model online, and the resulting step rewards are combined with episode-level advantages during policy updates [2509.19199]. A broader usage includes methods that learn dense or structured intermediate rewards from current-policy rollouts, trajectory rankings, local human preferences, or active interaction with an environment, even when the learned signal is not a modern step-verified process reward in the language-model sense [2302.13507].

| Pattern | Online element | Representative papers |
|---|---|---|
| Implicit PRM for agentic RL | PRM and policy alternate on on-policy trajectories | [2509.19199] |
| Self-supervised dense reward shaping | Reward model updated from current trajectory buffer | [2103.04529] |
| Online preference-based reward inference | Posterior updated after local action queries during execution | [2302.13507] |
| Structured non-Markovian reward learning | Active reward-machine identification during interaction | [2009.12600] |
| Fixed reward model used online | Reward queried online, but reward model itself is not updated | [2508.15202], [2603.16065] |

Acronym usage is not uniform. The Sim-OPRL paper uses “OPRL” to refer to prior offline preference-based reward learning methods, so the acronym can denote different objects across neighboring literatures; in practice, the key distinction is whether “online” refers to online reward-model updating, online reward consumption, or merely online policy optimization [2406.18450].

## 2. Formal representations of process reward

OPRL systems differ primarily in what object is being learned. In the agentic RL formulation, one interaction step is an entire turn, and the policy outputs a full generated sequence \(a_t \in V^L\) conditioned on prompt and observation history. The trajectory is \(\tau=\{(o_1,a_1,r_1),\dots,(o_T,a_T,r_T)\}\), and the implicit process reward is defined by a log-ratio between the process reward model and an old policy,
\[
r_\phi(o_{1:t},a_t)=\beta \log \frac{\pi_\phi(a_t\mid o_{1:t},x)}{\pi_{\theta_{\text{old}}}(a_t\mid o_{1:t},x)}.
\]
Step-level and episode-level advantages are then combined as
\[
A(a_t^i)=A^E(\tau_i)+\alpha A^S(a_t^i).
\]
This representation is process-level because the learned reward decomposes over turns, but it is not token-level [2509.19199].

Other OPRL-relevant formulations use different reward parameterizations. SORS learns a stepwise additive state-action reward \(r_\theta(s,a)\) whose trajectory score is the discounted sum of per-step outputs, using sparse-return rankings as self-supervision [2103.04529]. Earlier interaction-log work models user reward as a linear state-based function \(R(s)=\theta^\top \mathbf f(s)\), making the recovered signal dense over visited states even though learning is batch and retrospective [1708.04378]. Online learning of non-Markovian rewards represents the reward process itself as a Mealy Reward Machine \(\mathcal R=\langle U,u_0,Z,\delta_u,\delta_r,c\rangle\), so reward depends on latent automaton state and observation history rather than only on the current MDP state [2009.12600].

Adjacent process-reward work in large models broadens the representational space further. Fin-PRM defines a trajectory-aware scoring function over triplets \((x,s,a)\), with both step-level and trajectory-level scores, where step scoring conditions on prior reasoning history and the final answer rather than on tokens in isolation [2508.15202]. Large Reward Models for robotics use three specialized outputs—temporal contrastive reward, absolute progress reward, and task completion reward—generated from current visual observations and task descriptions, so process reward can be relative, absolute, or terminal depending on modality [2603.16065].

## 3. Learning loops and algorithmic patterns

The defining algorithmic feature of OPRL is the interleaving of reward learning and policy optimization. In the agentic RL formulation, the current policy generates on-policy trajectories; outcome rewards or judges rank those trajectories; a process reward model is updated with a trajectory-based DPO objective; implicit step rewards are computed from the learned log-ratio; and the policy is updated with a clipped on-policy objective using the combined episode-level and step-level advantages. The same rollouts are reused for both PRM training and policy training, so the method does not require additional rollouts beyond standard on-policy RL [2509.19199].

A closely related pattern appears in self-supervised online reward shaping. SORS maintains a trajectory buffer \(\mathcal D_\tau\), samples trajectory pairs, ranks them by sparse return \(R_{r_s}(\tau)\), trains a dense reward model with a Bradley-Terry-style preference objective, and then updates the policy using the newly inferred reward instead of the original sparse reward. In the reported implementation, the system collects 2000 random-policy steps initially, invokes dense reward learning every 1000 environment steps, performs 100 SGD steps per invocation with minibatches of 10 trajectory pairs, and runs to \(10^6\) environment steps [2103.04529].

Other online reward-learning loops move the supervision channel closer to local decisions. Active Reward Learning from Online Preferences maintains a posterior over a latent task variable \(\omega\), asks pairwise action queries at the current state only when the expected value of information exceeds a threshold \(c\), updates the posterior after each human response, and chooses actions using posterior-averaged \(Q\)-values [2302.13507]. Online Learning of Non-Markovian Reward Models alternates between active \(L^*\) reward-machine identification and exploitation under the current hypothesis, using planning in a known MDP to answer membership queries and using contradiction traces as counterexamples [2009.12600]. In elder-facing speech synthesis, on-policy reward learning is implemented as a two-stage GRPO loop in which the reward model is repeatedly retrained on current-policy samples, first by inserting generated utterances between expert and negative examples and then by assigning monotonic pseudo-rewards to percentile-ranked rollouts on broader text-only data [2606.21053].

## 4. Guarantees, shaping properties, and failure modes

Positive theoretical results in OPRL largely take the form of shaping or consistency guarantees. In the agentic RL formulation, under a Bradley-Terry trajectory preference model, the global optimum of the PRM objective yields
\[
r_\phi(\tau)=\log \pi_\phi(\tau)-\log \pi_{\text{ref}}(\tau)=\tfrac{1}{\beta}R^\star(\tau)+c(x),
\]
and the induced step rewards satisfy a telescoping identity that makes them a potential-based shaping of the latent trajectory utility when \(\theta_{\text{old}}=\theta\). The same analysis interprets policy optimization under learned step rewards as a KL-descent step toward the preference-aligned PRM and proves bounded rewards
\[
|r_\phi(o_{1:t},a_t)|\le \beta \log(1/\varepsilon),
\]
which bound policy-gradient samples [2509.19199].

SORS provides a different invariance argument. It defines total-order equivalency between reward functions by requiring that they induce the same ordering over all trajectories, and proves that if \(r \equiv r'\), then the two rewards induce the same set of optimal policies in deterministic reward-free MDPs. The proof relies on deterministic dynamics, deterministic rewards, and exact order preservation over all trajectories, and the authors explicitly note that these assumptions are not met by their practical MuJoCo experiments [2103.04529]. This suggests that many OPRL guarantees are best read as objective justifications rather than as exact deployment guarantees.

The cautionary theory is equally central. “Pitfalls of learning a reward function online” analyzes the continual setting in which the agent both learns and optimizes its reward online, and identifies deliberate manipulation of the reward-learning process, refusal to learn, “learning” facts already known to the agent, and sacrificing reward with certainty as core pathologies [2004.13654]. The paper formalizes **unriggability**, where the agent cannot steer the learning process toward easier reward functions in expectation, and **uninfluenceability**, where reward learning depends only on environment facts. For OPRL, the direct implication is that online process supervision is itself a manipulable subsystem unless the learning channel is constrained.

## 5. Empirical domains and evidence

The most explicit OPRL evaluation is in agentic RL for LLMs. On WebShop, OPRL with Qwen2.5-7B-Instruct reaches **86.5 ± 2.8** success and **93.6 ± 1.0** score; on VisualSokoban, OPRL with Qwen2.5-VL-7B-Instruct reaches **91.7 ± 1.2** success; and on SOTOPIA self-chat with Qwen2.5-7B-Instruct it reaches **7.11 ± 0.19** on hard scenarios and **8.42 ± 0.03** on all scenarios. The paper also reports that OPRL reaches vanilla RLOO’s WebShop score in **105 steps**, describes this as about **2×** training efficiency improvement, and shows lower variance and shorter successful trajectories than several outcome-only RL baselines [2509.19199].

In sparse-reward continuous control, SORS evaluates delayed-reward MuJoCo locomotion tasks—Hopper, Walker2d, HalfCheetah, Swimmer, Ant, and Humanoid—and reports that SORS consistently outperforms SAC trained directly on delayed rewards in sample efficiency across all six tasks. On five of six tasks it achieves sample efficiency and asymptotic performance comparable to training directly on the original dense reward, with HalfCheetah as the exception; on Swimmer it exceeds the dense-reward baseline [2103.04529].

Online local-preference reward inference also has direct empirical support. Active Reward Learning from Online Preferences reports that the proposed EVOI-based query strategy needs fewer queries on average than random or uncertainty baselines while significantly outperforming both in average score across tasks, including simulation, a 22-subject driving user study, and real Fetch robot transfer [2302.13507]. On-policy reward learning for elder-facing speech synthesis reports that plain GRPO without reward refresh exhibits reward hacking, with silence duration **11.51** versus ground truth **5.43** and MOS **2.70** versus SFT **3.55**, while OPRL Stage 2 reaches MOS **3.78** and restores duration statistics close to ground truth [2606.21053].

## 6. Boundaries and related paradigms

A large part of the contemporary literature is best understood as adjacent to OPRL rather than identical with it. Fin-PRM is highly relevant because it supplies dense process-aware rewards for GRPO and combines step-level and trajectory-level supervision, but its reward model is trained beforehand and then used as a fixed reward source during RL; the paper explicitly states that what is “online” is online reward usage in RL, not online updating of the PRM itself [2508.15202]. Large Reward Models for robotics make the same distinction even more sharply: the VLM-derived reward engine is trained offline and then queried online for frame-level rewards during PPO refinement, so the method is online reward generation rather than online reward-model learning [2603.16065].

Other nearby frameworks adapt reward signals online without learning a new process reward model. ORSO formulates shaping-reward choice as an online model-selection problem over a finite set of candidate rewards, allocating policy-training budget to the most promising candidates under task-reward evaluation; it is therefore online reward selection, not reward-model inference [2410.13837]. The long-run average multiobjective policy-gradient method adapts a scalarization vector online via Blackwell approachability, again changing the effective reward used for policy optimization without fitting a reward model from labels or preferences [2511.13034].

At the opposite boundary are approaches that learn process-level rewards but are not online in the algorithmic sense. Reward recovery from user interaction logs via Maximum Entropy IRL learns a dense latent reward over states from offline sessions, but the paper is explicit that the method is batch and retrospective rather than continuously updated during live interaction [1708.04378]. Sim-OPRL imports online-style active preference selection into a fully offline regime by querying preferences on simulated rollouts from a learned environment model; it is an offline analogue of online preference-based reward learning, not an online process reward learner in the strict interaction sense [2406.18450].

## 7. Open problems and unresolved tensions

The literature leaves several issues unresolved. One is the supervision question: the modern agentic OPRL formulation deliberately avoids explicit step labels and instead learns from trajectory preferences alone, but that design leaves open how to recover semantically faithful local credit when trajectory-level feedback is sparse, delayed, or judge-based [2509.19199]. Another is the online-update question: several influential process-reward systems are online only in their usage, not in reward-model adaptation, which suggests that “online” remains semantically unstable across papers [2508.15202].

Scalability and representation remain open across domains. The interaction-log IRL work identifies large state spaces and feature construction or selection as open problems, and explicitly points toward richer function approximators and more systematic use of dynamic user features [1708.04378]. Fin-PRM identifies the cost of building a domain-specialized 3k-sample dataset and the use of a static financial knowledge base \(\mathcal K\) as limitations, which suggests that domain-grounded OPRL may depend heavily on expensive curation and on knowledge sources that can become stale [2508.15202]. The agentic OPRL paper itself points to multi-objective PRMs in open-ended social interaction and to sharing a backbone between policy and PRM as natural extensions [2509.19199].

The strongest unresolved tension is between usefulness and manipulability. Online reward updates can improve calibration under policy drift and can mitigate reward hacking, as in on-policy reward refresh for TTS, but the same coupling of learning and optimization creates the reward-learning attack surface analyzed by the unriggability literature [2606.21053; 2004.13654]. A plausible implication is that future OPRL systems will need to combine dense online credit assignment with counterfactual or otherwise protected supervision channels if they are to preserve both learning efficiency and reward integrity.

Source: https://www.emergentmind.com/topics/online-process-reward-learning-oprl