Online Process Reward Learning (OPRL): Fundamentals

Updated 12 July 2026

OPRL is a reinforcement learning framework that learns intermediate reward signals online from trajectory preferences, seamlessly integrating reward learning with policy updates.
It employs an alternating loop where on-policy trajectories are used to train an implicit reward model via techniques like Bradley-Terry objectives, enhancing training efficiency.
OPRL improves sample efficiency, mitigates reward hacking risks, and supports applications in large language models, robotics, and other domains through contextual reward shaping.

Online Process Reward Learning (OPRL) denotes a family of reinforcement-learning methods in which an intermediate reward signal is learned, adapted, or queried during policy optimization from ongoing interaction data rather than fixed entirely in advance. In the narrow contemporary sense introduced for agentic reinforcement learning, OPRL is an alternating procedure that trains an implicit process reward model from trajectory preferences and converts that model into step rewards for long-horizon LLM agents (Liu et al., 23 Sep 2025). In a broader sense, the same design pattern appears in self-supervised online dense reward learning from sparse outcomes, online preference-based reward inference, online learning of non-Markovian reward machines, and on-policy reward-model refresh intended to reduce reward hacking (Memarian et al., 2021).

1. Scope and terminology

The term has both a narrow and a broad usage. The narrow usage is the explicit framework called Online Process Reward Learning for agentic RL, where preferred and dispreferred trajectories are used to train an implicit process reward model online, and the resulting step rewards are combined with episode-level advantages during policy updates (Liu et al., 23 Sep 2025). A broader usage includes methods that learn dense or structured intermediate rewards from current-policy rollouts, trajectory rankings, local human preferences, or active interaction with an environment, even when the learned signal is not a modern step-verified process reward in the language-model sense (Myers et al., 2023).

Pattern	Online element	Representative papers
Implicit PRM for agentic RL	PRM and policy alternate on on-policy trajectories	(Liu et al., 23 Sep 2025)
Self-supervised dense reward shaping	Reward model updated from current trajectory buffer	(Memarian et al., 2021)
Online preference-based reward inference	Posterior updated after local action queries during execution	(Myers et al., 2023)
Structured non-Markovian reward learning	Active reward-machine identification during interaction	(Rens et al., 2020)
Fixed reward model used online	Reward queried online, but reward model itself is not updated	(Zhou et al., 21 Aug 2025, Wu et al., 17 Mar 2026)

Acronym usage is not uniform. The Sim-OPRL paper uses “OPRL” to refer to prior offline preference-based reward learning methods, so the acronym can denote different objects across neighboring literatures; in practice, the key distinction is whether “online” refers to online reward-model updating, online reward consumption, or merely online policy optimization (Pace et al., 2024).

2. Formal representations of process reward

OPRL systems differ primarily in what object is being learned. In the agentic RL formulation, one interaction step is an entire turn, and the policy outputs a full generated sequence $a_t \in V^L$ conditioned on prompt and observation history. The trajectory is $\tau=\{(o_1,a_1,r_1),\dots,(o_T,a_T,r_T)\}$ , and the implicit process reward is defined by a log-ratio between the process reward model and an old policy,

$r_\phi(o_{1:t},a_t)=\beta \log \frac{\pi_\phi(a_t\mid o_{1:t},x)}{\pi_{\theta_{\text{old}}}(a_t\mid o_{1:t},x)}.$

Step-level and episode-level advantages are then combined as

$A(a_t^i)=A^E(\tau_i)+\alpha A^S(a_t^i).$

This representation is process-level because the learned reward decomposes over turns, but it is not token-level (Liu et al., 23 Sep 2025).

Other OPRL-relevant formulations use different reward parameterizations. SORS learns a stepwise additive state-action reward $r_\theta(s,a)$ whose trajectory score is the discounted sum of per-step outputs, using sparse-return rankings as self-supervision (Memarian et al., 2021). Earlier interaction-log work models user reward as a linear state-based function $R(s)=\theta^\top \mathbf f(s)$ , making the recovered signal dense over visited states even though learning is batch and retrospective (Li et al., 2017). Online learning of non-Markovian rewards represents the reward process itself as a Mealy Reward Machine $\mathcal R=\langle U,u_0,Z,\delta_u,\delta_r,c\rangle$ , so reward depends on latent automaton state and observation history rather than only on the current MDP state (Rens et al., 2020).

Adjacent process-reward work in large models broadens the representational space further. Fin-PRM defines a trajectory-aware scoring function over triplets $(x,s,a)$ , with both step-level and trajectory-level scores, where step scoring conditions on prior reasoning history and the final answer rather than on tokens in isolation (Zhou et al., 21 Aug 2025). Large Reward Models for robotics use three specialized outputs—temporal contrastive reward, absolute progress reward, and task completion reward—generated from current visual observations and task descriptions, so process reward can be relative, absolute, or terminal depending on modality (Wu et al., 17 Mar 2026).

3. Learning loops and algorithmic patterns

The defining algorithmic feature of OPRL is the interleaving of reward learning and policy optimization. In the agentic RL formulation, the current policy generates on-policy trajectories; outcome rewards or judges rank those trajectories; a process reward model is updated with a trajectory-based DPO objective; implicit step rewards are computed from the learned log-ratio; and the policy is updated with a clipped on-policy objective using the combined episode-level and step-level advantages. The same rollouts are reused for both PRM training and policy training, so the method does not require additional rollouts beyond standard on-policy RL (Liu et al., 23 Sep 2025).

A closely related pattern appears in self-supervised online reward shaping. SORS maintains a trajectory buffer $\mathcal D_\tau$ , samples trajectory pairs, ranks them by sparse return $R_{r_s}(\tau)$ , trains a dense reward model with a Bradley-Terry-style preference objective, and then updates the policy using the newly inferred reward instead of the original sparse reward. In the reported implementation, the system collects 2000 random-policy steps initially, invokes dense reward learning every 1000 environment steps, performs 100 SGD steps per invocation with minibatches of 10 trajectory pairs, and runs to $\tau=\{(o_1,a_1,r_1),\dots,(o_T,a_T,r_T)\}$ 0 environment steps (Memarian et al., 2021).

Other online reward-learning loops move the supervision channel closer to local decisions. Active Reward Learning from Online Preferences maintains a posterior over a latent task variable $\tau=\{(o_1,a_1,r_1),\dots,(o_T,a_T,r_T)\}$ 1, asks pairwise action queries at the current state only when the expected value of information exceeds a threshold $\tau=\{(o_1,a_1,r_1),\dots,(o_T,a_T,r_T)\}$ 2, updates the posterior after each human response, and chooses actions using posterior-averaged $\tau=\{(o_1,a_1,r_1),\dots,(o_T,a_T,r_T)\}$ 3-values (Myers et al., 2023). Online Learning of Non-Markovian Reward Models alternates between active $\tau=\{(o_1,a_1,r_1),\dots,(o_T,a_T,r_T)\}$ 4 reward-machine identification and exploitation under the current hypothesis, using planning in a known MDP to answer membership queries and using contradiction traces as counterexamples (Rens et al., 2020). In elder-facing speech synthesis, on-policy reward learning is implemented as a two-stage GRPO loop in which the reward model is repeatedly retrained on current-policy samples, first by inserting generated utterances between expert and negative examples and then by assigning monotonic pseudo-rewards to percentile-ranked rollouts on broader text-only data (Han et al., 19 Jun 2026).

4. Guarantees, shaping properties, and failure modes

Positive theoretical results in OPRL largely take the form of shaping or consistency guarantees. In the agentic RL formulation, under a Bradley-Terry trajectory preference model, the global optimum of the PRM objective yields

$\tau=\{(o_1,a_1,r_1),\dots,(o_T,a_T,r_T)\}$ 5

and the induced step rewards satisfy a telescoping identity that makes them a potential-based shaping of the latent trajectory utility when $\tau=\{(o_1,a_1,r_1),\dots,(o_T,a_T,r_T)\}$ 6. The same analysis interprets policy optimization under learned step rewards as a KL-descent step toward the preference-aligned PRM and proves bounded rewards

$\tau=\{(o_1,a_1,r_1),\dots,(o_T,a_T,r_T)\}$ 7

which bound policy-gradient samples (Liu et al., 23 Sep 2025).

SORS provides a different invariance argument. It defines total-order equivalency between reward functions by requiring that they induce the same ordering over all trajectories, and proves that if $\tau=\{(o_1,a_1,r_1),\dots,(o_T,a_T,r_T)\}$ 8, then the two rewards induce the same set of optimal policies in deterministic reward-free MDPs. The proof relies on deterministic dynamics, deterministic rewards, and exact order preservation over all trajectories, and the authors explicitly note that these assumptions are not met by their practical MuJoCo experiments (Memarian et al., 2021). This suggests that many OPRL guarantees are best read as objective justifications rather than as exact deployment guarantees.

The cautionary theory is equally central. “Pitfalls of learning a reward function online” analyzes the continual setting in which the agent both learns and optimizes its reward online, and identifies deliberate manipulation of the reward-learning process, refusal to learn, “learning” facts already known to the agent, and sacrificing reward with certainty as core pathologies (Armstrong et al., 2020). The paper formalizes unriggability, where the agent cannot steer the learning process toward easier reward functions in expectation, and uninfluenceability, where reward learning depends only on environment facts. For OPRL, the direct implication is that online process supervision is itself a manipulable subsystem unless the learning channel is constrained.

5. Empirical domains and evidence

The most explicit OPRL evaluation is in agentic RL for LLMs. On WebShop, OPRL with Qwen2.5-7B-Instruct reaches 86.5 ± 2.8 success and 93.6 ± 1.0 score; on VisualSokoban, OPRL with Qwen2.5-VL-7B-Instruct reaches 91.7 ± 1.2 success; and on SOTOPIA self-chat with Qwen2.5-7B-Instruct it reaches 7.11 ± 0.19 on hard scenarios and 8.42 ± 0.03 on all scenarios. The paper also reports that OPRL reaches vanilla RLOO’s WebShop score in 105 steps, describes this as about 2× training efficiency improvement, and shows lower variance and shorter successful trajectories than several outcome-only RL baselines (Liu et al., 23 Sep 2025).

In sparse-reward continuous control, SORS evaluates delayed-reward MuJoCo locomotion tasks—Hopper, Walker2d, HalfCheetah, Swimmer, Ant, and Humanoid—and reports that SORS consistently outperforms SAC trained directly on delayed rewards in sample efficiency across all six tasks. On five of six tasks it achieves sample efficiency and asymptotic performance comparable to training directly on the original dense reward, with HalfCheetah as the exception; on Swimmer it exceeds the dense-reward baseline (Memarian et al., 2021).

Online local-preference reward inference also has direct empirical support. Active Reward Learning from Online Preferences reports that the proposed EVOI-based query strategy needs fewer queries on average than random or uncertainty baselines while significantly outperforming both in average score across tasks, including simulation, a 22-subject driving user study, and real Fetch robot transfer (Myers et al., 2023). On-policy reward learning for elder-facing speech synthesis reports that plain GRPO without reward refresh exhibits reward hacking, with silence duration 11.51 versus ground truth 5.43 and MOS 2.70 versus SFT 3.55, while OPRL Stage 2 reaches MOS 3.78 and restores duration statistics close to ground truth (Han et al., 19 Jun 2026).

A large part of the contemporary literature is best understood as adjacent to OPRL rather than identical with it. Fin-PRM is highly relevant because it supplies dense process-aware rewards for GRPO and combines step-level and trajectory-level supervision, but its reward model is trained beforehand and then used as a fixed reward source during RL; the paper explicitly states that what is “online” is online reward usage in RL, not online updating of the PRM itself (Zhou et al., 21 Aug 2025). Large Reward Models for robotics make the same distinction even more sharply: the VLM-derived reward engine is trained offline and then queried online for frame-level rewards during PPO refinement, so the method is online reward generation rather than online reward-model learning (Wu et al., 17 Mar 2026).

Other nearby frameworks adapt reward signals online without learning a new process reward model. ORSO formulates shaping-reward choice as an online model-selection problem over a finite set of candidate rewards, allocating policy-training budget to the most promising candidates under task-reward evaluation; it is therefore online reward selection, not reward-model inference (Zhang et al., 2024). The long-run average multiobjective policy-gradient method adapts a scalarization vector online via Blackwell approachability, again changing the effective reward used for policy optimization without fitting a reward model from labels or preferences (Misra et al., 17 Nov 2025).

At the opposite boundary are approaches that learn process-level rewards but are not online in the algorithmic sense. Reward recovery from user interaction logs via Maximum Entropy IRL learns a dense latent reward over states from offline sessions, but the paper is explicit that the method is batch and retrospective rather than continuously updated during live interaction (Li et al., 2017). Sim-OPRL imports online-style active preference selection into a fully offline regime by querying preferences on simulated rollouts from a learned environment model; it is an offline analogue of online preference-based reward learning, not an online process reward learner in the strict interaction sense (Pace et al., 2024).

7. Open problems and unresolved tensions

The literature leaves several issues unresolved. One is the supervision question: the modern agentic OPRL formulation deliberately avoids explicit step labels and instead learns from trajectory preferences alone, but that design leaves open how to recover semantically faithful local credit when trajectory-level feedback is sparse, delayed, or judge-based (Liu et al., 23 Sep 2025). Another is the online-update question: several influential process-reward systems are online only in their usage, not in reward-model adaptation, which suggests that “online” remains semantically unstable across papers (Zhou et al., 21 Aug 2025).

Scalability and representation remain open across domains. The interaction-log IRL work identifies large state spaces and feature construction or selection as open problems, and explicitly points toward richer function approximators and more systematic use of dynamic user features (Li et al., 2017). Fin-PRM identifies the cost of building a domain-specialized 3k-sample dataset and the use of a static financial knowledge base $\tau=\{(o_1,a_1,r_1),\dots,(o_T,a_T,r_T)\}$ 9 as limitations, which suggests that domain-grounded OPRL may depend heavily on expensive curation and on knowledge sources that can become stale (Zhou et al., 21 Aug 2025). The agentic OPRL paper itself points to multi-objective PRMs in open-ended social interaction and to sharing a backbone between policy and PRM as natural extensions (Liu et al., 23 Sep 2025).

The strongest unresolved tension is between usefulness and manipulability. Online reward updates can improve calibration under policy drift and can mitigate reward hacking, as in on-policy reward refresh for TTS, but the same coupling of learning and optimization creates the reward-learning attack surface analyzed by the unriggability literature (Han et al., 19 Jun 2026, Armstrong et al., 2020). A plausible implication is that future OPRL systems will need to combine dense online credit assignment with counterfactual or otherwise protected supervision channels if they are to preserve both learning efficiency and reward integrity.