PRIME: Implicit Rewards for LLM Reasoning

• PRIME is a reinforcement learning framework that uses implicit token-level rewards computed from autoregressive log-ratios between a learned model and a reference model.
• It enhances credit assignment and sample efficiency for multi-step reasoning tasks in domains such as mathematics and coding.
• PRIME removes the need for explicit human-labeled process rewards by training its reward model online using only trajectory-level outcome supervision.

Searching arXiv for the PRIME RL paper and closely related process-reward / RL-for-LLM work.

PRIME, short for Process Reinforcement through IMplicit rEwards, is a reinforcement learning framework for LLMs designed for multi-step reasoning tasks such as mathematics and coding. Its central idea is to convert outcome-only supervision into dense token-level process rewards by training an implicit reward model from policy rollouts and outcome labels, then reading tokenwise reward increments from the model’s autoregressive log-ratio against a reference model. In this formulation, PRIME targets the two main weaknesses of sparse outcome-reward RL for long-horizon reasoning—credit assignment and sample efficiency—while avoiding explicit human-labeled process reward modeling and the static-reward overoptimization that accompanies offline process reward models (Cui et al., 3 Feb 2025).

1. Problem formulation and conceptual basis

PRIME is formulated in the standard autoregressive RL setting for reasoning-capable LLMs. At time $t$, the state is the prompt with previously generated tokens $(\mathbf{x}, \mathbf{y}_{<t})$, the action is the next token $y_t$, and policy optimization follows the policy-gradient objective

$$\nabla_\theta J(\theta) = \mathbb{E}_{\mathbf{x} \sim \mathcal{D},\mathbf{y} \sim \pi_\theta}\left[\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(y_t|\mathbf{y}_{<t}) A_t\right],$$

with a generic Monte Carlo advantage

$A_t = \sum_{s=t}^T \gamma^{s-t} r(y_s) - b.$

The paper contrasts this with the usual outcome-reward regime, where $r(y_t)=0$ for $t<T$ and only the final token receives a meaningful reward, so the optimization degenerates toward a bandit-style signal on long reasoning trajectories (Cui et al., 3 Feb 2025).

The motivating claim is that dense process rewards are preferable for multi-step reasoning because they can improve credit assignment, increase sample efficiency, and discourage spurious reasoning trajectories that happen to end in a correct answer. The practical obstacle is that explicit process reward models ordinarily require dense process labels and are expensive to maintain online; under policy distribution shift, static reward models are also vulnerable to reward hacking. PRIME addresses this by keeping the reward model online and on-policy while training it only from the same trajectory-level outcome labels already used for RL.

2. Implicit process rewards

The defining construction in PRIME is the use of a causal LLM as an implicit reward model. At the sequence level, the reward model is represented as

$$r_\phi(\mathbf{y}) := \beta \log \frac{\pi_\phi(\mathbf{y})}{\pi_\text{ref}(\mathbf{y})},$$

where $\pi_\phi$ is the implicit PRM, $\pi_\text{ref}$ is a reference model, and $(\mathbf{x}, \mathbf{y}_{<t})$0 is a scaling coefficient. Because autoregressive probabilities factorize,

$(\mathbf{x}, \mathbf{y}_{<t})$1

the sequence reward decomposes into token-level terms,

$(\mathbf{x}, \mathbf{y}_{<t})$2

and therefore

$(\mathbf{x}, \mathbf{y}_{<t})$3

This decomposition is the technical core of PRIME: a model trained only from trajectory-level outcomes is repurposed at inference time as a token-level process reward model (Cui et al., 3 Feb 2025).

The paper also introduces a cumulative implicit value

$(\mathbf{x}, \mathbf{y}_{<t})$4

with token reward interpreted as the increment

$(\mathbf{x}, \mathbf{y}_{<t})$5

This makes the process reward a temporal-difference-like quantity over the implicit reward model’s autoregressive score.

Outcome supervision remains explicit at the trajectory level. For mathematics, the verifier reward is

$(\mathbf{x}, \mathbf{y}_{<t})$6

while for coding it is the fraction of passed test cases,

$(\mathbf{x}, \mathbf{y}_{<t})$7

PRIME therefore combines sparse verifier-level supervision with dense token-level implicit rewards rather than replacing one with the other.

3. Online implicit PRM training and policy optimization

The implicit PRM is updated online using rollout trajectories and outcome labels only. Its training objective is a binary cross-entropy loss on sequence-level scores: $(\mathbf{x}, \mathbf{y}_{<t})$8 No process annotations are used; the reward model is supervised only by full-trajectory success or failure (Cui et al., 3 Feb 2025).

The training loop initializes the policy, old policy, implicit PRM, and reference model from the same starting checkpoint, typically an SFT model and in some experiments even the base model. Each RL iteration samples a minibatch of prompts, generates $(\mathbf{x}, \mathbf{y}_{<t})$9 responses per prompt from the current policy, scores them with the outcome verifier, filters prompts by an accuracy-based criterion, computes token-level implicit rewards, updates the implicit PRM on filtered on-policy rollouts, constructs token-level advantages, and then updates the policy with PPO-style clipping. The paper states that this setup forgoes the dedicated reward model training phase required by existing approaches in the sense that PRIME does not require a separately human-labeled PRM pipeline; however, it still trains a reward-model-like object online.

A further implementation distinction is between single-forward and double-forward variants. In single-forward mode, policy updates use rewards from the current PRM before PRM recomputation; in double-forward mode, the PRM is updated first and rewards are recomputed afterward. The paper reports that double-forward improves PRM classification accuracy somewhat but yields similar training reward, and therefore recommends single-forward in practice because of its lower compute cost.

4. Advantage construction and algorithmic variants

The default PRIME estimator combines discounted implicit process rewards with leave-one-out outcome baselines. For $y_t$0 sampled responses to the same prompt, the paper’s main formula is

$y_t$1

The two components are computed separately because directly mixing their raw scales is described as numerically unstable (Cui et al., 3 Feb 2025).

PRIME is not restricted to RLOO. The paper gives compatible constructions for REINFORCE and GRPO. For REINFORCE,

$y_t$2

and for GRPO the outcome and process terms are separately normalized by sample mean and standard deviation. In this sense PRIME is not a standalone policy-gradient estimator but a dense reward construction and integration scheme that can be inserted into several MC-style RL baselines.

Policy optimization then uses a PPO-style clipped loss $y_t$3. The experiments set the KL coefficient to $y_t$4. The paper also compares two reference-model choices for the implicit reward model: a fixed SFT reference and a policy reference. Their performance is reported as similar, with the paper framing the choice as a tradeoff between an on-policy interpretation and compatibility with initial-policy-relative KL computation.

5. Empirical results and ablations

The main experimental configuration starts from Qwen2.5-Math-7B-Base, with Eurus-2-7B-SFT as the warmup SFT model trained on 230K open-source instruction-response pairs from math, coding, and biomedicine, with average response length about 1390.75 tokens. RL uses 150K queries $y_t$5 4 samples, compared with the reported 66K queries $y_t$6 32 samples for Qwen2.5-Math-7B-Instruct. The optimizer configuration includes policy learning rate $y_t$7, PRM learning rate $y_t$8, batch size $y_t$9, micro-batch size $$\nabla_\theta J(\theta) = \mathbb{E}_{\mathbf{x} \sim \mathcal{D},\mathbf{y} \sim \pi_\theta}\left[\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(y_t|\mathbf{y}_{<t}) A_t\right],$$0, rollout batch $$\nabla_\theta J(\theta) = \mathbb{E}_{\mathbf{x} \sim \mathcal{D},\mathbf{y} \sim \pi_\theta}\left[\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(y_t|\mathbf{y}_{<t}) A_t\right],$$1, $$\nabla_\theta J(\theta) = \mathbb{E}_{\mathbf{x} \sim \mathcal{D},\mathbf{y} \sim \pi_\theta}\left[\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(y_t|\mathbf{y}_{<t}) A_t\right],$$2, $$\nabla_\theta J(\theta) = \mathbb{E}_{\mathbf{x} \sim \mathcal{D},\mathbf{y} \sim \pi_\theta}\left[\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(y_t|\mathbf{y}_{<t}) A_t\right],$$3, and KL coefficient $$\nabla_\theta J(\theta) = \mathbb{E}_{\mathbf{x} \sim \mathcal{D},\mathbf{y} \sim \pi_\theta}\left[\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(y_t|\mathbf{y}_{<t}) A_t\right],$$4. Evaluation is reported on seven reasoning benchmarks: AIME 2024, AMC, MATH-500, Minerva Math, OlympiadBench, LeetCode, and LiveCodeBench v2 (Cui et al., 3 Feb 2025).

The headline result is that Eurus-2-7B-PRIME reaches an average score of 43.9 versus 28.8 for Eurus-2-7B-SFT, an average gain of 15.1 points. The benchmark values reported for the SFT model are 3.3, 30.1, 66.2, 32.7, 29.8, 21.7, and 17.8, while the best PRIME checkpoint at step 592 reports 26.7, 57.8, 79.2, 38.6, 42.1, 33.3, and 28.6, respectively. The paper also compares PRIME to Qwen2.5-Math-7B-Instruct, which has an average of 34.6. The visible table shows PRIME ahead on AIME 2024, AMC, MinervaMath, OlympiadBench, LeetCode, and LiveCodeBench, and slightly behind on MATH-500. The abstract states that PRIME surpasses Qwen2.5-Math-7B-Instruct on seven reasoning benchmarks, whereas the visible table supports superiority on six of seven; the most conservative summary is that the paper presents PRIME as matching or surpassing that model on most evaluated reasoning benchmarks while using far less training data.

Against outcome-only RL, the matched comparison at roughly 240 training steps reports an average score of 36.9 for RLOO with outcome verifier only versus 41.0 for PRIME, a gain of 4.1, described in the paper as about 11.1% relative. The paper also reports a $$\nabla_\theta J(\theta) = \mathbb{E}_{\mathbf{x} \sim \mathcal{D},\mathbf{y} \sim \pi_\theta}\left[\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(y_t|\mathbf{y}_{<t}) A_t\right],$$5 sample-efficiency gain and a 6.9% performance improvement in the dense-versus-sparse comparison. In per-algorithm ablations at 240 steps, PRIME improves RLOO from 36.9 to 41.0, REINFORCE from 36.0 to 37.8, GRPO from 36.1 to 37.8, and PPO from 35.8 to 39.4.

The ablation evidence is consistent with the paper’s central thesis. It reports that offline PRM accuracy drops during RL because of distribution shift, while online PRM accuracy improves, which is presented as evidence that online adaptation mitigates reward overoptimization. Initializing the implicit PRM from the same SFT model works better than initializing it from a separately trained explicit PRM, which the paper interprets as reduced policy–reward mismatch. The paper also states that using the implicit PRM as a reward model is substantially better than using it as a value model, and that online prompt filtering stabilizes training by selecting prompts of medium difficulty. Finally, “Zero” experiments from Qwen2.5-Math-7B-Base and Qwen2.5-32B-Base show that direct RL from base models is possible, though training is reported to saturate early.

6. Scope, limitations, and significance

PRIME is tailored to verifiable reasoning domains. Its supervision source is the outcome verifier, so the method depends on reliable trajectory-level correctness signals such as exact-match answers or executable test outcomes. The paper validates the method only on math and coding, and does not establish broad generalization to open-ended dialogue, creative writing, or other settings lacking trustworthy verifier labels (Cui et al., 3 Feb 2025).

Several limits follow directly from that design. First, token-level rewards are implicit log-ratio rewards, not human-interpretable annotations of whether an individual reasoning step is correct. Second, online updating mitigates but does not eliminate the possibility of reward hacking if the outcome labels themselves are noisy or gameable. Third, the paper’s strongest empirical support comes from competition-style reasoning benchmarks rather than broader language tasks. Fourth, in the “Zero” setting the method is reported to saturate early, possibly because of loss of response diversity.

Within its intended scope, PRIME’s significance is methodological rather than terminological. It supplies a concrete mechanism for turning outcome-only supervision into online-updatable dense process rewards: $$\nabla_\theta J(\theta) = \mathbb{E}_{\mathbf{x} \sim \mathcal{D},\mathbf{y} \sim \pi_\theta}\left[\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(y_t|\mathbf{y}_{<t}) A_t\right],$$6 and integrates those rewards into standard RL estimators without requiring process labels. A plausible implication is that PRIME occupies an intermediate position between sparse verifier RL and explicitly supervised process-reward modeling: it preserves verifier-only supervision while recovering much of the fine-grained optimization signal associated with process rewards. The paper’s empirical claim is that this yields better credit assignment, better sample efficiency, and stronger downstream reasoning performance in settings where final-answer verification is available.