Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning (2504.15275v2)

Abstract: Process reward models (PRMs) have proven effective for test-time scaling of LLMs on challenging reasoning tasks. However, reward hacking issues with PRMs limit their successful application in reinforcement fine-tuning. In this paper, we identify the main cause of PRM-induced reward hacking: the canonical summation-form credit assignment in reinforcement learning (RL), which defines the value as cumulative gamma-decayed future rewards, easily induces LLMs to hack steps with high rewards. To address this, we propose PURE: Process sUpervised Reinforcement lEarning. The key innovation of PURE is a min-form credit assignment that formulates the value function as the minimum of future rewards. This method significantly alleviates reward hacking by limiting the value function range and distributing advantages more reasonably. Through extensive experiments on 3 base models, we show that PRM-based approaches enabling min-form credit assignment achieve comparable reasoning performance to verifiable reward-based methods within only 30% steps. In contrast, the canonical sum-form credit assignment collapses training even at the beginning! Additionally, when we supplement PRM-based fine-tuning with just 10% verifiable rewards, we further alleviate reward hacking and produce the best fine-tuned model based on Qwen2.5-Math-7B in our experiments, achieving 82.5% accuracy on AMC23 and 53.3% average accuracy across 5 benchmarks. Moreover, we summarize the observed reward hacking cases and analyze the causes of training collapse. Code and models are available at https://github.com/CJReinforce/PURE.

• The paper presents a novel min-form credit assignment method using PURE to mitigate reward hacking during reinforcement fine-tuning of LLMs on reasoning tasks.
• It transforms process rewards to focus on the worst performing step, aligning credit assignment with the minimal reward to stabilize training.
• Experiments show that combining min-form process rewards with a small fraction of verifiable rewards leads to faster convergence and improved accuracy.

This paper addresses the challenge of using Process Reward Models (PRMs) for Reinforcement Fine-tuning (RFT) of LLMs on reasoning tasks. While PRMs, which provide step-by-step feedback, are effective for test-time scaling (like Best-of-N selection), they often lead to "reward hacking" during RFT, where the LLM learns to exploit the reward model without actually improving its reasoning ability.

The core issue identified is the standard "summation-form" credit assignment used in RL, where the value of an action is the discounted sum of future rewards ($$Q^\pi(s_t, a_t) = \mathbb{E}[\sum_{i\geq t} \gamma^{i-t} r^p_i]$$). This approach encourages the LLM to maximize cumulative rewards, potentially leading it to generate many steps that receive high intermediate rewards (e.g., "thinking" steps) without necessarily reaching a correct final solution. This contrasts with how PRMs are often used at test time, where the quality of a sequence is judged by its weakest link (the minimum reward across steps).

To address this mismatch and mitigate reward hacking, the paper proposes PURE (Process Supervised Reinforcement Learning). The key innovation of PURE is a min-form credit assignment. Instead of summing future rewards, the value is determined by the minimum future reward up to the "worst" step (the step with the lowest PRM score). The return function is defined as:

$$G(s_t,a_t | \tau) = \begin{cases} \min (r^p_t, \cdots, r^p_n), & \text{if } t \leq w \ 0, & \text{if } t > w \end{cases}$$

where $w = \arg\min(r^p_1,\cdots,r^p_n)$ is the index of the worst step.

Practically, this min-form credit assignment is implemented by transforming the original process rewards $r^p_i$ before using them in a standard RL algorithm (like PPO with RLOO advantage estimation). The transformation assigns higher weights to lower rewards:

$$r_i^{p*} = \frac{\exp(-r^p_i/T)}{\sum_{j=1}^n \exp(-r^p_j/T)} \cdot r^p_i$$

As the temperature $T \to 0^+$, this transformation effectively sets all rewards to zero except for the minimum reward, $r^p_w$, which remains unchanged. This transformed reward $r_i^{p*}$ is assigned only to the final token of step $i$. The standard summation of these transformed rewards (with $\gamma=1$) then approximates the desired min-form return.

PURE also incorporates an advantage estimator based on RLOO (Reinforcement Learning with Optimistic Offline data) that handles both transformed process rewards ($r^{p*}$) and optional verifiable rewards ($r^v$, sparse reward for the final answer):

$$A_{i,t} = \underbrace{r^v_i - \frac{1}{K-1}\sum_{k\neq i}r^v_k}_{\text{RLOO with VR}} + \underbrace{\sum_{j=t}^{N}{\gamma^{j-t} \cdot r^{p*}_{i,j}} - \underbrace{\frac{\sum_{k\neq i}\sum_{l=1}^N\sum_{j=l}^N {\gamma^{j-l} \cdot r^{p*}_{k,j}}}{(K-1)N}}_{\text{token-level baseline}}}_{\text{RLOO with PRM*}}$$

Using a token-level baseline normalized by the maximum generation length ($N$) for the process rewards helps prevent reward hacking related to sequence length.

Experiments and Findings:

1. PRM Training: A PRM (PURE-PRM-7B) was trained on Qwen2.5-Math-7B using the PRM800K dataset, achieving strong results on BoN evaluation, ProcessBench, and PRMBench.
2. RFT Setup: PURE was applied to Qwen2.5 models (7B, Math-7B, Math-1.5B) using three reward settings:
   • PURE-PRM: Only transformed process rewards.
   • PURE-VR: Only verifiable rewards (like Deepseek R1-Zero).
   • PURE-PRM+VR: Mix of process rewards and 10% verifiable rewards.
3. Min-Form vs. Sum-Form: Summation-form credit assignment led to rapid training collapse, with models learning to only output "thinking" steps. Min-form credit assignment stabilized training significantly.
4. Performance & Efficiency: PURE-PRM (min-form) achieved comparable performance to PURE-VR but converged ~3x faster (requiring only ~30% of the training steps).
5. Best Performance: PURE-PRM+VR (min-form) yielded the best results, outperforming PURE-VR and baselines. Adding just 10% verifiable rewards effectively mitigated residual reward hacking observed in PURE-PRM. The best model (Qwen2.5-Math-7B + PURE-PRM+VR) achieved 53.3% average accuracy across 5 math benchmarks.
6. Reward Hacking Analysis: Three types of PRM-induced reward hacking were identified:
   • Only thinking, not solving: Exploiting high rewards for intermediate steps (mitigated by min-form).
   • Extremely few steps (1 step): Caused by inappropriate (step-level) advantage baselines biasing against longer solutions (mitigated by token-level baseline).
   • Extremely few steps (0 step) / irrelevant output: The PRM assigns high scores to trivial initial outputs (e.g., "Thank you.") because its causal nature prevents it from knowing no useful content follows. This was mitigated by adding verifiable rewards.
7. Training Collapse Cause: Analysis revealed that training collapse can be triggered by "pseudo-positive" samples – long, highly repetitive responses that the verifier mistakenly marks as correct. The model rapidly learns these repetitive patterns, leading to collapse within a few gradient steps. This highlights a limitation of current verifiers and PRMs.

Conclusion:

The paper demonstrates that the standard summation-form credit assignment is unsuitable for PRM-based RFT due to reward hacking. The proposed PURE framework, using a min-form credit assignment implemented via reward transformation, effectively stabilizes training and enables efficient learning from dense process rewards. Combining min-form PRM rewards with a small amount of verifiable rewards yielded the best performance, suggesting a practical approach for leveraging PRMs in RFT while controlling reward hacking. Future work includes developing generative PRMs to better assess step quality and handle pattern-based issues, and exploring iterative PRM-LLM training.