Process reward modeling for intermediate steps without direct supervision

Develop a principled process reward modeling scheme in reinforcement learning with verifiable rewards that assigns appropriate rewards to intermediate reasoning steps without direct step-level supervision and in a manner independent of the chosen step segmentation.

Background

Beyond final-answer verification, RLVR could benefit from process-level rewards that evaluate intermediate reasoning steps. However, when step-level ground-truth is unavailable, it is unclear how to design reward assignment policies that do not depend on arbitrary step segmentation.

The authors explicitly raise this as a related open question connected to the broader uncertainty about the necessity of CoT rationales, highlighting the need for methods to assign rewards to reasoning steps in the absence of direct supervision.

References

While CoT has proven useful in both reference-based~\citep{team2025kimi} and reference-free~\citep{zhang2024generative} settings, it remains an open question how necessary in-depth rationales are for assessing semantic equivalence between reference answers and model responses in the same language, particularly when focusing on the conclusive part of each response. This also raises a related question for process reward modeling~\citep{lightman2023let} in RLVR: how should rewards be assigned when there is no direct supervision for intermediate steps, regardless of the step segmentation method?

— Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains  (2503.23829 - Su et al., 31 Mar 2025) in Section 7 (Discussions and Conclusions)

Thus, how to derive dense supervision from solver feedback without verified answers remains an open question.

— Beyond Verified Answers: Solver-Informed Self-Distillation for Bootstrapping Operations Research Language Models  (2609.09957 - Zhu et al., 9 Sep 2026) in Section 2, Related Work, page 3

However, we empirically find that this signed variant underperforms the magnitude-based reward in eq:p2t_reward (see Table~\ref{tb:exp_ablation}(c)). One possible reason is that the positive and negative terms in eq:p2t_reward_signed are generally imbalanced within a step. We therefore adopt eq:p2t_reward by default, and leave a more effective use of the sign information to future work.

— Unlocking Token Rewards via Training-Free Reward Attribution  (2610.18704 - Wu et al., 13 Oct 2026) in Section 2, subsection “Process-to-Token Reward Attibution,” paragraph “Magnitude vs. signed influence”

Trajectory-level feedback also teaches some avoidance and discards most of what the verifier knows: which call, which rule, which predicate, and what change of state would have made the call permissible. How that information should enter the learning objective, and how the same scalar reward should drive online policy optimization against the live gate, are the questions this paper opens.

— Policy-Verified Self-Training: Learning Agent Compliance from Verifiable Policy Rewards  (2610.19372 - Yu, 14 Oct 2026) in Conclusion