The Weakest Link Tells It All: Outcome-Supervised Process Reward Modeling via Learnable Credit Assignment

Abstract: Process reward models (PRMs) enhance the reasoning capabilities of LLMs by providing fine-grained feedback, yet training PRMs typically requires expensive stepwise annotations. Outcome-supervised PRMs offer a scalable alternative by learning from final-answer correctness alone, but this introduces a fundamental credit assignment challenge, i.e., attributing outcomes to responsible reasoning steps. Existing approaches rely on either uniform or causal assignment, both of which fail to anchor credit in step correctness and thus hinder process error identification. In this work, we propose Outcome-Supervised Process Reward Modeling via Learnable Credit Assignment (LCA), an outcome-supervised PRM framework that jointly learns credit assignment and reward modeling under the principle of Weakest Link Assignment: a reasoning chain is as strong as its weakest link. To address mutual dependence between credit assignment and reward modeling, we formalize outcome-supervised PRM as a Multiple Instance Learning (MIL) problem and introduce Softmax-Weighted-Sum (SWS) pooling, an MIL pooling technique tailored for strong dependence and redundancy among reasoning states. We prove Bayes consistency of our algorithm under mild assumptions. Extensive experiments demonstrate that LCA consistently outperforms state-of-the-art outcome-supervised PRMs across multiple tasks and backbones. Code is available at https://anonymous.4open.science/r/LCA.

• The paper proposes a novel outcome-supervised PRM framework that leverages weakest link assignment via SWS pooling to precisely localize errors in reasoning steps.
• It formulates the task as a multiple instance learning problem and proves Bayes consistency under mild conditions, ensuring robust per-step error propagation.
• Empirical evaluations show significant F1 improvements over uniform and causal assignment methods, demonstrating superior performance on reasoning benchmarks.

Outcome-Supervised Process Reward Modeling with Learnable Credit Assignment

Introduction and Motivation

Process Reward Models (PRMs) for LLMs furnish dense, step-level feedback to improve control over reasoning-intensive tasks such as mathematical problem-solving and code generation. The central concern addressed in this work is the prohibitive annotation cost for step-level supervision required by traditional PRMs. Outcome-supervised PRMs, which only utilize easy-to-acquire final answer correctness as supervision, dramatically reduce this burden but pose a fundamental credit assignment problem: the necessity to attribute outcome-level supervision accurately to the causal steps in a reasoning trajectory.

The paper argues that both prevailing Uniform Assignment (additive pooling) and Causal Assignment (Monte Carlo rollout-based estimation) methods are misaligned with the core requirement of step-wise error localization, as neither grounds credit directly in logical step validity. The authors introduce an outcome-supervised PRM approach formalized as a Multiple Instance Learning (MIL) problem that leverages the "Weakest Link Assignment" principle, asserting that a chain of logical reasoning is only as sound as its most erroneous step. To instantiate this, they develop a Softmax-Weighted-Sum (SWS) pooling mechanism, theoretically prove its Bayes consistency under mild assumptions, and empirically validate its superiority over baselines.

Figure 1: Illustration of different credit assignment strategies.

Problem Formulation and Theoretical Framework

The formalization hinges on two key constructs:

1. Reasoning Trajectory as MIL Bag: Each reasoning trajectory $\tau$ consisting of a question $q$ and steps $s_1, \dots, s_T$ is framed as an MIL "bag", with each prefix $\tau_{:t}$ considered as an "instance". The final correctness label for a trajectory is therefore the max over the respective prefix correctness.
2. Weakest Link and Credit Assignment: The core credit paradigm assigns a positive label if and only if any prefix is erroneous, aligning precisely with Weakest Link semantics and standard MIL assumption.

Crucially, the authors identify that the nested, autoregressive, and monotonic structure of LLM-generated trajectories leads to strong inter-instance dependence and redundancy, undermining naive max-pooling, attention, or average pooling strategies typical of MIL.

SWS Pooling Mechanism

To resolve these issues, the proposed SWS pooling aggregates prefix-level predictions $p_t$ with weights

$$w_t = \frac{e^{p_t / \alpha}}{\sum_{j=1}^T e^{p_j / \alpha}}$$

and bag (trajectory)-level output

$\text{SWS}_\alpha(\mathbf{p}) = \sum_t w_t p_t$

where the temperature $\alpha$ controls the "smoothness": as $\alpha \to 0$, the pooling approaches max-pooling; as $\alpha \to \infty$, uniform averaging. SWS offers a strictly positive lower bound to each instance's influence, preventing collapse to terminal prefixes and ensuring robust, learnable per-step error signal propagation under outcome supervision.

Theoretical analysis culminates in a Bayes consistency result (for small enough $q$0, given bounded bag size and mild density assumptions), guaranteeing that, in the infinite data limit, the learned classifier converges to the true instance-level labels minimizing cross-entropy risk.

Figure 2: Overview of the PRM framework. The PRM outputs per-prefix error probabilities $q$1, aggregated with SWS pooling for end-to-end optimization.

Empirical Analysis

Benchmarks and Evaluation Protocol

Experiments are centered on mathematical reasoning, leveraging Math-Shepherd for training and ProcessBench for fine-grained error identification. Test-time scaling for solution re-ranking is evaluated on MATH-500 with best-of-N (BON) generation using diverse generator backbones.

Main Results

• On ProcessBench, the proposed method demonstrates robust error localization, outperforming baselines across all subsets and model scales—achieving +21.7% and +5.5% F1 improvement on 3B and 7B backbones, respectively, over previous SOTA.
• In best-of-N solution sampling, the method consistently outperforms Uniform and Causal Assignment PRMs, as well as DPO-style implicit methods, across all generation models and sample sizes. These gains are even more pronounced at higher N.

  Figure 3: Performance on ProcessBench splits as a function of SWS temperature $q$2. Performance tradeoffs are observed across datasets with different error characteristics.

Ablations show that $q$3 trades off between theoretical fidelity and empirical effectiveness: lower values favor max-assignment (beneficial for datasets with fewer errors), while higher values distribute credit more evenly, capturing multiple errors in complex trajectories.

Comparison with Pooling Methods

Empirical comparison against instance, embedding, attention, additive, and conjunctive pooling establishes SWS pooling ($q$4) as best, approaching the performance of step-supervised upper bounds, robust across diverse evaluation criteria.

Figure 4: Average F1 scores of various pooling techniques. SWS pooling achieves superior performance compared to other MIL strategies.

Qualitative Analysis and Failure Modes

Analysis of prefix prediction distributions and case studies underscore the capability to highlight first-error locations with high fidelity, even in trajectories with multiple or subtle errors.

Figure 5: Average predicted error probability versus ground-truth error rate along reasoning steps for incorrect trajectories.

Practical and Theoretical Implications

The proposed outcome-supervised PRM offers a scalable alternative to expensive manual process labeling, achieving high-fidelity error localization and robust error supervision in reasoning chains. By aligning the training objective with downstream evaluation semantics (first-error detection, min-form credit), it closes train-test mismatches endemic to prior approaches.

SWS pooling’s Bayes consistency guarantees and empirical resilience highlight a path to process supervision with only outcome-level task feedback—critical for large-scale LLM alignment where stepwise annotation is unfeasible.

Limitations and Future Work

Notwithstanding the strong performance, there remains a measurable gap versus PRMs trained with high-quality human-annotated step labels. Noisy outcome supervision can also miss "lucky" correct final answers arrived at through faulty reasoning, violating the underlying MIL assumption. Mitigation strategies (e.g., label-noise robust MIL, bag-level denoising) and synergy with RL reward models constitute open directions.

Adapting the framework for reward-policy co-evolution in RL, robust error identification in reflective/backtracking reasoning, and further reducing temperature tuning sensitivity remain promising lines for further exploration.

Figure 6: F1 score dynamics during max pooling training, evidencing instability prevented by SWS pooling.

Conclusion

This paper introduces a principled, outcome-supervised PRM framework deploying learnable credit assignment via SWS pooling, with both theoretical consistency and empirical state-of-the-art performance. It bridges the annotation gap for fine-grained process supervision, scalable process error identification, and effective utilization for reasoning-task LLMs, indicating a robust foundation for reward modeling in next-generation cognitive systems.

---

Key reference: "The Weakest Link Tells It All: Outcome-Supervised Process Reward Modeling via Learnable Credit Assignment" (2606.27739)