---
title: 'Math-Shepherd: Process-Oriented Reward Modeling'
url: https://www.emergentmind.com/topics/math-shepherd
type: topic
---

# Math-Shepherd: Process-Oriented Reward Modeling

Math-Shepherd is an advanced process-oriented reward modeling system designed to verify and reinforce mathematical reasoning in large language models (LLMs) without requiring manual human annotation [2312.08935]. Its architecture centers on assigning reward scores at the level of reasoning steps within multi-stage math solutions, enabling precise feedback and step-by-step guidance during both inference (verification) and training (reinforcement learning). Math-Shepherd thus represents a significant evolution in the development of autonomously supervised mathematical reasoning agents.

## 1. Process-Oriented Reward Model Architecture

Traditional outcome reward models (ORM) in LLM fine-tuning yield a scalar score only after an entire solution is produced. Math-Shepherd introduces a process reward model (PRM) that evaluates and guides the generator at each intermediate reasoning step. For a solution decomposed into steps $\{s_1, s_2, \dots, s_K\}$, the model $r_{s_i}$ outputs a sigmoid-based “goodness” score for each $s_i$.

Training data labels for each step are constructed automatically, leveraging continuation sampling rather than human annotation. Given $N$ full continuations (completions) generated from a step $s_i$ using a completer model, the scoring functions are:

- Hard Estimation (HE): $y^{\text{(HE)}}_{s_i} = 1$ if any sampled continuation reaches the correct final answer, $0$ otherwise.
- Soft Estimation (SE): $y^{\text{(SE)}}_{s_i} = \frac{1}{N}\sum_{j=1}^N \mathbb{I}(a_j = a^*)$.

The PRM is trained using cross-entropy loss:
$$
\mathcal{L}_{\text{PRM}} = \sum_{i=1}^K \Big[ y_{s_i} \log r_{s_i} + (1 - y_{s_i}) \log (1 - r_{s_i}) \Big],
$$
where $y_{s_i}$ is obtained by either HE or SE and $r_{s_i}$ is the predicted reward for $s_i$.

## 2. Verification as Stepwise Reranking

Math-Shepherd operates as a verifier by reranking candidate solutions from an LLM generator. Given a math problem $p$ and $N$ candidate solutions $\{S_1,\ldots,S_N\}$, Math-Shepherd computes the solution score (for solution $S_i$) as $\min_i r_{s_i}$, where $r_{s_i}$ is the per-step reward across all steps in $S_i$.

To increase reliability, the verifier can be composed with self-consistency voting:

$$
a_{\text{(sc+rm)}} = \arg\max_a \sum_{i=1}^N \mathbb{I}(a_i = a) \cdot RM(p, S_i),
$$
where $RM(p, S_i) = \min_i r_{s_i}$ and $a_i$ is the final answer of $S_i$. This compositional strategy ensures that the selected solution exhibits high intermediate step quality and majority final answer consensus.

## 3. Stepwise Reinforcement Learning via PPO

For reinforcement learning, Math-Shepherd supplies the stepwise reward in the context of Proximal Policy Optimization (PPO). Rather than a single episode-level reward, Math-Shepherd delivers $r_{s_i}$ immediately after each step $s_i$ in the generated solution. The PPO update then uses these dense rewards:

- Policy loss is calculated to maximize expected reward at each reasoning transition.
- The fine-tuned generator incrementally corrects and optimizes the chain of reasoning, improving stepwise logic and solution reliability.

Empirically, PPO with Math-Shepherd produces significant performance boosts: for example, Mistral-7B accuracy on GSM8K (a mathematical word problem dataset) moves from 77.9% to 84.1%; on MATH (complex math problems), accuracy advances from 28.6% to 33.0%. Verification with Math-Shepherd further amplifies performance, to 89.1% (GSM8K) and 43.5% (MATH), respectively.

## 4. Automatic Process Supervision via Completion Sampling

Eliminating the need for human labelers, Math-Shepherd exploits the generative capacity of LLMs to build its training set. By deploying a completer LLM to sample full solutions from any intermediate step $s_i$ (for N samples per step), it automatically infers ground-truth labels for step validity. This method leverages the notion that the local correctness of $s_i$ can be extrapolated from the solution trajectories it enables.

This “process-wise supervision” methodology scales seamlessly, enabling PRMs to be trained on large datasets for a variety of mathematical domains—without the prohibitive cost, latency, or inconsistency associated with manual stepwise annotation.

## 5. Quantitative Results and Comparative Evaluation

Math-Shepherd’s efficacy was validated on open-source models such as DeepSeek-67B, LLaMA2-70B, and LLemma-34B, evaluated with GSM8K and MATH. Table 1 presents representative results:

| Model                 | GSM8K (Verification) | MATH (Verification) |
|-----------------------|---------------------|---------------------|
| DeepSeek-67B (256 candidates) | 93.3% | 48.1% |
| Mistral-7B (stepwise PPO)     | 84.1% | 33.0% |
| Mistral-7B (w/ verification)  | 89.1% | 43.5% |

Math-Shepherd consistently outperforms verification baselines using either self-consistency voting or ORM-based reranking. The system demonstrates robustness to hallucinated steps, error propagation, and can reliably filter high-quality chains from diverse candidate pools.

## 6. Implications and Future Directions

Math-Shepherd offers a scalable, automatic process supervision pipeline that is adaptable to a wide array of mathematical reasoning regimes in LLMs. The ability to train and supervise without manual annotation removes a fundamental bottleneck, particularly for stepwise mathematical reasoning tasks where human annotation is otherwise impractical.

Looking forward, Math-Shepherd’s paradigm can be generalized to other domains involving multi-step reasoning, multi-modal math problems (see [2502.20808] for multi-visual scenarios), and further improved using richer completion models or more granular stepwise reward estimation. Its step-level feedback and reinforcement learning integration provide an extensible foundation for self-improving, interpretable mathematical AI systems.

## 7. Key Mathematical Formulas

Representative training losses:

- Outcome Reward Model (ORM): $\mathcal{L}_{\text{ORM}} = y_s \log r_s + (1 - y_s)\log(1 - r_s)$
- Process Reward Model (PRM): $\mathcal{L}_{\text{PRM}} = \sum_{i=1}^K [ y_{s_i}\log r_{s_i} + (1 - y_{s_i})\log(1 - r_{s_i}) ]$

Verifier selection score:

$$
RM(p, S_i) = \min_{s_j \in S_i} r_{s_j}
$$

Self-consistency composition with process verification:

$$
a_{\text{(sc+rm)}} = \arg\max_a \sum_{i=1}^N \mathbb{I}(a_i = a) \cdot RM(p, S_i)
$$

## 8. Summary

Math-Shepherd builds a process-level evaluation and training pipeline for mathematical reasoning in LLMs, with automatic, scalable construction of supervision, effective inference-time verification, and reinforcement learning using granular reward feedback. Its integration dramatically boosts solution accuracy and reliability in widely used LLMs, suggesting that process-oriented reward modeling is a critical advance for building robust and interpretable mathematical AI agents.

Source: https://www.emergentmind.com/topics/math-shepherd