Structural Reward Model (SRM)

• SRM is a reward modeling approach that decomposes reward assignment over intermediate steps, subparts, or dimensions to provide dense, actionable supervision.
• It employs structured process supervision via value functions and multi-dimensional scoring in applications such as mathematical, multimodal reasoning and virtual agent training.
• Empirical findings show that SRMs enhance search and preference scoring performance, though they emphasize trajectory consistency over causal correctness.

Searching arXiv for the cited SRM-related papers to ground the article in current literature. arXiv search: (Ma et al., 2024) Structural Reward Model step-level mathematical reasoning Structural Reward Model (SRM) is not a universally standardized term across the arXiv literature. In recent reward-modeling work, it refers to designs that impose explicit structure on reward assignment rather than relying only on a single end-to-end scalar: step-level value or action scoring in mathematical reasoning, sub-question-level verifiable rewards in multimodal reasoning, step-wise multi-dimensional reward modeling for virtual agents, and modular dimension-specific auxiliary analysis for industrial reward modeling (Ma et al., 2024, Zhang et al., 7 Aug 2025, Miao et al., 24 Mar 2025, Liu et al., 29 Sep 2025). Taken together, these works suggest a common organizing idea: reward is decomposed over steps, subparts, dimensions, or auxiliary branches so that supervision is denser, more diagnosable, and more directly usable for search or policy optimization. A survey of reward models does not define SRM as an official taxonomy branch, but it places the closest ideas under process reward, multi-objective reward, and structured preference supervision (Zhong et al., 12 Apr 2025).

1. Conceptual scope and relation to reward modeling

Within LLM and agent alignment, SRM-like methods are motivated by the limits of outcome-only or scalar reward modeling. Outcome reward models evaluate whether a completion results in a correct answer; process reward models instead assign scores to intermediate steps, and multi-objective reward models allocate reward across several criteria. The survey literature treats these as major reward-modeling design axes, especially for reasoning, reranking, search, and inference-time selection (Zhong et al., 12 Apr 2025).

This framing matters because the recent SRM literature is heterogeneous. In mathematical reasoning, an SRM is formulated as a step-level reward or value model that scores intermediate states or next steps (Ma et al., 2024). In multimodal reasoning, the reward is structured over sub-questions and then aggregated (Zhang et al., 7 Aug 2025). In virtual-agent training, the reward is step-wise and explicitly multi-dimensional, with separate signals for Helpfulness, Odds of Success, Efficiency, Task Relevance, and Coherence (Miao et al., 24 Mar 2025). In industrial preference learning, SRM denotes a modular framework in which side-branch models generate dimension-specific auxiliary text that conditions the final Bradley-Terry reward decision (Liu et al., 29 Sep 2025).

A plausible implication is that “structural” in SRM does not denote one fixed architecture. It denotes a modeling stance in which reward is made legible and decomposable, whether temporally, compositionally, or diagnostically.

2. Formalizations: value functions, process supervision, and structured reward objectives

In the MCTS-boosted mathematical reasoning formulation, the reasoning process is an MDP,

$\text{MDP} = (S, A, P, R),$

with

$s_i = (T_k, E_k)_{k=0}^i,\qquad a_i = T_{i+1},$

where $s_i$ is the history of thoughts $T_k$ and equations $E_k$, and $a_i$ is the next natural-language thought. The transition $P(s_{i+1}\mid s_i,a_i)$ is induced by a world model that generates the next equation after the thought. The reward is terminal and binary: $$R(s_i, a_i, s_{i+1}) = \begin{cases} 1, & \text{final answer is correct} \ 0, & \text{final answer is incorrect.} \end{cases}$$ Within this formulation, an SRM is either a state value function $V(s)$ or an action value function $Q(s,a)$, used for step-level preference alignment during training or as a scoring function during greedy search or beam search at inference time (Ma et al., 2024).

The paper defines several concrete variants: $s_i = (T_k, E_k)_{k=0}^i,\qquad a_i = T_{i+1},$0

$s_i = (T_k, E_k)_{k=0}^i,\qquad a_i = T_{i+1},$1

$s_i = (T_k, E_k)_{k=0}^i,\qquad a_i = T_{i+1},$2

$s_i = (T_k, E_k)_{k=0}^i,\qquad a_i = T_{i+1},$3

It also uses the standard optimal-value relations

$s_i = (T_k, E_k)_{k=0}^i,\qquad a_i = T_{i+1},$4

together with limiting statements intended to justify SRM-guided greedy or beam search (Ma et al., 2024).

A complementary formalization comes from general reward-model analysis. There, an ideal reward function is written as

$s_i = (T_k, E_k)_{k=0}^i,\qquad a_i = T_{i+1},$5

with process supervision defined by

$s_i = (T_k, E_k)_{k=0}^i,\qquad a_i = T_{i+1},$6

A learned reward model $s_i = (T_k, E_k)_{k=0}^i,\qquad a_i = T_{i+1},$7 is trained on labeled examples $s_i = (T_k, E_k)_{k=0}^i,\qquad a_i = T_{i+1},$8, with $s_i = (T_k, E_k)_{k=0}^i,\qquad a_i = T_{i+1},$9, using binary cross-entropy: $s_i$0 This formal objective says the reward model should estimate correctness probability, but later perturbation results show that learned reward often tracks structural consistency more strongly than causal correctness (Xu et al., 20 Feb 2025).

3. MCTS-based SRMs in mathematical reasoning

The clearest step-level SRM pipeline is the AlphaZero-like MCTS setup for mathematical reasoning. The search starts from a root node $s_i$1 corresponding to the problem; at each node the agent samples candidate next thoughts, a world model executes those thoughts into math expressions and resulting states, and MCTS repeats Selection, Expansion, Rollout, and Back-propagation. The approximate UCT-style selection rule balances exploitation through $s_i$2 and exploration through a UCT bonus. The implementation uses Llama-3-8B-Instruct as both agent and world model, samples $s_i$3 candidate actions per expansion, runs 500 iterations per problem, uses depth limit 8, checks rollout correctness with an SFT model based on DeepSeek-Math-7B-Base, and retains only preference pairs with value difference greater than 0.7 (Ma et al., 2024).

After MCTS, step-level preference pairs are built by comparing node values within each tree, and SRMs are trained by contrastive learning. The four principal variants are summarized below.

Variant	Input structure	Reported pattern
FC-SRM	Thoughts and equations	Similar to MO-SRM in preference prediction accuracy and greedy-search performance
MO-SRM	Equations only	Can outperform FC-SRM on MATH in some setups
SSMO-SRM	Current math expression only	Worse than MO-SRM
NT-SRM	Next natural-language thought	Trains poorly and performs poorly in search

The central empirical finding is that removing natural-language thought descriptions barely hurts performance. FC-SRM and MO-SRM have very similar preference prediction accuracy and greedy-search performance; on MATH, MO-SRM can even outperform FC-SRM in some setups; and statistical tests show no significant difference between FC and MO across datasets and base models, with all reported $s_i$4 (Ma et al., 2024). The paper interprets this as evidence that SRMs are not primarily rewarding natural-language rationales.

The stronger signal appears to be logical coherence in mathematical language. MO-SRM outperforms SSMO-SRM, implying that the historical sequence of equations matters more than the current equation alone. Reported gains illustrate the effect. For DeepSeek-Math-7B-Base, MO-SRM gives $s_i$5 on GSM8K and $s_i$6 on MATH, whereas SSMO-SRM gives $s_i$7 on GSM8K and $s_i$8 on MATH. For Qwen2-7B, MO-SRM gives $s_i$9 on GSM8K and $T_k$0 on MATH, whereas SSMO-SRM gives $T_k$1 on GSM8K and $T_k$2 on MATH (Ma et al., 2024). By contrast, NT-SRM performs poorly, which the authors read as evidence that implicit logic in natural language is harder for LLM-based reward models to capture than the explicit structure exposed by mathematical expressions.

A further observation is that greedy search with SRMs tends to favor shorter reasoning paths, although the paper notes that this may be due to insufficient MCTS exploitation and needs more study (Ma et al., 2024).

4. Structural consistency versus causal validity

A major challenge for SRM-like reward modeling is that structural sensitivity need not imply causal grounding. The distinction is made explicit in the analysis of reward models as detectors of consistency rather than causality. The paper separates causal validity, where the answer and reasoning are correct because they solve the specific input question, from structural consistency, where the reasoning path is internally coherent, numerically aligned, and looks like a plausible solution trace even when the causal connection to the original question is weakened or removed (Xu et al., 20 Feb 2025).

To probe this distinction, the study applies controlled perturbations: question truncation, initial steps truncation, last steps truncation, all steps truncation, question shuffling, and numerical value modification. The empirical pattern is consistent across datasets and model families. Removing the question causes relatively small absolute reward errors; initial or last step truncation leads to larger errors; all-steps truncation is usually the most damaging; and question shuffling and numeric modification produce much larger changes than question removal (Xu et al., 20 Feb 2025). Spearman rank correlation is highest when only the question is removed, and much lower when reasoning is truncated or question-solution alignment is disrupted. In Best-of-$T_k$3 selection, performance stays close to normal under question truncation, degrades more under step truncation, and drops the most under question shuffling.

These results support a critical reading of SRM behavior. They indicate that present reward models may be better at ranking complete, coherent trajectories than at verifying whether a trajectory is causally correct with respect to the prompt. The paper therefore argues that current reward models are better understood as trajectory consistency detectors than as true validators of logical correctness, and calls for causality-aware reward models trained with counterfactual and causally informative supervision (Xu et al., 20 Feb 2025). For SRM, this is both an endorsement and a warning: structure is clearly being modeled, but structure alone should not be conflated with correctness.

5. Step-wise, sub-question-level, and multi-dimensional SRMs

Beyond mathematical reasoning, the SRM idea has been generalized by changing the unit of decomposition. In multimodal reasoning, StructVRM replaces a single binary reward with sub-question-level, verifiable scoring. A model-based verifier takes the predicted answer $T_k$4 and reference answer $T_k$5 and outputs

$T_k$6

where each $T_k$7 is a correctness score for the $T_k$8-th sub-question, typically binary $T_k$9 or $E_k$0, based on semantic or mathematical equivalence. The reward is then

$E_k$1

and is also described operationally as

$E_k$2

The verifier is trained on over 200,000 annotated examples, applies filtering and verification including format validity checks, completeness review, semantic consistency checks, and removal of low-quality samples, and achieves 96.83% agreement with expert judgments on a held-out human-annotated evaluation set (Zhang et al., 7 Aug 2025). The method is explicitly designed for partial credit, semantic equivalence, and mathematical equivalence, and Seed-StructVRM is reported to rank first on 6 of 12 public multimodal benchmarks. On STEM-Bench, the reported scores are 79.23 for Seed-StructVRM, 75.51 for the baseline, 77.85 for Gemini-2.5-pro, 61.53 for o3, 57.55 for Claude-Sonnet-4, and 45.94 for Qwen2.5-VL-72B; the ablation gives 79.23 for full Seed-StructVRM, 76.66 w/o StructVRM, and 75.47 w/o RL (Zhang et al., 7 Aug 2025).

In generalist virtual agents, the decomposition is not by sub-question but by action dimension. The SRM benchmark in this setting contains SRMTrain with 78k automatically annotated data points and SRMEval with 32k manually selected test points, for about 110k annotated points in total. The learned reward model, Similar, evaluates each agent step using five dimensions: Helpfulness $E_k$3, Odds of Success $E_k$4, Efficiency $E_k$5, Task Relevance $E_k$6, and Coherence $E_k$7. MCTS-P is used to collect and annotate data, with node value defined as

$E_k$8

and selection based on an Upper Confidence Bound-style rule (Miao et al., 24 Mar 2025). Similar is trained with the Triple-M strategy—Multi-step, Multi-objective, Multi-modal—using a regression stage for the five-dimensional reward vector and a prompt-aware gating stage trained with Bradley-Terry preference loss. On SRMEval, Similar-RL-Llama reaches 50.6 Avg versus 43.8 for Llama-3.2-Vision, and Similar-3M-Llama reaches 60.8 Avg. Human acceptance rates exceed 78.8% for every dimension, with H: 87.9%, OS: 78.8%, E: 82.6%, TR: 84.7%, and C: 93.5% (Miao et al., 24 Mar 2025).

These examples show that SRM can be instantiated by different decomposition operators: equation chains, sub-question score vectors, or multi-dimensional step assessments. The common objective is denser credit assignment than binary outcome supervision.

6. Modular industrial SRMs: side branches, interpretability, and efficiency

A later formulation makes the structure modular rather than temporal. Here SRM is a reward-modeling framework in which several side-branch models generate auxiliary textual features for a prompt-response pair, and those auxiliary texts are concatenated with the original input before a standard Bradley-Terry reward model makes the preference judgment (Liu et al., 29 Sep 2025). The five side branches are SB-Semantic, SB-Entity, SB-FactCheck, SB-Style, and SB-Quality, each fine-tuned from LLaMA3-8B with LoRA. The auxiliary data pipeline uses Best-of-$E_k$9 sampling to produce candidate analyses, an LLM-as-a-judge $a_i$0 to filter them using a threshold $a_i$1, and maximum-likelihood training for each side branch.

For chosen and rejected responses, the model constructs

$a_i$2

$a_i$3

then scores them with the final reward model,

$a_i$4

under Bradley-Terry preference loss. Because the side branches can run in parallel, the framework is intended to preserve much of the efficiency of scalar reward models while adding interpretable dimension-specific analysis (Liu et al., 29 Sep 2025).

The paper reports inference time per 1,000 samples as 18.7 sec for Scalar RM and 92.5 sec for GRM on public data, versus 22.8 sec for SRM; on industrial data the figures are 21.3 sec, 106.1 sec, and 25.4 sec, respectively (Liu et al., 29 Sep 2025). The authors summarize this as SRM achieving up to about 4× faster inference than GRM. Public evaluation uses RM-Bench, JudgeBench, and IFBench. Example gains include Llama3-8B Instruct improving from 11.3 to 60.8 overall, from 9.3 to 75.4 on RM-Bench normal, and from 2.6 to 59.4 on JudgeBench; InternLM2-7B-Reward improves from 52.0 to 63.1 overall; InternLM2-20B-Reward improves from 56.4 to 64.3 overall (Liu et al., 29 Sep 2025). On an industrial benchmark with 1.8 million preference-labeled training samples and 150,000 black-box industrial test samples, SRM improves over vanilla RM under DPO, PPO, and GRPO. Under DPO, for example, Accuracy changes from 78.1 to 81.6, Knowledge from 78.8 to 81.9, Hallucination from 14.5 to 8.6, Creativity from 75.2 to 79.5, and Complex reasoning from 61.2 to 67.5 (Liu et al., 29 Sep 2025).

The ablation study identifies Fact-Checking and Semantic Understanding as the largest contributors: removing Fact-Checking causes drops of $a_i$5 on RM-Bench, $a_i$6 on JudgeBench, and $a_i$7 on IFBench, while removing Semantic Understanding causes drops of $a_i$8, $a_i$9, and $P(s_{i+1}\mid s_i,a_i)$0 (Liu et al., 29 Sep 2025). The paper also notes limitations: side-branch design requires domain knowledge and tuning, training depends on high-quality auxiliary data, LLM-as-a-judge filtering may introduce bias, and concatenating many auxiliary texts can make sequences long and increase processing load.

7. Terminological ambiguity and unrelated uses of “SRM”

The acronym SRM has several unrelated meanings in adjacent literatures, and these should be distinguished from Structural Reward Model in the reward-modeling sense. In competitive programming, SRM means single round matches, as in TopCoder SRM; an Elo-based rating paper analyzes TopCoder SRM ratings and proposes an alternative rank-based Elo system (Batty et al., 2019). In learning theory and control, SRM means Structural Risk Minimization, as in model-class selection for nonlinear dynamics and for inverse reinforcement learning; those works minimize empirical risk plus a complexity penalty over model hierarchies rather than designing reward models for LLM alignment (Stamouli et al., 2023, Qu et al., 2023).

Other nearby uses are structurally related but conceptually distinct. In SpiNNaker 2 neuromorphic learning, a reward-based synaptic sampling model with structural plasticity uses latent synaptic parameters $P(s_{i+1}\mid s_i,a_i)$1, stochastic drift-diffusion updates, rewiring, and random reallocation; this is a reward-shaped stochastic structural plasticity system rather than a reward model over language or agent trajectories (Yan et al., 2019). In inverse reinforcement learning, SMIRL introduces structural motifs by first learning a finite-state automaton or reward machine and then performing Maximum Entropy IRL in the product space $P(s_{i+1}\mid s_i,a_i)$2; this is reward learning conditioned on inferred task structure, not SRM in the sense of reward-model architectures for LLMs or agents (Saqur, 2022).

This terminological spread indicates that “SRM” is best interpreted contextually. In present alignment and reasoning literature, the most coherent usage is the family of reward models that make reward assignment explicitly structured—over steps, subparts, dimensions, or modular auxiliary analyses—while remaining subject to the central open problem identified by recent work: structural consistency is useful for ranking and search, but it is not identical to causal correctness (Xu et al., 20 Feb 2025).