---
title: 'VSRM: Rule-based Stepwise Rewards'
url: https://www.emergentmind.com/topics/rule-based-verifiable-stepwise-reward-mechanism-vsrm
type: topic
---

# VSRM: Rule-based Stepwise Rewards

Searching arXiv for recent papers on VSRM and closely related verifiable stepwise reward mechanisms.
Rule-based Verifiable Stepwise Reward Mechanism (VSRM) denotes a class of reward constructions in which intermediate reasoning or action steps are scored by explicit, auditable verifiers rather than only by terminal outcomes or opaque learned judges. In the cited literature, the term is used explicitly for TableQA and efficient mathematical reasoning, and closely related formulations are presented for structured medical reasoning, code execution reasoning, multimodal visual reasoning, Text-to-SQL, GUI agents, search-augmented generation, and reward-stable RLVR optimization [2601.22530] [2508.10293] [2601.17223] [2603.11226] [2605.26520] [2505.12380] [2602.00575] [2602.18037]. Across these works, the common design principle is dense supervision from deterministic or tightly constrained checks over state transitions, intermediate artifacts, or structured traces, combined with a verifiable terminal signal.

## 1. Conceptual scope and problem setting

VSRM arises from a recurrent limitation of outcome-only reinforcement learning with verifiable rewards. In RLVR, final-answer checks such as exact match, unit tests, or task-completion predicates are straightforward to supervise, but they are sparse and delayed. Multiple papers therefore motivate stepwise reward as a way to improve credit assignment on long trajectories, reduce drift, and constrain reasoning to remain grounded in observable or executable state transitions [2508.10293] [2509.25598] [2601.17223].

The explicit term “Rule-based Verifiable Stepwise Reward Mechanism” is central in RE-Tab, where TableQA is modeled as a Partially Observable Markov Decision Process and each table transformation is evaluated by a verifiable state-based reward [2601.22530]. It is also central in “Promoting Efficient Reasoning with Verifiable Stepwise Reward,” where intermediate reasoning states are scored by truncating a chain of thought, sampling completions, and measuring whether each step increases the probability of reaching a correct final answer [2508.10293]. In “Beyond Outcome Verification,” the paper states that VSRM is the same construct as Verifiable Process Reward Models, namely deterministic, rule-based step supervision for structured reasoning [2601.17223].

Several other papers use VSRM as a mapping or extension rather than the core formalism. In Agentic Reward Modeling for GUI agents, the paper itself focuses on outcome-level rewards, and the stepwise VSRM formulation is presented as an extension consistent with the paper’s notation [2602.00575]. In REAL, the paper is centered on treating verifiable rewards as labels for classification; the stepwise REAL objective is explicitly presented as a principled extension rather than a reported experiment [2602.05630]. In search-augmented generation, multimodal CoT reward modeling, and writing, the VSRM interpretation similarly systematizes stepwise verification over rubrics, program traces, or rubricized pairwise critiques [2510.14660] [2504.06606] [2506.00103].

A central conceptual distinction in this literature is between rule-based verification and passive judgment. Rule-based scripts can be precise but brittle, while LLM-as-a-Judge methods can be flexible but opaque and susceptible to reward hacking or partial observability. VSRM occupies the middle ground only in some settings. In RE-Tab, ExecVerify, StepRTM, VPRMs, and medical composite rewards, the step signals are deterministic. In InterSketch and Search-Gen-V, deterministic checks are mixed with evaluator-based sufficiency or generative rubric judgments, so a fully deterministic reading is only partially applicable [2601.22530] [2603.11226] [2505.12380] [2601.17223] [2509.15557] [2605.26520] [2510.14660].

## 2. Formal foundations

A canonical formalization appears in RE-Tab, which models the task as a POMDP
$M = \langle S, A, T, R, O, \gamma \rangle$,
with latent environment state, partial observations, deterministic tool transitions, and a stepwise scalar reward $r_t$ that evaluates the quality of the newly obtained state [2601.22530]. The two-phase search objective separates state transition, “What is the best action?”, from simulative reasoning, “Am I sure about the output?”, and defines
$$
R(\mathcal{T}) = \sum_{l=0}^{S} \gamma^l \cdot r_l, \qquad
\mathcal{T}^* = \arg\max_{\mathcal{T} \in \mathcal{T}} R(\mathcal{T}).
$$
Its generic VSRM form is
$$
r_t = \alpha \cdot V_{\mathrm{trans}}(a_t, s_t, Q) + \beta \cdot V_{\mathrm{sim}}(s_t, Q, \hat{y}_t),
$$
where $V_{\mathrm{trans}}$ is a verifiable transition score and $V_{\mathrm{sim}}$ captures trajectory reliability [2601.22530].

A second general form appears in VPRMs for structured reasoning. There, each step has a deterministic verifier for step identifier correctness and step label correctness, producing
$$
r_t(Y; x) = w_t^{n} v_t^{n}(s_t, s_t^\star) + w_t^{l} v_t^{l}(\hat{\ell}_t, \ell_t^\star),
$$
with total reward
$$
R(\tau) = \sum_{t=1}^{T} \alpha_t r_t + \beta r_{\mathrm{outcome}}.
$$
This formulation emphasizes discrete rule paths, finite-state decision structure, and explicit coherence between step labels and final labels [2601.17223].

Efficient mathematical reasoning introduces a different, outcome-derived stepwise signal. Let $A_i$ denote the average correctness of completions after truncating at step $i$. The mechanism defines
$$
d_{i-1} = A_i - A_{i-1},
$$
then assigns positive reward to effective steps and penalties or discounted propagation to plateaus and regressions. This converts outcome verification into a stepwise signal without training a process reward model [2508.10293].

ExecVerify makes the step signal fully white-box by deriving labels from interpreter traces. For sampled white-box questions, intermediate control-flow and data-flow correctness are mixed with terminal I/O reward as
$$
R_{\mathrm{white-box}} = 2 \cdot \big((1-\alpha)\cdot R^{(I\to O)} + \alpha \cdot R_{\mathrm{white}}\big),
$$
where $R_{\mathrm{white}}$ is the average accuracy over verifiable trace questions [2603.11226].

InterSketch uses a trajectory-level composite reward with a stepwise improvement term. Its step score is converted into
$$
R_{\mathrm{step}}(\tau)=\frac{1}{T}\sum_{t=1}^{T}\frac{s_t-s_{t-1}}{s_t+s_{t-1}}, \qquad
R(\tau)=R_{\mathrm{fmt}}(\tau)+\alpha R_{\mathrm{acc}}(\tau)+\beta R_{\mathrm{step}}(\tau),
$$
with $\alpha = 0.7$ and $\beta = 0.3$ [2605.26520].

These formulations differ in surface form but share the same underlying contract: a step becomes rewardable when a verifier can map it to a bounded, auditable predicate or score over an intermediate state.

## 3. Verification substrates and rule design

The decisive variation among VSRMs lies in what is being verified. In TableQA, the verified object is the transformed table state. RE-Tab serializes the full post-action table with a structure-preserving encoding and evaluates it using TABROUGE,
$$
\mathrm{TABROUGE}(a_l, T_{l+1}, Q) = \frac{\mathrm{LCS}(Q,\mathrm{Enc}(T_{l+1}))}{|\mathrm{Enc}(T_{l+1})|},
$$
while also supporting rule-based validators for schema checks, row or column selection correctness, aggregation validity, sort consistency, computed columns, joins, and missing-value normalization [2601.22530].

In code execution reasoning, the verified object is an interpreter trace. ExecVerify derives exact next-statement labels, variable values, and variable types from $\tau=\{(l_t,\sigma_t)\}_{t=1}^{T}$, then checks predictions by exact statement match, semantic equality on canonicalized values, and exact type-name matching. The step signal is therefore grounded in concrete execution semantics rather than teacher-written explanations [2603.11226].

In Text-to-SQL, the verified object is a normalized SQL graph. Graph-Reward-SQL represents queries as canonical relational operator graphs and uses StepRTM for deterministic partial matching over Common Table Expression subqueries:
$$
R_{\mathrm{step}}^{(i)}(\hat{q}_{\mathrm{cte}}, q^\star)
=
\frac{|(M_i \cup \mathcal{G}_i)\cap V(G^\star)|}{|V(G^\star)|}.
$$
The paper is explicit that StepRTM is the core VSRM, while GMNScore is a learned, execution-free outcome reward that complements the stepwise mechanism rather than replacing it [2505.12380].

In multimodal reasoning, SVIP turns visual program execution into stepwise multimodal CoT supervision. Each code block becomes a CoT step and receives three binary labels: Relevance, Logic, and Attribute. Relevance is tied to compilation success and executability, Logic is checked by property tests over intermediate variables, and Attribute is checked by cross-validation of external visual-module outputs [2504.06606].

In structured medical reasoning, VPRMs use guideline-defined macros, step schemas, and deterministic parsers. The verifier checks step names, categorical labels, allowed transitions, and domain decision trees that map step labels to a final risk-of-bias classification. Because the rules derive from fixed RoB2 decision paths, the reward trace is explicitly inspectable [2601.17223].

In GUI agents, the evidence substrate is more complicated because the task is partially observable. The paper on Agentic Reward Modeling states that GUI tasks are “easy to verify but hard to solve,” and introduces a verifier agent with Check Screenshot, Execute Shell, Execute Python, and Computer Use. The paper itself defines outcome-level binary rewards, while the stepwise VSRM reading keeps rules as predicates over states and operations and uses those tools to gather evidence for each predicate online [2602.00575].

This diversity makes “rule-based” a family resemblance rather than a single implementation template. Some VSRMs verify symbolic transitions, some verify executable traces, some verify partial graph coverage, and some verify interactive state predicates.

## 4. Representative instantiations

The following table organizes prominent VSRM instantiations and closely related mappings across domains.

| Setting | Stepwise verifier | Characteristic signal |
|---|---|---|
| TableQA | TABROUGE + deterministic table validators | Query-grounded state transition reward |
| Mathematical reasoning | Truncated-rollout correctness differences | Effective vs ineffective step reward |
| Code execution reasoning | Interpreter-trace checks | Next statement, value, and type correctness |
| Text-to-SQL | RelPM over normalized SQL graphs | Incremental subquery coverage |
| Medical evidence synthesis | Guideline macros and rule-based parsers | Step identifier and label correctness |
| Multimodal CoT | Visual program execution tests | Relevance, Logic, Attribute labels |

RE-Tab is the cleanest explicit VSRM formulation because the reward is both state-based and stepwise. The mechanism is training-free, plug-and-play, and designed to guide both action selection and trajectory selection. Its validators are deterministic, and its principal state reward is computed from the full serialized table rather than a truncated embedding [2601.22530].

“Promoting Efficient Reasoning with Verifiable Stepwise Reward” addresses a different pathology: overthinking. Its key distinction is that step reward is not assigned by checking the literal content of a reasoning step, but by checking whether truncating at that step improves or degrades the probability of eventually reaching the correct answer. This makes the verifier outcome-based yet still stepwise [2508.10293].

ExecVerify is a white-box instantiation. It does not infer hidden process quality from text. Instead, it poses structured questions derived from an execution trace and scores the model on control flow and data flow. Because the intermediate labels come from the interpreter, the step supervision is semantically anchored [2603.11226].

Graph-Reward-SQL shows that VSRM can operate over structural decompositions rather than natural-language traces. StepRTM rewards CTE subqueries only when they contribute new matched nodes relative to a normalized gold graph, which makes repeated or redundant matches yield no extra credit [2505.12380].

VPRMs show the same principle in a domain with strongly codified reasoning. The rule system is not a unit test or exact-match checker but a domain decision macro over a finite set of categorical step labels. The paper states that VSRM is the same construct as VPRM in this setting [2601.17223].

A plausible implication is that VSRM is most straightforward when the environment supplies an executable or canonical intermediate representation: a table state, interpreter trace, SQL graph, visual program state, or guideline-defined label path. Where that representation is absent, later papers tend to either extend the method conceptually or mix deterministic checks with evaluator models.

## 5. Optimization, normalization, and stability

VSRM is not tied to a single optimizer. RE-Tab is training-free and uses reward-guided search rather than policy optimization, but its reward objective is already written in RL form [2601.22530]. Efficient reasoning integrates stepwise rewards with PPO and Reinforce++, using sparse nonzero rewards only at segmented step boundaries and the terminal position [2508.10293]. ExecVerify uses GRPO with $n=8$ rollouts per prompt and a mixed reward over white-box and terminal correctness [2603.11226]. VPRMs use GRPO or DAPO with group-normalized advantages over total stepwise-plus-outcome reward [2601.17223].

Several papers address the risk that dense process rewards may diverge from final correctness. Principle Process Reward introduces a principle-based reward model for non-verifiable intermediate search steps and uses Reward Normalization,
$$
r_{p,t} = \hat{r}_{p,t} + r_o - 1,
$$
so that when the final answer is wrong, all step rewards become non-positive, and when the final answer is correct, they become non-negative. This is explicitly designed to prevent “process-only score inflation” and to align local credit with global correctness [2509.25598].

REAL reframes RLVR itself as classification over categorical labels rather than scalar reward weighting. The reported paper is rollout-level, not stepwise, but it explicitly provides a stepwise REAL objective as a principled extension. Its key claim is that classification-style optimization yields monotonic and bounded gradient weighting, which mitigates Gradient Misassignment in Positives and Gradient Domination in Negatives [2602.05630].

Gradient Regularization addresses a different failure mode: reward hacking against proxy rewards, including rule-based rewards and LLM judges. The paper proposes optimizing
$$
\max_\phi \ \mathbb{E}[R(s,a)] - \gamma \|\nabla_\phi J(\phi,\theta)\|^2,
$$
and argues that flatter optima maintain higher proxy-reward accuracy. In rule-based math RLVR, it reports that GR avoids overly focusing on the format in rule-based rewards, and in LLM-as-a-Judge settings it prevents hacking the judge [2602.18037].

These results establish that VSRM is not merely a verifier design problem. It is also an optimization problem: dense stepwise signals improve credit assignment, but they must be normalized, bounded, or regularized if they are not to create new reward-hacking channels.

## 6. Empirical findings, limitations, and disputed boundaries

Across domains, VSRM-style mechanisms consistently improve either accuracy, efficiency, or both. RE-Tab reports up to 41.77% improvement in QA accuracy, almost 25% drop in inference cost, and 33.33% fewer test-time samples to reach a consistent answer [2601.22530]. Efficient mathematical reasoning reports substantial output length reduction while maintaining original reasoning performance across AIME24, AIME25, MATH-500, AMC23, Minerva, and OlympiadBench, and reduces detected overthinking from 279/500 to 132/500 for DeepScaleR and from 312/500 to 126/500 for DS-Distill-1.5B after VSRM-PPO [2508.10293]. VPRMs achieve up to 20% higher F1 than state-of-the-art models and 6.5% higher than verifiable outcome rewards, with Coherence of 89.5 and Coherent Accuracy of 75.0 on CochraneForest for Qwen2.5-7B-GRPO-VPRM [2601.17223]. ExecVerify reports that a 7B model reaches overall average 80.8 on code reasoning benchmarks and improves pass@1 by up to 5.9 on code generation tasks over strong post-training baselines [2603.11226].

Related mapped variants show similar effects in settings where the paper itself is not purely stepwise. Agentic Reward Modeling reports that the verifier succeeds more often than the actor on GUI tasks, with Actor SR 55.9% versus Verifier SR 83.1% on OSWorld-Verified and Actor SR 62.1% versus Verifier SR 93.1% on AndroidWorld, and that VAGEN achieves Acc 92.9%, F1 94.6% on class-balanced OSWorld-Verified and Acc 93.1%, F1 93.9% on AndroidWorld for Claude-Sonnet-4.5 [2602.00575]. Search-Gen-V reports rubric-level F1=0.70 and sample-level F1=0.73 on TREC RAG24-style long-form validation, and a hybrid verifier of EM plus Search-Gen-V-4B reaches P/R/F1≈0.95/0.93/0.94 on HotpotQA short-form evaluation [2510.14660]. InterSketch reports that adding stepwise reward improves TIR-Bench Maze from 80.8 to 85.0 over RL without stepwise reward and yields SOTA average 51.8 on TIR-Bench, surpassing Gemini-3-Pro at 49.0 [2605.26520].

The principal controversy is definitional. Not every cited paper uses “VSRM” as its native term, and not every stepwise reward in the set is fully deterministic. In Agentic Reward Modeling, the stepwise VSRM is explicitly an extension consistent with the paper’s notation rather than the reported training reward [2602.00575]. In REAL, the stepwise formulation is likewise a principled extension [2602.05630]. In InterSketch and Search-Gen-V, part of the step signal can come from evaluator models rather than solely deterministic rules [2605.26520] [2510.14660]. This suggests that the term has both a strict sense and a broader family sense. In the strict sense, VSRM is best exemplified by RE-Tab, StepRTM, ExecVerify, VPRMs, and the mathematical truncation-based mechanism, where step verification is deterministic and auditable [2601.22530] [2505.12380] [2603.11226] [2601.17223] [2508.10293].

A second limitation is engineering burden. Multiple papers emphasize that high-quality rule schemas, canonicalization pipelines, or domain DSLs are non-trivial to design. RE-Tab notes the importance of deterministic encoding and validator completeness; VPRMs require guideline macros and parsers; Graph-Reward-SQL depends on canonical SQL graph normalization; ExecVerify depends on instrumented traces and controlled synthesis; GUI stepwise verification remains difficult across heterogeneous applications [2601.22530] [2601.17223] [2505.12380] [2603.11226] [2602.00575].

A plausible implication is that VSRM scales best where intermediate states are naturally executable, serializable, or canonically matchable. Where intermediate states remain latent, partially observable, or semantically under-specified, the literature increasingly supplements rules with active probing, rubrics, or evaluator models rather than relying on purely static scripts.

Source: https://www.emergentmind.com/topics/rule-based-verifiable-stepwise-reward-mechanism-vsrm