VSRM: Rule-based Stepwise Rewards

• VSRM is a reward mechanism that uses deterministic verifiers to score intermediate reasoning steps, ensuring dense credit assignment based on observable state transitions.
• It applies canonical representations such as table states, interpreter traces, and SQL graphs to execute strict rule-based validations over structured tasks.
• Empirical studies show VSRM boosts accuracy and efficiency, reducing inference cost and mitigating overthinking in diverse applications like medical reasoning and code execution.

Searching arXiv for papers on VSRM and closely related verifiable stepwise reward mechanisms. Rule-based Verifiable Stepwise Reward Mechanism (VSRM) denotes a class of reward constructions in which intermediate reasoning or action steps are scored by explicit, auditable verifiers rather than only by terminal outcomes or opaque learned judges. In the cited literature, the term is used explicitly for TableQA and efficient mathematical reasoning, and closely related formulations are presented for structured medical reasoning, code execution reasoning, multimodal visual reasoning, Text-to-SQL, GUI agents, search-augmented generation, and reward-stable RLVR optimization (Kwok et al., 30 Jan 2026, Yue et al., 14 Aug 2025, Pronesti et al., 23 Jan 2026, Tang et al., 11 Mar 2026, Ning et al., 26 May 2026, Weng et al., 18 May 2025, Cui et al., 31 Jan 2026, Ackermann et al., 20 Feb 2026). Across these works, the common design principle is dense supervision from deterministic or tightly constrained checks over state transitions, intermediate artifacts, or structured traces, combined with a verifiable terminal signal.

1. Conceptual scope and problem setting

VSRM arises from a recurrent limitation of outcome-only reinforcement learning with verifiable rewards. In RLVR, final-answer checks such as exact match, unit tests, or task-completion predicates are straightforward to supervise, but they are sparse and delayed. Multiple papers therefore motivate stepwise reward as a way to improve credit assignment on long trajectories, reduce drift, and constrain reasoning to remain grounded in observable or executable state transitions (Yue et al., 14 Aug 2025, Xu et al., 29 Sep 2025, Pronesti et al., 23 Jan 2026).

The explicit term “Rule-based Verifiable Stepwise Reward Mechanism” is central in RE-Tab, where TableQA is modeled as a Partially Observable Markov Decision Process and each table transformation is evaluated by a verifiable state-based reward (Kwok et al., 30 Jan 2026). It is also central in “Promoting Efficient Reasoning with Verifiable Stepwise Reward,” where intermediate reasoning states are scored by truncating a chain of thought, sampling completions, and measuring whether each step increases the probability of reaching a correct final answer (Yue et al., 14 Aug 2025). In “Beyond Outcome Verification,” the paper states that VSRM is the same construct as Verifiable Process Reward Models, namely deterministic, rule-based step supervision for structured reasoning (Pronesti et al., 23 Jan 2026).

Several other papers use VSRM as a mapping or extension rather than the core formalism. In Agentic Reward Modeling for GUI agents, the paper itself focuses on outcome-level rewards, and the stepwise VSRM formulation is presented as an extension consistent with the paper’s notation (Cui et al., 31 Jan 2026). In REAL, the paper is centered on treating verifiable rewards as labels for classification; the stepwise REAL objective is explicitly presented as a principled extension rather than a reported experiment (Zhai et al., 5 Feb 2026). In search-augmented generation, multimodal CoT reward modeling, and writing, the VSRM interpretation similarly systematizes stepwise verification over rubrics, program traces, or rubricized pairwise critiques (Ma et al., 16 Oct 2025, Gao et al., 9 Apr 2025, Jia et al., 30 May 2025).

A central conceptual distinction in this literature is between rule-based verification and passive judgment. Rule-based scripts can be precise but brittle, while LLM-as-a-Judge methods can be flexible but opaque and susceptible to reward hacking or partial observability. VSRM occupies the middle ground only in some settings. In RE-Tab, ExecVerify, StepRTM, VPRMs, and medical composite rewards, the step signals are deterministic. In InterSketch and Search-Gen-V, deterministic checks are mixed with evaluator-based sufficiency or generative rubric judgments, so a fully deterministic reading is only partially applicable (Kwok et al., 30 Jan 2026, Tang et al., 11 Mar 2026, Weng et al., 18 May 2025, Pronesti et al., 23 Jan 2026, Tarek et al., 19 Sep 2025, Ning et al., 26 May 2026, Ma et al., 16 Oct 2025).

2. Formal foundations

A canonical formalization appears in RE-Tab, which models the task as a POMDP $M = \langle S, A, T, R, O, \gamma \rangle$, with latent environment state, partial observations, deterministic tool transitions, and a stepwise scalar reward $r_t$ that evaluates the quality of the newly obtained state (Kwok et al., 30 Jan 2026). The two-phase search objective separates state transition, “What is the best action?”, from simulative reasoning, “Am I sure about the output?”, and defines

$$R(\mathcal{T}) = \sum_{l=0}^{S} \gamma^l \cdot r_l, \qquad \mathcal{T}^* = \arg\max_{\mathcal{T} \in \mathcal{T}} R(\mathcal{T}).$$

Its generic VSRM form is

$$r_t = \alpha \cdot V_{\mathrm{trans}}(a_t, s_t, Q) + \beta \cdot V_{\mathrm{sim}}(s_t, Q, \hat{y}_t),$$

where $V_{\mathrm{trans}}$ is a verifiable transition score and $V_{\mathrm{sim}}$ captures trajectory reliability (Kwok et al., 30 Jan 2026).

A second general form appears in VPRMs for structured reasoning. There, each step has a deterministic verifier for step identifier correctness and step label correctness, producing

$$r_t(Y; x) = w_t^{n} v_t^{n}(s_t, s_t^\star) + w_t^{l} v_t^{l}(\hat{\ell}_t, \ell_t^\star),$$

with total reward

$$R(\tau) = \sum_{t=1}^{T} \alpha_t r_t + \beta r_{\mathrm{outcome}}.$$

This formulation emphasizes discrete rule paths, finite-state decision structure, and explicit coherence between step labels and final labels (Pronesti et al., 23 Jan 2026).

Efficient mathematical reasoning introduces a different, outcome-derived stepwise signal. Let $A_i$ denote the average correctness of completions after truncating at step $i$. The mechanism defines

$r_t$0

then assigns positive reward to effective steps and penalties or discounted propagation to plateaus and regressions. This converts outcome verification into a stepwise signal without training a process reward model (Yue et al., 14 Aug 2025).

ExecVerify makes the step signal fully white-box by deriving labels from interpreter traces. For sampled white-box questions, intermediate control-flow and data-flow correctness are mixed with terminal I/O reward as

$r_t$1

where $r_t$2 is the average accuracy over verifiable trace questions (Tang et al., 11 Mar 2026).

InterSketch uses a trajectory-level composite reward with a stepwise improvement term. Its step score is converted into

$r_t$3

with $r_t$4 and $r_t$5 (Ning et al., 26 May 2026).

These formulations differ in surface form but share the same underlying contract: a step becomes rewardable when a verifier can map it to a bounded, auditable predicate or score over an intermediate state.

3. Verification substrates and rule design

The decisive variation among VSRMs lies in what is being verified. In TableQA, the verified object is the transformed table state. RE-Tab serializes the full post-action table with a structure-preserving encoding and evaluates it using TABROUGE,

$r_t$6

while also supporting rule-based validators for schema checks, row or column selection correctness, aggregation validity, sort consistency, computed columns, joins, and missing-value normalization (Kwok et al., 30 Jan 2026).

In code execution reasoning, the verified object is an interpreter trace. ExecVerify derives exact next-statement labels, variable values, and variable types from $r_t$7, then checks predictions by exact statement match, semantic equality on canonicalized values, and exact type-name matching. The step signal is therefore grounded in concrete execution semantics rather than teacher-written explanations (Tang et al., 11 Mar 2026).

In Text-to-SQL, the verified object is a normalized SQL graph. Graph-Reward-SQL represents queries as canonical relational operator graphs and uses StepRTM for deterministic partial matching over Common Table Expression subqueries:

$r_t$8

The paper is explicit that StepRTM is the core VSRM, while GMNScore is a learned, execution-free outcome reward that complements the stepwise mechanism rather than replacing it (Weng et al., 18 May 2025).

In multimodal reasoning, SVIP turns visual program execution into stepwise multimodal CoT supervision. Each code block becomes a CoT step and receives three binary labels: Relevance, Logic, and Attribute. Relevance is tied to compilation success and executability, Logic is checked by property tests over intermediate variables, and Attribute is checked by cross-validation of external visual-module outputs (Gao et al., 9 Apr 2025).

In structured medical reasoning, VPRMs use guideline-defined macros, step schemas, and deterministic parsers. The verifier checks step names, categorical labels, allowed transitions, and domain decision trees that map step labels to a final risk-of-bias classification. Because the rules derive from fixed RoB2 decision paths, the reward trace is explicitly inspectable (Pronesti et al., 23 Jan 2026).

In GUI agents, the evidence substrate is more complicated because the task is partially observable. The paper on Agentic Reward Modeling states that GUI tasks are “easy to verify but hard to solve,” and introduces a verifier agent with Check Screenshot, Execute Shell, Execute Python, and Computer Use. The paper itself defines outcome-level binary rewards, while the stepwise VSRM reading keeps rules as predicates over states and operations and uses those tools to gather evidence for each predicate online (Cui et al., 31 Jan 2026).

This diversity makes “rule-based” a family resemblance rather than a single implementation template. Some VSRMs verify symbolic transitions, some verify executable traces, some verify partial graph coverage, and some verify interactive state predicates.

4. Representative instantiations

The following table organizes prominent VSRM instantiations and closely related mappings across domains.

Setting	Stepwise verifier	Characteristic signal
TableQA	TABROUGE + deterministic table validators	Query-grounded state transition reward
Mathematical reasoning	Truncated-rollout correctness differences	Effective vs ineffective step reward
Code execution reasoning	Interpreter-trace checks	Next statement, value, and type correctness
Text-to-SQL	RelPM over normalized SQL graphs	Incremental subquery coverage
Medical evidence synthesis	Guideline macros and rule-based parsers	Step identifier and label correctness
Multimodal CoT	Visual program execution tests	Relevance, Logic, Attribute labels

RE-Tab is the cleanest explicit VSRM formulation because the reward is both state-based and stepwise. The mechanism is training-free, plug-and-play, and designed to guide both action selection and trajectory selection. Its validators are deterministic, and its principal state reward is computed from the full serialized table rather than a truncated embedding (Kwok et al., 30 Jan 2026).

“Promoting Efficient Reasoning with Verifiable Stepwise Reward” addresses a different pathology: overthinking. Its key distinction is that step reward is not assigned by checking the literal content of a reasoning step, but by checking whether truncating at that step improves or degrades the probability of eventually reaching the correct answer. This makes the verifier outcome-based yet still stepwise (Yue et al., 14 Aug 2025).

ExecVerify is a white-box instantiation. It does not infer hidden process quality from text. Instead, it poses structured questions derived from an execution trace and scores the model on control flow and data flow. Because the intermediate labels come from the interpreter, the step supervision is semantically anchored (Tang et al., 11 Mar 2026).

Graph-Reward-SQL shows that VSRM can operate over structural decompositions rather than natural-language traces. StepRTM rewards CTE subqueries only when they contribute new matched nodes relative to a normalized gold graph, which makes repeated or redundant matches yield no extra credit (Weng et al., 18 May 2025).

VPRMs show the same principle in a domain with strongly codified reasoning. The rule system is not a unit test or exact-match checker but a domain decision macro over a finite set of categorical step labels. The paper states that VSRM is the same construct as VPRM in this setting (Pronesti et al., 23 Jan 2026).

A plausible implication is that VSRM is most straightforward when the environment supplies an executable or canonical intermediate representation: a table state, interpreter trace, SQL graph, visual program state, or guideline-defined label path. Where that representation is absent, later papers tend to either extend the method conceptually or mix deterministic checks with evaluator models.

5. Optimization, normalization, and stability

VSRM is not tied to a single optimizer. RE-Tab is training-free and uses reward-guided search rather than policy optimization, but its reward objective is already written in RL form (Kwok et al., 30 Jan 2026). Efficient reasoning integrates stepwise rewards with PPO and Reinforce++, using sparse nonzero rewards only at segmented step boundaries and the terminal position (Yue et al., 14 Aug 2025). ExecVerify uses GRPO with $r_t$9 rollouts per prompt and a mixed reward over white-box and terminal correctness (Tang et al., 11 Mar 2026). VPRMs use GRPO or DAPO with group-normalized advantages over total stepwise-plus-outcome reward (Pronesti et al., 23 Jan 2026).

Several papers address the risk that dense process rewards may diverge from final correctness. Principle Process Reward introduces a principle-based reward model for non-verifiable intermediate search steps and uses Reward Normalization,

$$R(\mathcal{T}) = \sum_{l=0}^{S} \gamma^l \cdot r_l, \qquad \mathcal{T}^* = \arg\max_{\mathcal{T} \in \mathcal{T}} R(\mathcal{T}).$$0

so that when the final answer is wrong, all step rewards become non-positive, and when the final answer is correct, they become non-negative. This is explicitly designed to prevent “process-only score inflation” and to align local credit with global correctness (Xu et al., 29 Sep 2025).

REAL reframes RLVR itself as classification over categorical labels rather than scalar reward weighting. The reported paper is rollout-level, not stepwise, but it explicitly provides a stepwise REAL objective as a principled extension. Its key claim is that classification-style optimization yields monotonic and bounded gradient weighting, which mitigates Gradient Misassignment in Positives and Gradient Domination in Negatives (Zhai et al., 5 Feb 2026).

Gradient Regularization addresses a different failure mode: reward hacking against proxy rewards, including rule-based rewards and LLM judges. The paper proposes optimizing

$$R(\mathcal{T}) = \sum_{l=0}^{S} \gamma^l \cdot r_l, \qquad \mathcal{T}^* = \arg\max_{\mathcal{T} \in \mathcal{T}} R(\mathcal{T}).$$1

and argues that flatter optima maintain higher proxy-reward accuracy. In rule-based math RLVR, it reports that GR avoids overly focusing on the format in rule-based rewards, and in LLM-as-a-Judge settings it prevents hacking the judge (Ackermann et al., 20 Feb 2026).

These results establish that VSRM is not merely a verifier design problem. It is also an optimization problem: dense stepwise signals improve credit assignment, but they must be normalized, bounded, or regularized if they are not to create new reward-hacking channels.

6. Empirical findings, limitations, and disputed boundaries

Across domains, VSRM-style mechanisms consistently improve either accuracy, efficiency, or both. RE-Tab reports up to 41.77% improvement in QA accuracy, almost 25% drop in inference cost, and 33.33% fewer test-time samples to reach a consistent answer (Kwok et al., 30 Jan 2026). Efficient mathematical reasoning reports substantial output length reduction while maintaining original reasoning performance across AIME24, AIME25, MATH-500, AMC23, Minerva, and OlympiadBench, and reduces detected overthinking from 279/500 to 132/500 for DeepScaleR and from 312/500 to 126/500 for DS-Distill-1.5B after VSRM-PPO (Yue et al., 14 Aug 2025). VPRMs achieve up to 20% higher F1 than state-of-the-art models and 6.5% higher than verifiable outcome rewards, with Coherence of 89.5 and Coherent Accuracy of 75.0 on CochraneForest for Qwen2.5-7B-GRPO-VPRM (Pronesti et al., 23 Jan 2026). ExecVerify reports that a 7B model reaches overall average 80.8 on code reasoning benchmarks and improves pass@1 by up to 5.9 on code generation tasks over strong post-training baselines (Tang et al., 11 Mar 2026).

Related mapped variants show similar effects in settings where the paper itself is not purely stepwise. Agentic Reward Modeling reports that the verifier succeeds more often than the actor on GUI tasks, with Actor SR 55.9% versus Verifier SR 83.1% on OSWorld-Verified and Actor SR 62.1% versus Verifier SR 93.1% on AndroidWorld, and that VAGEN achieves Acc 92.9%, F1 94.6% on class-balanced OSWorld-Verified and Acc 93.1%, F1 93.9% on AndroidWorld for Claude-Sonnet-4.5 (Cui et al., 31 Jan 2026). Search-Gen-V reports rubric-level F1=0.70 and sample-level F1=0.73 on TREC RAG24-style long-form validation, and a hybrid verifier of EM plus Search-Gen-V-4B reaches P/R/F1≈0.95/0.93/0.94 on HotpotQA short-form evaluation (Ma et al., 16 Oct 2025). InterSketch reports that adding stepwise reward improves TIR-Bench Maze from 80.8 to 85.0 over RL without stepwise reward and yields SOTA average 51.8 on TIR-Bench, surpassing Gemini-3-Pro at 49.0 (Ning et al., 26 May 2026).

The principal controversy is definitional. Not every cited paper uses “VSRM” as its native term, and not every stepwise reward in the set is fully deterministic. In Agentic Reward Modeling, the stepwise VSRM is explicitly an extension consistent with the paper’s notation rather than the reported training reward (Cui et al., 31 Jan 2026). In REAL, the stepwise formulation is likewise a principled extension (Zhai et al., 5 Feb 2026). In InterSketch and Search-Gen-V, part of the step signal can come from evaluator models rather than solely deterministic rules (Ning et al., 26 May 2026, Ma et al., 16 Oct 2025). This suggests that the term has both a strict sense and a broader family sense. In the strict sense, VSRM is best exemplified by RE-Tab, StepRTM, ExecVerify, VPRMs, and the mathematical truncation-based mechanism, where step verification is deterministic and auditable (Kwok et al., 30 Jan 2026, Weng et al., 18 May 2025, Tang et al., 11 Mar 2026, Pronesti et al., 23 Jan 2026, Yue et al., 14 Aug 2025).

A second limitation is engineering burden. Multiple papers emphasize that high-quality rule schemas, canonicalization pipelines, or domain DSLs are non-trivial to design. RE-Tab notes the importance of deterministic encoding and validator completeness; VPRMs require guideline macros and parsers; Graph-Reward-SQL depends on canonical SQL graph normalization; ExecVerify depends on instrumented traces and controlled synthesis; GUI stepwise verification remains difficult across heterogeneous applications (Kwok et al., 30 Jan 2026, Pronesti et al., 23 Jan 2026, Weng et al., 18 May 2025, Tang et al., 11 Mar 2026, Cui et al., 31 Jan 2026).

A plausible implication is that VSRM scales best where intermediate states are naturally executable, serializable, or canonically matchable. Where intermediate states remain latent, partially observable, or semantically under-specified, the literature increasingly supplements rules with active probing, rubrics, or evaluator models rather than relying on purely static scripts.