Sub-goal Verifiable Reward (SGVR)

• SGVR is a reinforcement learning framework that decomposes global rewards into algorithmically verifiable subgoals, enabling precise credit assignment.
• It employs deterministic scoring functions and dense feedback mechanisms—such as rule-based and transformer verifiers—to improve structured reasoning in tasks like long-context QA and geometric proofs.
• Empirical results show that SGVR enhances model accuracy and robustness, with notable gains in metrics such as Qwen2.5 performance and Skeleton Rate in geometric reasoning.

Sub-goal Verifiable Reward (SGVR) represents a class of reinforcement learning reward schemes in which a global outcome signal is replaced or augmented with a sequence of fine-grained, verifiable milestones. These subgoal-level rewards are algorithmically checkable (i.e., “verifiable”), modular, and provide dense feedback, enabling stable optimization and effective credit assignment in structured reasoning tasks, especially for large language and vision-LLMs. SGVR methods have shown substantial empirical gains in complex, multi-step domains such as long-context retrieval, multimodal science QA, symbolic geometric proof, and expert process supervision.

1. Formal Definitions and Mathematical Principles

SGVR decomposes a reasoning trajectory into atomic subgoals, each admitting a deterministic scoring function. Let a model’s trajectory be $Y = (o_1, o_2, \ldots, o_T)$, where $o_t$ denotes the output at subgoal $t$.

Let $r_t(Y;x)$ denote the reward for subgoal $t$ (possibly further decomposed into identifier and label correctness, as in (Pronesti et al., 23 Jan 2026)), and let $r_\mathrm{label}(Y;x)$ be the terminal outcome reward. The aggregate SGVR is:

$$R(Y;x) = \sum_{t=1}^T r_t(Y;x) + r_\mathrm{label}(Y;x)$$

In domains where subgoals correspond to filling $k$ answer slots (e.g., multimodal QA), let $\hat{y} = (\hat{y}_1, \ldots, \hat{y}_k)$ be the structured answer, $y = (y_1, \ldots, y_k)$ the reference, and $o_t$0 the subgoal correctness:

$o_t$1

For long-context RLVR, SGVR (as in LongRLVR) introduces a verifiable context reward $o_t$2, e.g., via the $o_t$3-score between model-selected chunks $o_t$4 and ground-truth evidence $o_t$5 (Chen et al., 2 Mar 2026):

$o_t$6

with

$o_t$7

In geometric proof domains, the Skeleton Rate (SR) is employed:

$o_t$8

which is used as the per-instance reward (Chen et al., 8 Jan 2026).

SGVR rewards are typically integrated into a policy-gradient RL pipeline—most commonly via PPO or GRPO (Pronesti et al., 23 Jan 2026, Chen et al., 2 Mar 2026, Chen et al., 8 Jan 2026, Zhang et al., 7 Aug 2025).

2. Algorithmic Implementation and Optimizer Integration

SGVR algorithms modify standard policy-gradient updates by (1) substituting a dense, per-subgoal or per-chunk reward in place of single binary terminal reward, and (2) deploying deterministic, rule-based verifiers or model-based critics to check each subgoal.

A prototypical SGVR reinforcement loop is:

1. Sample a trajectory $o_t$9 using the current policy $t$0.
2. For each subgoal $t$1, compute $t$2 by comparing the emitted output with a ground-truth or a verifiable programmatic check.
3. Optionally, compute a terminal or outcome-level reward.
4. Aggregate subgoal and terminal rewards for the total $t$3.
5. Use normalized advantage estimates (e.g., in GRPO) and update $t$4 by minimizing the clipped surrogate loss, optionally regularizing KL to a reference model.
6. Hyperparameters such as the reward weights ($t$5, $t$6, $t$7), the blending parameter ($t$8), and the $t$9-score are tuned empirically.

SGVR-based methods have adopted rule-based verifiers for symbolic steps (Pronesti et al., 23 Jan 2026), transformer-based verifiers for semantic and mathematical equivalence in sub-answers (Zhang et al., 7 Aug 2025), and formal verification engines for geometric subgoals (Chen et al., 8 Jan 2026).

3. Theoretical Motivation: Vanishing Gradients and Credit Assignment

Sparse, outcome-only rewards yield vanishing gradients for subgoal-relevant parameters, as formalized by the "Vanishing Grounding Gradient under Outcome-Only Reward" theorem (Chen et al., 2 Mar 2026):

$r_t(Y;x)$0

where $r_t(Y;x)$1 is a grounding-head logit, $r_t(Y;x)$2 is the marginal selection probability, and $r_t(Y;x)$3 is the probability that all other required evidence chunks are selected—exponentially small in the size of the gold set. This drives the gradient to near zero early in RL, rendering credit assignment intractable for complex or long-context tasks.

SGVR circumvents this by supplying a dense, decomposed reward with stable per-subgoal gradients:

$r_t(Y;x)$4

so that the variance term remains nonzero, sustaining consistent learning signals for the grounding or subgoal-selection heads.

4. Empirical Performance and Experimental Protocols

SGVR frameworks have yielded systematic increases in model performance across diverse domains:

• In long-context retrieval, LongRLVR improved Qwen2.5-14B-1M on RULER-QA from 73.17 to 88.90 and LongBench v2 from 39.8 to 46.5 (Chen et al., 2 Mar 2026).
• Verifiable Process Reward Models (VPRMs) delivered +20 points macro-F1 over best pretrained LLMs and +6.5 over outcome-only RLVR, with coherence on reasoning traces rising from ~50% to ~89% (Pronesti et al., 23 Jan 2026).
• StructVRM on Seed-StructVRM outperformed supervised and RL-only baselines by 3.7–5.3 points on STEM-Bench, ScienceQA, and RealworldQA (Zhang et al., 7 Aug 2025).
• In geometric proof, Skeleton Rate on GeoGoal Test improved from 50.2% (baseline) to 87.7% (SGVR). Cross-domain generalization included gains of 8.0% on general math, and 2.8% on general reasoning benchmarks (Chen et al., 8 Jan 2026).
• For few-shot vision-language reasoning (e.g., satellite imagery), rule-based and IoU-based SGVRs yielded strong improvements over the base model with as few as a single reward-checked example, and robust generalization as the example count scaled (Koksal et al., 29 Jul 2025).

SGVR systems typically require carefully curated reward checkers (rule-based programs, trained verifiers, or formal engines), group advantage normalization, and empirical tuning of reward weights and regularization parameters.

5. Subgoal Granularity: Decomposition Schemes and Verifier Construction

Effective application of SGVR requires a principled decomposition of the global task:

• In long-context QA, subgoals correspond to selecting context chunks, matched with ground-truth evidence sets; chunk selection is scored via $r_t(Y;x)$5 between selected and gold indices (Chen et al., 2 Mar 2026).
• In open-domain reasoning or procedural tasks (e.g., medical evidence assessment), steps are synchronized to task-specific guideline schemas; correctness is checked stepwise by deterministic logic (Pronesti et al., 23 Jan 2026).
• In multimodal VQA or STEM QA, each answer slot, blank, or sub-question is individually compared via model-based verifiers, often using semantic equivalence (Zhang et al., 7 Aug 2025).
• For geometric reasoning, subgoals align with formal proof milestones, mapped to auto-verified numeric or symbolic objectives (Chen et al., 8 Jan 2026).

Constructing subgoal verifiers varies. For domain-standardized tasks (e.g., Cochrane RoB2), rule-based or programmatic extraction is possible; in open domains, learned verifier models (transformers) are trained against LLM-generated or human-annotated subgoal judgments, reaching high agreement with expert evaluation.

6. Extensions, Limitations, and Prospective Directions

SGVR’s architecture lends itself to extensive transfer and generalization:

• Multi-stage or multi-hop tasks (e.g., multi-hop retrieval, program synthesis with linewise verifiability) can instantiate layered or modular subgoal rewards (Chen et al., 2 Mar 2026).
• Joint training with multiple verifiers or jointly optimized critics for discrete reasoning skills is plausible (Pronesti et al., 23 Jan 2026, Zhang et al., 7 Aug 2025).
• Preliminary results indicate SGVR-trained models develop broadly transferable deductive strategies, with gains not strictly confined to the training domain (Chen et al., 8 Jan 2026).

Limitations arise from the dependence on programmatically or formally verifiable subgoals. Symbolic, diagrammatic, or “fuzzy” intermediate steps are not fully addressed, and human-generated argument diversity may exceed what current verifier/engine pipelines capture (Chen et al., 8 Jan 2026). Over-specialization to synthetic, auto-verified subgoal scaffolds can limit cross-domain robustness if not carefully mediated (e.g., through subgoal masking (Chen et al., 8 Jan 2026)).

Future research aims to enrich the representational expressivity of verifiers, integrate advanced formal systems (e.g., Lean4), and expand process-oriented reward templates to cover more diverse reasoning genres including multimodal interpretation and hybrid symbolic-visual chains.

7. Significance and Comparative Impact

SGVR establishes a unified principle for process supervision in complex reasoning domains: transform black-box outcome supervision into a transparent, modular, and verifiable mesh of subgoal signals. This enables dense and stable credit assignment, rectifies vanishing gradients in long-horizon settings, and provides actionable feedback for both training and diagnosis. Across language, vision-language, and mathematical domains, SGVR consistently produces models with higher end-task accuracy, more coherent intermediate reasoning, and greater generalization relative to outcome-only RLVR approaches (Chen et al., 2 Mar 2026, Pronesti et al., 23 Jan 2026, Zhang et al., 7 Aug 2025, Chen et al., 8 Jan 2026, Koksal et al., 29 Jul 2025).