---
title: Sub-Goal Verifiable Reward (SGVR) Framework
url: https://www.emergentmind.com/topics/sub-goal-verifiable-reward-sgvr-framework
type: topic
---

# Sub-Goal Verifiable Reward (SGVR) Framework

The Sub-Goal Verifiable Reward (SGVR) Framework is a general methodology for reinforcement learning (RL) and agent training that decomposes complex tasks into verifiable sub-goals, allowing reward signals to be assigned based on independent, auditable evidence for each intermediate milestone or answer component. This approach is motivated by the inadequacy of end-to-end (outcome-based) reward methods, which impede robust learning in reasoning, multimodal, tool-using, or code-generation settings due to sparsity, misalignment, and reward hacking vulnerabilities. SGVR frameworks systematically extract, verify, and aggregate fine-grained signals, enabling dense, reliable supervision via explicit sub-goal success. This paradigm has been instantiated in diverse domains including multimodal reasoning, geometric proof, long-form retrieval-augmented generation, code synthesis, and agent skill acquisition.

## 1. Formal Structure of Sub-Goal Verifiable Reward

SGVR relies on a decomposition of the agent’s output or trajectory into a set of sub-goals—atomic components (blanks, proof steps, nuggets, unit tests, skills) with deterministic, independently checkable verification criteria. Each sub-goal corresponds to a unique, reconstructible evidence item, and the SGVR reward aggregates component-wise scores to a structured or scalar feedback signal.

Let $a$ denote the agent’s answer, $G = \{g_1, \ldots, g_J\}$ the set of sub-goals, and $y$ the ground-truth answer annotated at the sub-goal level. The SGVR verifier $f_\theta(a, y)$ outputs a vector $s = [s_1, \ldots, s_J]$, $s_j \in \{0,1\}^{m_j}$, encoding the binary (or graded) correctness of each blank or component in sub-goal $g_j$ [2508.05383], [2601.05073], [2510.14660], [2601.03525].

A scalar reward is derived as
$$
R_{\text{SGVR}}(a, y) = \frac{1}{K} \sum_{j=1}^J \text{mean}(s_j), \quad K = \sum_{j=1}^J m_j,
$$
or, more generally,
$$
R_\phi(q, \hat{y}) = \frac{\sum_{i=1}^{k} w_i V_\varphi(q, \hat{y}, r_i)}{\sum_{j=1}^k w_j},
$$
where $V_\varphi(q, \hat{y}, r_i)$ is a learned verifiable score per rubric $r_i$ (e.g., "nugget-as-rubric" for long-form QA) and $w_i$ are rubric weights [2510.14660].

The formal structure extends to multi-turn trajectories or skills in agentic systems, where the total reward is a sum over sub-goal verifications with evidence bundles:
$$
r(\tau) = \sum_j w_j f_j(E_j(\tau)).
$$
This evidence-centric decomposition enables replayable, auditable, and security-hardened reward signal construction [2512.23760].

## 2. Verification Mechanisms and Architectures

SGVR requires a sub-goal verifier capable of evaluating, with high reliability, the correctness or support for each atomic sub-goal:

- **Multimodal Reasoning:** The verifier is a model consuming vision features $V \in \mathbb{R}^{H_v \times D_v}$ and text embeddings $T \in \mathbb{R}^{L_t \times D_t}$. After fusion, per-sub-goal heads classify blanks via $\sigma(W_j h_j + b_j)$ [2508.05383].
- **Proof/Mathematical Reasoning:** Each sub-goal is a numeric checkpoint extracted from formal proof skeletons; correctness is evaluated as exact or tolerance-based numeric match [2601.05073].
- **Retrieval-Augmented Generation:** For each nugget-based rubric, a generative verifier $V_\varphi$ outputs ternary support labels (support/partial/not), with aggregation across blocks, employing models like Search-Gen-V (Qwen3-4B-Instruct backbone) trained with distillation and RL [2510.14660].
- **Code Generation:** Each unit test forms a sub-goal; verification is done via test execution, with pass/fail outcomes. Weights are dynamically estimated from pass rates and density-normalized [2601.03525].
- **Agentic Skills:** Each skill is accompanied by pre/postcondition contracts. Replay-based verification is conducted over held-out suites, registering signed evidence bundles to ensure skill validity [2512.23760].

All frameworks rely on deterministic, replayable evaluation functions to enable offline or post-hoc reward auditing.

## 3. Reward Aggregation and RL Optimization

The per-rollout or per-trajectory reward is a (weighted) sum or average over verified sub-goals. To maximize robust learning, RL algorithms are adapted:

- **Policy Optimization:** SGVR is deployed with PPO or Group Relative Policy Optimization (GRPO), where groups of rollouts per prompt are scored, mean-normalized, and used in clipped-importance-sampling objectives. For example, in code generation:
  $$
  \mathcal{J}_{\rm VeRPO}(\theta) = \mathbb{E}_{x, \{\tau\}} \left[ \frac{1}{\sum_i |\tau_i|} \sum_{i,t} \min (r^{(i)}_t A(\tau_i, t), \mathrm{clip}(r^{(i)}_t, 1\pm\epsilon) A(\tau_i, t)) \right]
  $$
  with $A(\tau, t)$ mixing global and turn-level advantages [2601.03525].

- **Skill Graph Environments:** In agent self-improvement, each candidate skill's promotion and reward eligibility depend on passing signed contract checks and accumulating replayable evidence, with periodic adversarial stress testing and bounded-memory discipline [2512.23760].

- **Hybrid and Hierarchical Tasks:** For reasoning chains or table reconstructions, reward terms may target schema compliance, intermediate reconstructions, stepwise process conformity, and final answers, each admitted as a verifiable component [2510.10973], [2508.05383].

## 4. Domain-Specific Instantiations and Experimental Evaluations

### Table: Representative SGVR Applications

| Domain           | Sub-goal Extraction           | Verification Signal              |
|------------------|------------------------------|----------------------------------|
| Multimodal VQA   | Fill-in blanks, multi-steps  | Model-based, semantic+numeric    |
| Geometry Proof   | Formal proof skeleton steps  | Numeric equality/checkpoints     |
| Retrieval QA     | Nugget/rubric mining         | Ternary generative/verifiable    |
| Code Generation  | Unit test decomposition      | Test execution, weighted         |
| Agent Skills     | Skill nodes/edges in graph   | Interface contract checks        |

- **Multimodal Reasoning:** Seed-StructVRM achieved state-of-the-art on 6/12 public VQA-style benchmarks and a curated STEM-Bench, with improvements of +3.72% absolute on total score and up to +8% per domain [2508.05383].
- **Geometry/Math:** SGVR yielded +9.7% accuracy in geometric reasoning, +8.0% in general math, +2.8% in general reasoning benchmarks versus pretrained models. Gains were attributed to the Skeleton Rate providing dense, aligned supervision [2601.05073].
- **Retrieval QA:** Rubric-based SGVR delivered F1 scores (rubric-level/sample-level) near those achieved by much larger oracles, while boosting correlation with independent comprehensiveness metrics ($r=0.70$). Verification efficiency improved by $>$6$\times$ for 1,000 rubrics [2510.14660].
- **Code Generation:** VeRPO’s SGVR offered up to +8.83% absolute gain in pass@1 compared to outcome-based or RM-based baselines, and reduced degenerate-group ratios by $>4\times$ (i.e., non-zero signal efficiency) [2601.03525].
- **Agentic LLMs/Skills:** Audited skill-graph self-improvement leverages SGVR to gate agent improvement with evidence-backed reward decomposition, addressing reward hacking, behavioral drift, and compliance logging [2512.23760].

## 5. Limitations, Security, and Future Extensions

While SGVR corrects key weaknesses of outcome-only supervision, several limitations persist:

- **Verifier Reliance:** Performance is bottlenecked by the accuracy and coverage of the verifier (semantic, numeric, rubric-based, or tool-execution)—bias or gaps in the verifier propagate directly to SGVR reward fidelity [2508.05383], [2510.14660].
  
- **Sub-goal Formulation:** In domains such as non-numeric proofs or diagrammatic reasoning, sub-goal decomposition and reliable mapping from abstract predicates to verifiable tests remain unresolved [2601.05073].
  
- **Sparse Sub-goal Regimes:** Very short test suites or homogeneous sub-task difficulties can degrade SGVR robustness, suggesting the need for adaptive weighting or hybrid reward designs [2601.03525].

- **Security/Robustness:** Threat models assume append-only logging, enclave-based verifier isolation, cryptographic signing, and deterministic replay for evidence—security guarantees depend on rigorous enforcement of boundaries and transparency [2512.23760].

Proposed extensions include integrating proof assistant outputs as sub-goal sources, hybridizing with human feedback for coverage of non-automatable tasks, and expanding SGVR to new domains such as planning and code synthesis with compositional tool checkers [2508.05383], [2601.05073].

## 6. Significance and Broader Impact

SGVR reframes reward modeling as the aggregation of independently verifiable intermediate achievement, bridging the gap between coarse supervised approaches and opaque, model-based reward functions. Empirically, SGVR yields:

- Dense, low-variance, and interpretable feedback, reducing overfitting and improving out-of-distribution robustness [2510.10973].
- Stronger, more faithful chain-of-thought and process conformity, enhancing model trust and transparency [2510.14660].
- Auditability and reproducibility in long-horizon agentic learning and self-improvement, since every reward component is reconstructible from signed evidence artifacts [2512.23760].
- Broad transferability, with gains in cross-domain performance, as step-level reasoning competencies generalize beyond training targets [2601.05073].

SGVR thus establishes a new foundation for verifiable, modular and governance-ready RL architectures in complex AI systems.

Source: https://www.emergentmind.com/topics/sub-goal-verifiable-reward-sgvr-framework