Seed-StructVRM Multimodal Reasoning

• The paper demonstrates a novel approach by integrating a structured reward system that assigns fine-grained, sub-question-level scores to support complex STEM-style multimodal reasoning.
• Seed-StructVRM is built on ByteDance’s internal SEED vision-language stack and a large-scale MoE model, refined through supervised fine-tuning and PPO-based reinforcement learning.
• The method achieves state-of-the-art performance on six public benchmarks and strong results on STEM-Bench, highlighting its effectiveness in handling multi-part, free-form questions.

Seed-StructVRM is a trained multimodal policy model produced by applying the StructVRM method—“Structured and Verifiable Reward Models”—to ByteDance’s internal SEED vision-language stack. Its defining characteristic is that reinforcement learning is driven not only by rule-based verification for simple outputs, but also by a model-based verifier that assigns fine-grained, sub-question-level rewards and recognizes semantic and mathematical equivalence rather than exact string identity. In the formulation reported for StructVRM, this yields a reward pipeline intended for complex STEM-style multimodal reasoning, especially multi-part and free-form problems for which binary end-to-end correctness is too coarse. The resulting model, Seed-StructVRM, is reported to achieve state-of-the-art performance on six of twelve public multimodal benchmarks and strong performance on the newly curated STEM-Bench (Zhang et al., 7 Aug 2025).

1. Definition, scope, and conceptual placement

Seed-StructVRM is the final aligned vision-LLM, not the verifier and not the reward model. StructVRM is the method or framework; Seed-StructVRM is the trained multimodal model obtained by applying that framework to an internal SEED vision-language stack. The base model is described as an internal 20B/200B parameter Mixture-of-Experts (MoE) model, trained first with supervised fine-tuning and then with PPO-based reinforcement learning under structured verifiable rewards (Zhang et al., 7 Aug 2025).

The framework is motivated by the inadequacy of conventional binary reward signals for complex multimodal reasoning. In the reported setting, many tasks are multi-question exam items with mixed answer types, including open-ended explanation, symbolic derivation, numerical answers, and multi-blank fill-in responses. Under such conditions, a single scalar reward for the entire response obscures partial correctness and worsens credit assignment. The paper quantifies this task heterogeneity by stating that about 62.88% of questions are “verifiable” with simple rules, whereas 37.12% are hard-to-verify or non-verifiable under exact-match scoring (Zhang et al., 7 Aug 2025).

A recurrent misconception is to equate Seed-StructVRM with the verifier itself. The reported architecture separates these roles. A model-based verifier is trained to score sub-answers, and a rule-based verifier remains available for simple answer formats; Seed-StructVRM is the downstream policy optimized under those reward signals. This distinction matters because the central claim is not that the policy performs verification internally, but that policy learning benefits from a reward mechanism that is more granular than binary exact-match evaluation (Zhang et al., 7 Aug 2025).

2. Structured and verifiable reward modeling

The core technical object in StructVRM is a verifier that maps a model response $\hat{y}$ and a reference answer $y$ to a structured score vector. The formulation is given as

$$\mathbf{s} = f_\theta(\hat{y}, y) = [s_1, s_2, \dots, s_j]$$

where each $s_j$ is a list of binary correctness indicators for the $j$-th sub-question. A sub-question with multiple blanks may therefore yield a nested structure such as $s_j = [1,0,1]$ rather than a single bit. The structured reward used during RL is then

$$R_{StructVRM} = \frac{1}{k} \sum_{j=1}^k \text{mean}(s_j),$$

with $k$ the number of sub-questions. An equivalent implementation described later computes the mean over the flattened score list and clips the final value to $[0,1]$ (Zhang et al., 7 Aug 2025).

This construction is meant to support partial credit in previously difficult answer formats. The verifier is explicitly instructed to score final answers rather than intermediate reasoning, and to treat mathematically equivalent or semantically equivalent responses as correct. That design is presented as a direct response to the brittleness of rule-based exact matching on open-ended STEM tasks, especially where multiple valid symbolic forms or paraphrases exist (Zhang et al., 7 Aug 2025).

The verifier operates as a structured classifier whose output is serialized as JSON, typically with a score field of the form:

$y$9

Its training set contains more than 200,000 structured examples, each annotated at the sub-question level. The reported held-out evaluation gives 96.83% agreement with expert human judgments, which is the principal reliability claim supporting its use as an online reward source during PPO (Zhang et al., 7 Aug 2025).

3. Data construction and training pipeline

The training pipeline has three major components: supervised fine-tuning of the policy model, supervised training of the verifier, and PPO-based RL with mixed rule-based and model-based rewards. Data curation is itself part of the method, because the framework assumes that multimodal problems can be decomposed into sub-questions or transformed into formats that expose finer-grained supervision (Zhang et al., 7 Aug 2025).

A notable feature is structured data augmentation. Two conversions are described explicitly. In choice → True/False, each multiple-choice option becomes an independent true/false statement, increasing the number of verifiable units. In choice → fill-in-the-blank, the options are removed and the model must generate the answer directly. These augmentations are described as increasing reward granularity and reducing shortcut dependence on option patterns (Zhang et al., 7 Aug 2025).

Stage	Data volume	Function
SFT	51,254 multimodal CoT problems	Initialize reasoning policy
Verifier training	>200,000 structured examples	Train sub-question scorer
RL	>100,000 challenging examples	PPO under structured rewards

The supervised fine-tuning corpus is described as a curated multimodal set spanning math, physics, chemistry, and biology, with multiple-choice, fill-in-the-blank, and open-ended short-answer formats. The CoT traces are generated by multiple internal vision-LLMs, sampled five times per instance under a CoT prompt, then filtered by a preliminary verifier and heuristics. The RL set is a dedicated collection of more than 100,000 challenging examples, selected by a verifier-guided difficulty estimator and targeted at long reasoning chains, symbolic ambiguity, and strong visual grounding (Zhang et al., 7 Aug 2025).

The prompting format used for SFT is also specified. A visual reasoning prompt instructs the model first to describe key visual information, then to perform detailed reasoning, and finally to wrap answers as $\boxed{ANSWER}$ per question. This formatting discipline is later coupled to reward routing, since structured output simplifies both rule-based verification and sub-question parsing (Zhang et al., 7 Aug 2025).

4. Optimization and system training

The SFT stage uses AdamW with $y$0, $y$1, and weight decay $y$2. The learning-rate schedule is cosine decay from $y$3 to $y$4 with 10% warmup. The reported sequence length is 131,072 tokens, with batch size equivalent to 16 × sequence length, and all sequences are truncated to the 131k-token limit (Zhang et al., 7 Aug 2025).

RL is performed with Proximal Policy Optimization (PPO) rather than DPO, GRPO, or pairwise preference optimization. Reward routing is type-dependent. Verifiable prompts, such as simple MCQ or single-answer fill-in questions, use the rule-based reward

$y$5

Hard-to-verify and non-verifiable prompts use the model-based structured reward derived from the verifier output (Zhang et al., 7 Aug 2025).

The RL hyperparameters are unusually explicit. Context length is 8,192 tokens, maximum output length is 16,384, each PPO episode samples 4,096 rollouts, mini-batch size is 512, and gradient steps per episode are 8. The PPO clip range is 0.2. The actor learning rate is $y$6 and the critic learning rate is $y$7. KL regularization is also routed by task type: for general prompts the KL coefficient is $y$8, whereas for verifiable prompts it is 0, which the paper interprets as allowing more exploration when the reward is highly reliable (Zhang et al., 7 Aug 2025).

A second misconception is that the system is reward-modeled only through learned verification. In fact, the reported design is hybrid. Rule-based verifiers remain in place for tasks with deterministic answer formats, and the learned verifier extends RLVR to answer spaces where exact-match rules are brittle or unavailable. This hybridization is presented as a practical expansion of verifiable-reward training rather than a wholesale replacement of symbolic checking (Zhang et al., 7 Aug 2025).

5. Benchmarking, ablations, and empirical profile

The evaluation spans 12 public multimodal benchmarks—including VLM2 Bench, EMMA-Mini, Zerobench (sub), ScienceQA, Mathvision, CMMMU, MMMU, MMMU-Pro, VLMs are Blind, RealworldQA, MME Realworld-cn, and MME Realworld-en—and an additional benchmark, STEM-Bench, introduced specifically to stress high-difficulty multimodal STEM reasoning (Zhang et al., 7 Aug 2025).

Among the public benchmarks, Seed-StructVRM is reported to rank first on six of twelve. The listed first-place scores are 69.8 on VLM2 Bench, 32.6 on Zerobench (sub), 95.1 on ScienceQA, 73.2 on CMMMU, 81.6 on RealworldQA, and 67.1 on MME Realworld-en (Zhang et al., 7 Aug 2025). The paper also notes that the model is competitive rather than uniformly dominant on some math-heavy or closed-model-comparison benchmarks.

STEM-Bench is described as a high-difficulty benchmark in math, physics, chemistry, and biology, with both MCQ and free-form questions, the latter often bundling several sub-questions into a single response. On this benchmark, Seed-StructVRM reports a Final Total of 79.23, compared with 75.51 for the baseline without structured RL, 77.85 for Gemini-2.5-pro, 61.53 for o3, 57.55 for Claude-Sonnet-4, and 45.94 for Qwen2.5-VL-72B (Zhang et al., 7 Aug 2025). The reported subject totals are 72.11 in physics, 77.11 in chemistry, 81.56 in biology, and 86.15 in math.

The ablation study isolates the contributions of RL and the structured verifier. On STEM-Bench, the full system reaches 79.23, the variant w/o StructVRM reaches 76.66, and the variant w/o RL reaches 75.47 (Zhang et al., 7 Aug 2025). The paper interprets this as evidence that PPO-based RL is a major source of gains, while StructVRM contributes an additional improvement, especially on free-form multi-part questions.

A recurring empirical pattern is that the largest gains appear on FFQ rather than MCQ. This is consistent with the design premise: when correctness is distributed across multiple sub-answers, structured partial credit changes the reward landscape much more than it does for single-answer deterministic tasks (Zhang et al., 7 Aug 2025).

6. Limitations, failure modes, and relation to the broader SEED line

The reported limitations are concentrated in visual-symbolic parsing and domain-specific reasoning failure modes. Examples include under-counting bonds in organic chemistry diagrams and physics reasoning that over-relies on heuristic symmetry assumptions. These failures indicate that structured rewards improve credit assignment without eliminating upstream perception or domain-modeling errors (Zhang et al., 7 Aug 2025).

The benchmark profile also shows residual gaps against top closed models in some settings. On STEM-Bench Math, Gemini-2.5-pro remains higher at 91.44, whereas Seed-StructVRM records 86.15. The paper likewise notes that some public math benchmarks remain stronger for systems such as o3 or Gemini, so the method should not be interpreted as uniformly dominant across all multimodal reasoning regimes (Zhang et al., 7 Aug 2025).

In the broader SEED ecosystem, Seed-StructVRM should be understood as a reward-aligned multimodal reasoning model rather than as a tokenizer innovation. The SEED line also includes the SEED tokenizer and SEED-LLaMA, which were introduced to represent text and images interchangeably within a unified autoregressive Transformer and to support both multimodal comprehension and generation (Ge et al., 2023). Seed-StructVRM is reported as operating on ByteDance’s internal SEED vision-language stack (Zhang et al., 7 Aug 2025). This suggests an architectural lineage in which a SEED-family multimodal model is further aligned through structured, verifiable rewards, though the reward-model paper does not specify tokenizer-level architectural details.

The central significance of Seed-StructVRM lies in the claim that verifiability in multimodal RL need not collapse to a single binary outcome. By replacing monolithic end-of-response reward with sub-question-level structured verification, the framework expands RLVR from simple deterministic formats to free-form, multi-part STEM reasoning. Within the evidence reported so far, the strongest support for that claim is the combination of verifier reliability (96.83% agreement with expert judgments), the STEM-Bench ablations, and the benchmark results showing disproportionate gains on free-form tasks (Zhang et al., 7 Aug 2025).