Mix-GRM: Modular Generative Reward Modeling

• Mix-GRM is a generative reward model that decomposes evaluation into Breadth-CoT, emphasizing broad multi-principle judgments, and Depth-CoT, focusing on detailed, step-by-step verification.
• The framework uses a modular synthesis pipeline to parse raw rationales into a uniform Principle-Judgment-Verdict schema and merge multiple inputs for robust performance.
• RLVR fine-tuning acts as a switching amplifier, aligning the model’s reasoning style with task requirements and yielding significant improvements in both subjective preference and objective correctness.

Searching arXiv for Mix-GRM and closely related generative reward model papers. arxiv_search(query="Mix-GRM generative reward model", max_results=10, sort_by="relevance") arxiv_search(query="generative reward model breadth depth chain of thought RLVR reward model", max_results=10, sort_by="relevance") Mix-GRM is a generative reward modeling framework that factorizes reward-model reasoning into two explicit mechanisms—Breadth-CoT and Depth-CoT—and then recombines them within a single Generative Reward Model (GRM). In this formulation, the reward model does not merely output a scalar preference or a pairwise label; it generates a rationale together with a verdict, and the central claim is that the structure of that rationale matters at least as much as its length. Mix-GRM was introduced to address the limitation of unstructured Chain-of-Thought length scaling in GRMs, particularly for the contrast between subjective preference evaluation and objective correctness evaluation (Zhang et al., 2 Mar 2026).

1. Position within reward modeling

A conventional reward model in RLHF is discriminative: given an instruction $x$ and one or more candidate responses $\{y\}$, it outputs either a scalar score $r_\theta(x,y)\in\mathbb{R}$ or a pairwise preference probability $\Pr_\theta(y_A \succ y_B\mid x)$. A Generative Reward Model instead treats the judge as a LLM that emits both a rationale $c$ and a final verdict $v$, typically of the form

$(c,v)=M(y_A,y_B\mid x),$

with $v\in\{\text{[[A]]},\text{[[B]]}\}$ (Zhang et al., 2 Mar 2026).

The immediate backdrop for Mix-GRM is prior work on length scaling in GRMs. That line of work had shown that longer CoTs often improve evaluation reliability, but it had generally treated additional reasoning tokens as a homogeneous resource. Mix-GRM rejects that assumption. It distinguishes between two orthogonal reasoning mechanisms: a breadth-oriented mode that expands evaluation across multiple principles, and a depth-oriented mode that emphasizes substantive step-by-step verification. The framework therefore reframes GRM post-training as a problem of mechanism selection and internalization rather than simple rationale elongation.

This suggests that Mix-GRM is not only a scaling method but also a structural hypothesis about reward modeling: different evaluation regimes require different CoT topologies, and forcing a single monolithic rationale style across all tasks is suboptimal.

2. Breadth-CoT and Depth-CoT

Mix-GRM defines two reasoning styles.

Breadth-CoT (B-CoT) emphasizes multi-dimensional principle coverage. An evaluation is decomposed into units

$S=\{(p_k,j_k,u_k)\}_{k=1}^K,$

where $p_k$ is an evaluation principle, $\{y\}$0 is a principle-specific judgment, and $\{y\}$1 is a sub-verdict. In the reported pipeline, $\{y\}$2. B-CoT is therefore a parallel aggregation over diverse criteria such as instruction adherence, tone, linguistic alignment, or practical insight. Its intended strength is coverage.

Depth-CoT (D-CoT) emphasizes substantive, step-by-step reasoning. It first generates a reasoning trace $\{y\}$3 that solves or verifies the underlying task, then rewrites judgments to be explicitly grounded in that trace. Conceptually, it evaluates responses after internally solving the problem, rather than by scanning many surface-level principles.

Mechanism	Construction	Strongest reported domain
B-CoT	Multi-principle aggregation over $\{y\}$4	Preference
D-CoT	Reasoning trace $\{y\}$5 followed by trace-grounded judgments	Correctness
Mixed	Single GRM trained to internalize both mechanisms	Preference and Correctness

The empirical dissociation reported for these mechanisms is central. B-CoT benefits subjective preference tasks, whereas D-CoT benefits objective correctness tasks. Conversely, misalignment is detrimental: using breadth when rigorous correctness checking is required degrades performance, and using depth for subjective preference tasks also degrades performance (Zhang et al., 2 Mar 2026).

A common misconception is that Mix-GRM merely asks the model to "reason more." The framework instead asserts that reasoning style is task-dependent. Longer CoTs are only useful insofar as their structure matches the evaluation regime.

3. Modular synthesis pipeline and representation

Mix-GRM is trained on synthesized rationales rather than on raw unstructured judge outputs. The first stage is modular schema standardization. Starting from a conventional Vanilla-CoT generated by a judge-style prompt, a strong LLM, specifically DeepSeek-V3, parses the rationale into a uniform Principle–Judgment–Verdict schema,

$\{y\}$6

Each principle receives a descriptive name and normative definition, and each judgment is paraphrased so that the comparison between A and B is explicit (Zhang et al., 2 Mar 2026).

The second stage is mechanism synthesis.

For B-CoT, the pipeline samples $\{y\}$7 independent Vanilla-CoTs for the same input, parses each into structured units, pools all units, and applies an LLM-based Merge & Deduplicate transformation. Semantically similar principles are grouped, names and definitions are refined, and source judgments are synthesized into merged judgments. The sub-verdict is chosen by majority vote across source units; ties yield $\{y\}$8. A "Top-10 frequent principles" heuristic is reported to improve robustness by suppressing noisy or idiosyncratic criteria. Formally,

$\{y\}$9

For D-CoT, the pipeline first generates a reasoning trace

$r_\theta(x,y)\in\mathbb{R}$0

then selects a focused subset $r_\theta(x,y)\in\mathbb{R}$1 with $r_\theta(x,y)\in\mathbb{R}$2, and refines judgments conditioned on the trace: $r_\theta(x,y)\in\mathbb{R}$3 The refined judgments explicitly cite the trace using a special marker such as <Answer>, thereby forcing evaluation to be grounded in the internally produced derivation. The result is a serialized rationale in which the model first reasons, then judges.

At inference time, Mix-GRM remains a single GRM with input $r_\theta(x,y)\in\mathbb{R}$4 and output $r_\theta(x,y)\in\mathbb{R}$5. The distinction lies in how the training targets for $r_\theta(x,y)\in\mathbb{R}$6 are synthesized and aligned.

4. Supervised fine-tuning and RLVR

Training proceeds in two phases: supervised fine-tuning on synthesized mixture CoTs and reinforcement learning with verifiable rewards via GRPO.

The reported data source consists of 30,876 samples from five datasets: HelpSteer3, Code-Preference, Math-DPO, WildGuard, and OffsetBias. Following the paper’s task split, examples are partitioned into a Preference domain for subjective tasks and a Correctness domain for objective tasks. Preference examples receive synthesized B-CoTs, while Correctness examples receive synthesized D-CoTs. The resulting mixture dataset is

$r_\theta(x,y)\in\mathbb{R}$7

The base model is Qwen3-8B-Base, and the reported SFT stage uses 9K synthesized samples, 2 epochs, learning rate $r_\theta(x,y)\in\mathbb{R}$8, batch size 128 with gradient accumulation, sequence length 12,288, no weight decay, and 5% warmup (Zhang et al., 2 Mar 2026).

The SFT objective is standard next-token cross-entropy over the full rationale-plus-verdict sequence: $r_\theta(x,y)\in\mathbb{R}$9

The second phase, RLVR via GRPO, uses only final-verdict correctness as reward. For a rollout $\Pr_\theta(y_A \succ y_B\mid x)$0,

$\Pr_\theta(y_A \succ y_B\mid x)$1

No reward term depends directly on CoT length, principle count, or trace presence. The reported GRPO objective is

$\Pr_\theta(y_A \succ y_B\mid x)$2

with 8 rollouts per prompt, temperature $\Pr_\theta(y_A \succ y_B\mid x)$3, learning rate $\Pr_\theta(y_A \succ y_B\mid x)$4, and KL coefficients of $\Pr_\theta(y_A \succ y_B\mid x)$5 (Zhang et al., 2 Mar 2026).

The notable result is that RLVR is reported to act as a switching amplifier. After SFT, the structural match rate between domain and mechanism is about 73%. After RLVR, despite the absence of explicit structural supervision, that rate reportedly rises to 95%, with the model using Breadth-like CoTs for preference tasks and Depth-like CoTs for correctness tasks. The paper describes this as an emergent polarization of reasoning style.

5. Empirical performance and downstream use

Mix-GRM is evaluated on five reward benchmarks: RewardBench v1, RewardBench v2, RM-Bench, RMB, and PPE. The evaluation metric is pairwise preference accuracy. The reported comparison set includes discriminative and generative baselines such as Skywork-Reward-8B, JudgeLRM-7B, RM-R1-7B, DeepSeek-GRM-16B, FARE-8B, and RubricRM-8B (Zhang et al., 2 Mar 2026).

The main aggregate results reported are as follows. In the SFT-only setting, Mix-GRM achieves 75.1 average across benchmarks, exceeding RubricRM-8B at 74.7 and DeepSeek-GRM-16B at 65.2, while approaching FARE-8B at 75.9 despite using only 9K SFT samples versus 2.5M. In the SFT+RLVR setting, Base-GRM reaches 76.9 and Mix-GRM reaches 79.4. The abstract summarizes the gain over leading open-source RMs as an average of 8.2% (Zhang et al., 2 Mar 2026).

The domain-specific breakdown is equally important. At the SFT stage, Breadth-only Mix-GRM improves the Preference domain but hurts Correctness, while Depth-only Mix-GRM improves Correctness but hurts Preference. Mixed SFT improves both. After RLVR, the mixed model reaches 72.6 on Preference and 78.7 on Correctness, surpassing Base-GRM, Breadth-only, and Depth-only variants on both sides of the split. The paper therefore argues that the gain is not attributable to either mechanism in isolation, but to joint internalization plus RLVR-driven mechanism selection.

The paper also reports downstream utility. As a reward source for RLHF or DPO, policies trained with Mix-GRM achieve the highest win rates on AlpacaEval-2 and Arena-Hard for instruction following, and the best average accuracy for mathematical reasoning at 46.4, compared with 45.9 for RubricRM and 45.8 for RM-R1-Instruct. As a verifier for Best-of-10 test-time scaling on MATH, CHAMP, MBPP+, and BigCodeBench, Mix-GRM is reported to outperform FARE-8B, RM-R1-7B, DeepSeek-GRM-16B, and SkyCritic; on MATH Best-of-10, the reported accuracy is 43.2%, versus approximately 35.2 for FARE-8B and 37.7 for RM-R1-7B (Zhang et al., 2 Mar 2026).

Ablation analyses support the framework’s internal claims. Increasing the number of parallel CoTs $\Pr_\theta(y_A \succ y_B\mid x)$6 from 1 to 4 improves preference performance for B-CoT. Among breadth principle-selection strategies, "Top-10 frequent" performs best. Pipeline-stage accuracies are reported as 87.1% for raw rationale, 90.2% after B-CoT merging, and 88.5% after D-CoT generation. Training with noisy versus filtered CoTs yields nearly identical SFT performance, approximately 69.8 versus 70.1.

6. Limitations, failure modes, and significance

The paper identifies several limitations. The Breadth/Depth dichotomy is explicitly described as a low-rank approximation to a richer manifold of reasoning mechanisms. Real evaluation tasks may combine stylistic nuance, safety, instruction following, factual verification, code execution, and long-horizon reasoning in ways that are not exhaustively captured by the Preference-versus-Correctness split. The model’s RLVR-induced polarization may therefore become rigid in hybrid scenarios requiring simultaneous deep derivation and nuanced preference awareness (Zhang et al., 2 Mar 2026).

The framework also depends on LLM-generated rationales from DeepSeek-V3 for schema extraction and mechanism synthesis. Although the reported pipeline is robust to noise, this dependence implies that biases or blind spots in the generator can propagate into the trained reward model. The synthesis stage itself introduces a nontrivial one-time compute cost because it requires multiple passes per training example.

The reported failure cases are mechanism-specific. B-CoT failure in correctness is characterized as feature interference: superficial detail, verbosity, or formatting may be mistaken for correctness. D-CoT failure in preference is characterized as attentional tunneling: the model may focus on logical or technical strength while ignoring language, tone, or cultural alignment. One case study described in the paper involves choosing a fluent English answer even when the query was posed in Japanese because the English answer contained richer historical analysis.

Within the reward-model literature, Mix-GRM’s significance lies in elevating rationale structure to a primary design axis. Its core claim is not merely that CoTs should be longer, but that evaluation rationales should be modular, mechanism-specific, and domain-aligned. The framework accordingly treats B-CoT and D-CoT not as prompt-engineering artifacts but as trainable internal competencies whose deployment can be sharpened through verdict-only reinforcement. A plausible implication is that future GRMs may move beyond a binary breadth/depth split toward more granular reasoning taxonomies, soft mechanism routing, or multi-stage evaluators that first scan broadly and then verify deeply.