DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification

Abstract: Aligning Multimodal LLMs (MLLMs) requires reliable reward models, yet existing single-step evaluators can suffer from lazy judging, exploiting language priors over fine-grained visual verification. While rubric-based evaluation mitigates these biases in text-only settings, extending it to multimodal tasks is bottlenecked by the complexity of visual reasoning. The critical differences between responses often depend on instance-specific visual details. Robust evaluation requires dynamically synthesizing rubrics that isolate spatial and factual discrepancies. To address this, we introduce $\textbf{DeltaRubric}$, an approach that reformulates multimodal preference evaluation as a plan-and-execute process within a single MLLM. DeltaRubric operates in two steps: acting first as a $\textit{Disagreement Planner}$, the model generates a neutral, instance-specific verification checklist. Transitioning into a $\textit{Checklist Verifier}$, it executes these self-generated checks against the image and question to produce the final grounded judgment. We formulate DeltaRubric as a multi-role reinforcement learning problem, jointly optimizing planning and verification capabilities. Validated on Qwen3-VL 4B and 8B Instruct models, DeltaRubric achieves solid empirical gains. For instance, On VL-RewardBench, it improves base model overall accuracy by $\textbf{+22.6}$ (4B) and $\textbf{+18.8}$ (8B) points, largely outperforming standard no-rubric baselines. The results demonstrate that decomposing evaluation into structured, verifiable steps leads to more reliable and generalizable multimodal reward modeling.

• The paper introduces DeltaRubric, which decomposes multimodal evaluation into a dual-role planner-verifier framework to overcome lazy judging.
• It leverages joint reinforcement learning and instance-specific checklist generation to enhance visual verification and reduce hallucinated responses.
• Empirical tests on Qwen3-VL-4B and 8B Instruct models show accuracy improvements of up to +22.6 points, marking significant performance gains.

DeltaRubric: Structured Multimodal Reward Modeling via Joint Planning and Verification

Motivation and Problem Statement

High-fidelity reward modeling is essential for aligning Multimodal LLMs (MLLMs) through RLHF protocols. Existing multimodal evaluators rely heavily on monolithic, single-step judgment paradigms, frequently exhibiting "lazy judging"—models exploit language priors or superficial text features rather than robust visual verification. While rubric-driven evaluation is established in text-only domains, its extension to multimodal tasks is bottlenecked by the complexity inherent to instance-specific visual reasoning. DeltaRubric directly addresses this bottleneck by decomposing multimodal evaluation into a sequential plan-and-execute procedure, operationalized entirely within a single shared MLLM.

DeltaRubric Framework

DeltaRubric introduces a dual-role architecture—Disagreement Planner and Checklist Verifier—implemented as roles within a single MLLM. For each evaluation input $(I, q, y_A, y_B)$, the Disagreement Planner synthesizes a neutral, instance-specific checklist isolating factual and spatial disagreements between candidate responses. The Checklist Verifier then systematically executes these checks against the image and query, producing a grounded rationale and a final judgment.

Figure 1: DeltaRubric overview, showing sequential Planner and Verifier operations for targeted disagreement-driven evaluation.

Joint optimization of planning and verification is achieved via multi-role reinforcement learning: Planning rewards are computed for checklists that correct the Verifier's baseline blind spots, and Verification rewards reflect both accuracy and checklist-guided improvement. Advantages are decoupled per role, stably aggregated across distinct rollout groups to prevent cross-task reward contamination.

Empirical Evaluation and Results

DeltaRubric was validated on Qwen3-VL-4B and 8B Instruct models using challenging vision-language reward benchmarks. On VL-RewardBench, DeltaRubric improved base model accuracy by +22.6 (4B) and +18.8 (8B) points, outperforming no-rubric RLHF baselines by +4.3 and +8.1, respectively. Gains are most pronounced in tasks requiring granular visual grounding, such as hallucination detection and spatial reasoning.

Figure 2: Verifier training dynamics illustrating accuracy improvements and alignment stability over time.

DeltaRubric's ability to synthesize targeted checklists enables systematic detection of visual inconsistencies and hallucinations. Qualitative analyses reveal DeltaRubric catches errors overlooked by baselines, such as hallucinated object attributes, illogical spatial relationships, and fine-grained attribute binding.

Figure 3: Targeted checklist generation and enforced visual verification allows DeltaRubric to correctly catch hallucinated objects missed by standard baselines.

Figure 4: DeltaRubric isolates fine-grained visual discrepancies—e.g., shoe color hallucination—yielding correct preference judgments.

Figure 5: Logical inconsistencies are systematically mitigated by the checklist-driven verification in DeltaRubric.

Figure 6: Explicit verification of visual attribute binding in DeltaRubric prevents confusions that lead to preference errors in baselines.

DeltaRubric further demonstrates strong generalization in Multimodal RewardBench, with the 8B model improving overall accuracy by +5.5 points and outperforming the no-rubric baseline by +4.5. Ablation experiments confirm the necessity of active Planner RL optimization, relative improvement-based reward formulation, and image-conditioned checklist generation. Static rubrics yield moderate improvement, but lack the adaptability for diverse, instance-specific discrepancies—DeltaRubric surpasses static checklist approaches by substantial margins in reasoning-intensive tasks.

Theoretical and Practical Implications

DeltaRubric’s structural decomposition realigns reward modeling as an active visual investigation process, as opposed to passive holistic scoring. This enforces rigorous empirical grounding, mitigating shortcut exploitation by reward models. The approach establishes a new standard for multimodal preference modeling: disagreement-driven, checklist-guided evaluation is more reliable, interpretable, and generalizable across dataset domains and architectures.

The dual-role RL optimization paradigm and instance-specific rubric generation augment both visual and language reasoning capabilities. Notably, multimodal fine-tuning with DeltaRubric enhances text-only benchmark performance, indicating improved structural logic without catastrophic modality forgetting.

Future Directions

Several avenues are ripe for extension: (1) dynamic routing for selective checklist generation in ambiguous instances, (2) application to temporal modalities (e.g., video), and (3) integration with alternative RL algorithms to further probe architectural independence. The planner-verifier split may inform future agentic evaluation protocols, enabling model self-verification by explicit disagreement isolation and targeted evidence querying.

Conclusion

DeltaRubric reframes multimodal reward modeling as a plan-and-execute process, inducing superior visual verification and grounded rationale via joint planner-verifier RL optimization. Empirical evidence supports substantial performance gains across vision-language benchmarks, with explicit mitigation of lazy judging and enhanced structural reasoning. The framework’s structural priors and decoupled advantage assignment advance the field towards more interpretable, robust, and adaptable alignment strategies for MLLMs (2605.09269).