Learning What Matters: Dynamic Dimension Selection and Aggregation for Interpretable Vision-Language Reward Modeling

Abstract: Vision-language reward modeling faces a dilemma: generative approaches are interpretable but slow, while discriminative ones are efficient but act as opaque "black boxes." To bridge this gap, we propose VL-MDR (Vision-Language Multi-Dimensional Reward), a framework that dynamically decomposes evaluation into granular, interpretable dimensions. Instead of outputting a monolithic scalar, VL-MDR employs a visual-aware gating mechanism to identify relevant dimensions and adaptively weight them (e.g., Hallucination, Reasoning) for each specific input. To support this, we curate a dataset of 321k vision-language preference pairs annotated across 21 fine-grained dimensions. Extensive experiments show that VL-MDR consistently outperforms existing open-source reward models on benchmarks like VL-RewardBench. Furthermore, we show that VL-MDR-constructed preference pairs effectively enable DPO alignment to mitigate visual hallucinations and improve reliability, providing a scalable solution for VLM alignment.

• The paper introduces VL-MDR, a framework that decomposes reward signals into fine-grained dimensions for interpretable vision-language evaluation.
• It employs a transformer backbone with decoupled heads for dimension prediction, fine-grained scoring, and adaptive weighting, optimizing performance with minimal overhead.
• Empirical results demonstrate that VL-MDR attains state-of-the-art accuracy on VL-RewardBench by balancing interpretability with computational efficiency.

Interpretable Vision-Language Reward Modeling via Dynamic Dimension Selection and Aggregation

Introduction

The paper "Learning What Matters: Dynamic Dimension Selection and Aggregation for Interpretable Vision-Language Reward Modeling" (2604.05445) identifies fundamental limitations in current paradigms for reward modeling (RM) in Large Vision-LLMs (LVLMs). Generative RMs offer interpretability by supplying textual reasoning but are computationally prohibitive and potentially biased, while discriminative RMs are efficient yet reduce multimodal evaluation to uninformative scalar outputs. This work proposes VL-MDR, a framework that integrates the interpretability of multidimensional generative critiques with the efficiency of discriminative scoring by dynamically decomposing reward signals into fine-grained, contextually relevant dimensions.

Figure 1: VL-MDR achieves both interpretability and efficiency, unlike traditional generative and discriminative reward models.

Dataset Construction and Taxonomy

A comprehensive, fine-grained preference dataset was curated to support dimension-based supervision. The construction pipeline builds on 414k preference pairs aggregated from seven public VLM feedback sources, spanning both human and AI-generated supervision. Samples are annotated by multi-model agreement with their top-3 most relevant evaluation dimensions from a hierarchical taxonomy, which includes 21 fine-grained sub-dimensions branching from 7 core capabilities (e.g., Perception, Reasoning, Safety, Math).

Figure 2: Dataset construction and capability distribution illustrating filtered source contributions and diversity of tagged skills.

The annotation protocol involves automated dimension prediction and preference verification using multiple SOTA VLM judges, with only high-agreement cases retained (77.6% retention). This results in 321k robust pairs and 964k dimension labels, distributing coverage across both general and domain-specific competencies, thus facilitating precise, disentangled supervision for vision-language evaluation.

Methodology: The VL-MDR Framework

VL-MDR formalizes reward modeling as a process of learning what matters for each multimodal input. The model architecture, built atop a transformer VLM backbone, consists of three decoupled heads:

1. Dimension Prediction: Given an instruction, the model predicts a probability vector over the 21 evaluation dimensions, using a visual-aware gating procedure to select the top-$k$ most relevant axes (empirically, $k=3$ is optimal).
2. Fine-Grained Scoring: Candidate responses are independently scored on each active dimension, capturing response quality with respect to specific, interpretable criteria.
3. Adaptive Weighting: The overall reward is computed as a weighted sum of the fine-grained scores, where weights are predicted conditioned on the instruction and masked to exclude irrelevant dimensions.

   Figure 3: Decoupled evaluation pipeline: query-conditioned dimension selection, per-dimension scoring, and adaptive aggregation.

This design explicitly separates evaluation context (instruction intent) from response content, simulating human cognitive evaluation processes, and ensures that only contextually-relevant aspects contribute to the final reward.

The training objective combines binary cross-entropy for dimension prediction, a unified pairwise ranking loss on dimension-wise preferences, and scalar overall reward alignment. This hierarchical multi-objective strategy leverages granularity in annotation to incentivize both accurate selection and high-fidelity comparison across fine-grained dimensions.

Empirical Evaluation

Benchmark Results

VL-MDR is evaluated against both open-source and proprietary reward models on three challenging multimodal benchmarks: VL-RewardBench, Multimodal RewardBench, and MM-RLHF-Reward Bench. Notably, VL-MDR achieves:

• VL-RewardBench Macro Accuracy: 71.8% (SOTA among open 7B-scale RMs)
• Consistent high accuracy across General, Hallucination, and Reasoning categories
• Comparable or superior overall and strict ranking performance to large-scale proprietary and generative RMs

This demonstrates that dimension-aware supervision yields reliable preference identification across diverse multimodal tasks, matching or exceeding models using monolithic supervision pipelines.

Ablations and Analysis

Ablations confirm that both dimensional granularity and instruction-aware gating are critical. Removing per-dimension supervision or gating degrades overall reward accuracy, while increasing dimension count beyond $k=3$ introduces noise, indicating a sparsity-optimal trade-off. Computationally, VL-MDR incurs marginal additional inference cost (0.25% parameter growth over backbone; $\sim$0.22 GPU-hours for full benchmark), far outperforming generative RMs in throughput.

Figure 4: Selecting three active dimensions optimally balances information retention and noise suppression.

Interpretability and Fine-Grained Judgments

Appendix analyses demonstrate strong internal consistency (64% of cases see all three dimension-wise judgments align with overall preference) and robust identification of dimension co-occurrence structures (e.g., "Location + Attribute" frequently co-occur). Ambiguity analysis further shows tie rates are highest for subjective or ill-defined axes (e.g., "Celebrity") and lowest for objective ones (e.g., "Geometry"), highlighting the model’s capability to capture both clear-cut and ambiguous evaluation subtleties.

Figure 5: High internal consistency between dimension-specific and overall judgment signals.

Figure 6: Most frequent dimension co-occurrence pairs, reflecting naturally coupled vision-language tasks.

Figure 7: Judgment tie rates, proxying dimension difficulty and subjectivity.

Downstream Utility: Reward-Conditioned Alignment

VL-MDR’s reward signals provide higher-fidelity supervision for preference-model-based alignment methods. Using DPO to fine-tune LLaVA-OneVision-7B on VL-MDR-generated pairs yields the strongest or comparable results to all leading baselines on image understanding and hallucination mitigation tasks. This supports the claim that dimensionally-structured RMs capture error modes missed by scalar counterparts and provide a scalable enhancement for LVLM alignment pipelines.

Theoretical and Practical Implications

VL-MDR establishes that interpretable yet efficient reward modeling is achievable via dynamic, context-dependent dimension selection rather than fixed or holistic scalar approaches. The method provides explicit rationales for responses, supports targeted debugging and iterative improvement of VLMs, and enables principled isolation of specific failure modes (e.g., visual hallucination vs. reasoning errors).

The fine-grained dimensional dataset and training paradigm also suggest a unified architecture for multi-task VLM evaluation, easily extensible to new modalities, and more robust to distribution shift due to its explicit handling of task granularity.

Conclusion

This work advances vision-language reward modeling by introducing a dynamic, interpretable, and efficient framework that synergizes the clarity of generative evaluation with the tractability of discriminative scoring. VL-MDR demonstrates strong accuracy, interpretability, and utility for preference-based alignment. Its multi-dimensional supervision protocol provides a template for future VLM development, supporting both practical reliability improvements and detailed theoretical analysis of model failures. Extensions into non-image modalities, more granular taxonomies, and adversarial robustness are natural next steps.