Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following (2511.21662v1)

Abstract: Large multimodal models (LMMs) are increasingly adopted as judges in multimodal evaluation systems due to their strong instruction following and consistency with human preferences. However, their ability to follow diverse, fine-grained evaluation criteria remains underexplored. We develop Multi-Crit, a benchmark for evaluating multimodal judges on their capacity to follow pluralistic criteria and produce reliable criterion-level judgments. Covering both open-ended generation and verifiable reasoning tasks, Multi-Crit is built through a rigorous data curation pipeline that gathers challenging response pairs with multi-criterion human annotations. It further introduces three novel metrics for systematically assessing pluralistic adherence, criterion-switching flexibility, and the ability to recognize criterion-level preference conflicts. Comprehensive analysis of 25 LMMs reveals that 1) proprietary models still struggle to maintain consistent adherence to pluralistic criteria--especially in open-ended evaluation; 2) open-source models lag further behind in flexibly following diverse criteria; and 3) critic fine-tuning with holistic judgment signals enhances visual grounding but fails to generalize to pluralistic criterion-level judgment. Additional analyses on reasoning fine-tuning, test-time scaling, and boundary consistency between open-source and proprietary models further probe the limits of current multimodal judges. As a pioneering study, Multi-Crit lays the foundation for building reliable and steerable multimodal AI evaluation.

• The paper introduces Multi-Crit, a benchmark that evaluates large multimodal model judges on five human-defined criteria, revealing widespread conflicts and trade-offs.
• It employs a rigorous data pipeline to curate and annotate diverse prompt-response pairs from multiple real-world and reasoning tasks.
• Experiments show that while proprietary models excel in certain criteria, both proprietary and open-source models struggle with consistent pluralistic accuracy and trade-off sensitivity.

Multi-Crit: Evaluating Multimodal Judges on Pluralistic Criteria-Following

Motivation and Benchmark Design

The proliferation of large multimodal models (LMMs) has driven their adoption as automated judges for evaluating AI-generated multimodal responses. However, existing benchmarks overwhelmingly provide only a single scalar or pairwise preference label per response pair, failing to capture the pluralistic, criterion-specific nature of human evaluation. The Multi-Crit benchmark directly addresses this limitation by introducing systematic evaluation of LMM judges under pluralistic, fine-grained, and potentially conflicting multi-criterion settings.

Multi-Crit encompasses both open-ended generation and verifiable reasoning tasks, built through a rigorous pipeline that collects challenging prompt–response instances, ensures annotation diversity, and retains only those pairs with nuanced, nontrivial trade-offs (Figure 1).

Figure 1: Multi-Crit’s data pipeline: diverse multimodal prompts, paired LMM responses, and multi-criterion human annotations capturing intra-instance conflicts.

Comprehensive prompt/response generation incorporates content from both real-world scenarios (captioning, visual conversations, instructional queries) and complex reasoning domains (mathematics, STEM tasks, code, visual puzzle solving), drawing from 8 datasets and 11 top LMM families, including both proprietary (e.g., GPT, Gemini, Claude) and advanced open-source models (e.g., InternVL, Qwen, LLaVA). The response pair curation process involves length normalization, correctness filtering for reasoning, and ensemble-based difficulty screening to yield balanced, challenging samples.

Multi-Criterion Human Annotation

Rather than relying on a single preference per comparison, Multi-Crit provides detailed human annotations covering five evaluation criteria per task type. For open-ended tasks, dimensions include Completeness and Coverage, Visual Grounding, Factuality/No Hallucination, Creativity and Expressiveness, and Clarity and Coherence. Reasoning tasks are annotated for Visual Grounding, Logic Coherence, Factuality, Reflection and Exploration, and Conciseness/Efficiency.

Annotations are provided by trained PhD annotators, with rigorous calibration, adjudication, and verified inter-annotator agreement (Cohen's κ of 0.718 for open-ended, 0.805 for reasoning). This annotation approach directly exposes empirical preference conflicts: over 68% of open-ended and 86% of reasoning prompts exhibit at least one criterion-level preference conflict, illustrating that human evaluations are deeply multidimensional and often non-congruent (Figure 2).

Figure 2: Distribution of prompt sources (left) and evaluation criteria (right) in Multi-Crit.

Pluralistic Judge Evaluation: Metrics and Protocol

To characterize judge performance beyond one-dimensional label accuracy, Multi-Crit introduces three core evaluation metrics:

1. Pluralistic Accuracy (PAcc): The proportion of instances for which the judge correctly predicts human-preferred responses for every applicable criterion.
2. Trade-off Sensitivity (TOS): The fraction of conflict cases in which the judge assigns different preferences across conflicting criteria—in other words, the judge’s ability to recognize that not all criteria imply the same choice.
3. Conflict Matching Rate (CMR): For all pairs of conflicting criteria within a prompt, the proportion in which the judge matches the human-preferred response for both criteria.

This framework allows granular analysis of where and why judges fail or succeed: are errors due to generalized failures, inability to recognize nuanced trade-offs, or the tendency to "flatten" their judgments across criteria?

Experimental Results and Analysis

Multi-Crit evaluates 25 LMM judges, including proprietary (GPT-4o, GPT-5, Gemini-2.5, Claude, o4-mini), leading open-source LMMs (InternVL, Qwen, GLM, MiniCPM, MiMo, Molmo), and several fine-tuned critic/reward models (LLaVA-Critic, R1-Reward, UnifiedReward). Models are assessed strictly criterion-wise, using curated prompt templates and standardized evaluation.

Key findings include:

• Low Pluralistic Accuracy: Even leading models, e.g., o4-mini, achieve only 32.8% (open-ended) and 53.2% (reasoning) PAcc—highlighting that reliably following all evaluated criteria within a sample remains an open challenge. Open-source models lag further behind (Figure 3).

  Figure 3: Average performance across each criterion. Model strengths are not uniform, and all models perform better in reasoning than in open-ended tasks.

• No Ubiquitous Model Dominance: Individual models show strengths on specific criteria but not across the board. For example, proprietary models lead overall but differ in their strong points across Logic, Hallucination, and Expressiveness.
• Open-Source Models’ Limitation: There is a notable performance gap in pluralistic evaluation (both CMR and TOS) between proprietary and open-source LMMs, apart from pure single-criterion accuracy, demonstrating that aggregation of multi-criterion consistency is more difficult than optimizing for average per-criterion correctness.
• Fine-Tuned Critics’ Narrow Effect: Reward model or judge-specific fine-tuning on holistic preference data (e.g., LLaVA-Critic, R1-Reward) enhances visual grounding but yields inconsistent or no improvement for pluralistic, conflict-rich settings. Fine-tuned models tend to reinforce dominant criteria (often visual grounding), failing to generalize to divergent, conflicting multi-criterion judgments.
• Reasoning-Specific Fine-Tuning: RLHF/RLVR training on reasoning datasets improves stepwise accuracy but can diminish sensitivity to trade-offs and make models more rigid, reducing the ability to reflect human-level pluralism (Figure 4).

  Figure 4: RL-tuned reasoning models on Multi-Crit reasoning split: gains in base accuracy, but reduced sensitivity to trade-offs and conflicts.

• Test-Time Scaling (Self-Consistency): Only top-performing LMMs such as o4-mini and GPT-4o show robust improvements from ensembling/judgment scaling, and effects are not uniform across models or settings (Figure 5).

  Figure 5: Test-time scaling (majority voting) shows most consistent gains for top models in pluralistic accuracy.

• Criterion-Level Agreement with Human Consistency: For proprietary models, criterion-wise accuracy correlates with human inter-annotator agreement (r=0.73), indicating alignment with evaluators’ consistency boundaries; open-source models show a weaker, less reliable relationship (r=0.36) (Figure 6).

  Figure 6: Criterion-level judge-human top-accuracy correlation. Proprietary models align closely with human judgment in achievable consistency.

Characterizing Human Pluralism and Conflict

Analysis of annotation data confirms that pluralistic conflicts and trade-offs are ubiquitous in non-trivial multimodal generation. Certain criterion pairs (e.g., Completeness vs. No Hallucination, Expressiveness vs. Clarity) have high conflict frequency (Figures 7 and 8). This underscores the necessity of criterion-aware, multi-dimensional evaluation protocols for both research and deployment contexts.

Figure 7: Correlation of criterion-level human preferences for prompts with intra-sample preference conflicts; substantial off-diagonal correlation reflects pluralistic trade-offs.

Figure 8: Frequency of conflicting criterion pairs in open-ended (left) and reasoning (right) splits: no criterion pair is immune to conflict.

Qualitative examples (Figure 9) illustrate the diversity and complexity of the dataset, spanning story generation, image analysis, math, and scientific reasoning, with nuanced and sometimes contradictory strengths across model responses under different criteria.

Figure 9: Example prompts in open-ended (top) and reasoning (bottom) splits; detailed and challenging annotations highlight real-world pluralism.

Practical and Theoretical Implications

Multi-Crit exposes substantial limitations in current multimodal judge models regarding reliability, steerability, and fidelity to human multi-criterion annotation. Practically, this signals that scaling LMMs or fine-tuning on aggregate preference data is necessary but insufficient for robust deployment in evaluation, reward modeling, or safe alignment.

From a training protocol perspective, findings support the argument that criterion-separated, fine-grained, and conflict-rich supervision is critical—not only aggregate or scalar preferences—if models are to become truly steerable and human-compatible judges. The plateauing of open-source model performance in pluralistic consistency, despite improvement in per-criterion accuracy, identifies annotation quality and coverage as substantial bottlenecks.

For future developments, three directions are persuasive:

• Pluralistic and criterion-structured reward modeling: Integrate joint and criterion-wise supervision in RM and critic pipelines, possibly with explicit conflict annotation.
• Interpretability in judge decisions: Criterion-aware justification and attribution mechanisms, which could assist users and developers in understanding model blind spots.
• Toward superhuman AI judgment: As leading proprietary models’ criterion-level consistency approaches human annotator agreement, research must examine to what extent further scaling or curriculum design can break through inherent human subjectivity bounds.

Conclusion

Multi-Crit establishes the new paradigm for evaluating multimodal judges—pluralistic, criterion-level, and conflict-driven—demonstrating systematic gaps between current LMMs and reliable, steerable, human-aligned evaluation. These findings motivate both methodological reformulation of judge training/evaluation and the next generation of multimodal supervision resources. The benchmark serves as both a measuring stick and a catalyst for progress toward genuinely pluralistic AI feedback and alignment (2511.21662).