MVPBench: Diverse Multi-Modal Evaluation Suites

• MVPBench is a polysemous term used for multiple distinct benchmarks that assess tasks ranging from population-aware value alignment to multi-level visual perception.
• It encompasses evaluation methods such as graph-based multi-path reasoning, numerical decision protocols for video analysis, and diagnostic tests on natural versus manipulated images.
• The benchmarks emphasize structured performance metrics over single-number scores and require disambiguation via modality, task description, and arXiv identifiers.

MVPBench denotes several distinct benchmarks in recent arXiv literature rather than a single canonical resource. The name has been used for at least four unrelated evaluation suites: a benchmark and fine-tuning framework for aligning LLMs with diverse human values across 75 countries; a graph-based benchmark for multi-path visual physical chain-of-thought; a multi-video perception evaluation benchmark for multi-modal video understanding; and a multi-level visual perception benchmark for large vision–LLMs on natural and manipulated images (Liang et al., 9 Sep 2025, Dong et al., 30 May 2025, Bai et al., 24 Mar 2026, Li et al., 2024). The shared acronym therefore requires disambiguation by title, modality, and arXiv identifier.

1. Nomenclature and scope

In current usage, “MVPBench” is polysemous. Each work defines a different task family, dataset structure, and metric suite, and the overlap is nominal rather than methodological.

Benchmark name	Target capability	Salient composition
MVPBench	Population-aware value alignment	24,020 instances, ~1,500 users, 75 countries, 7 value dimensions
MVPBench	Visual physical reasoning with multi-path CoT	1,211 examples, 4,701 images, graph-based CoT metrics
MVPBench	Multi-video perception	2,774 videos, 5,050 QA instances, 14 subtasks
MVP-Bench	Multi-level visual perception	530 natural–manipulated image pairs, 1,872 questions

The value-alignment benchmark defines MVPBench as a “Multi-Value Preference Benchmark”; the visual-physics benchmark uses MVPBench for “Multi-path Visual Physics Benchmark”; the video benchmark expands it to “Multi-Video Perception Evaluation Benchmark”; and the 2024 visual-perception paper uses the hyphenated form MVP-Bench for “Multi-level Visual Perception Benchmark” (Liang et al., 9 Sep 2025, Dong et al., 30 May 2025, Bai et al., 24 Mar 2026, Li et al., 2024). A separate paper on multi-view generation explicitly distinguishes “MVGBench” from “MVPBench,” which further indicates that citation by arXiv id is necessary to avoid ambiguity (Xie et al., 11 Jun 2025).

2. MVPBench as a benchmark for diverse human values

In "MVPBench: A Benchmark and Fine-Tuning Framework for Aligning LLMs with Diverse Human Values" (Liang et al., 9 Sep 2025), MVPBench is a benchmark and fine-tuning framework for evaluating and improving alignment with multi-dimensional human value preferences across countries and demographic groups. Its central construct is population-aware alignment: alignment is conditioned on which user profile the model is interacting with rather than treated as a universal notion of helpfulness or harmlessness.

The dataset is built from PRISM’s stated_prefs and extended through GPT-4o generation plus human validation. It contains 24,020 instances, about 1,500 users, 75 countries, and seven value dimensions: creativity, fluency, factuality, diversity, safety, personalization, and helpfulness. Ratings are mapped into discrete preferences with threshold rules: rating $\ge 80$ denotes high preference, and rating $\le 60$ denotes low preference. Each final instance contains structured user profile metadata, value preferences, a personalized question, and two contrastive answers: answer_w, aligned with the user’s value preferences, and answer_l, deliberately misaligned.

The construction pipeline has three stages: value preference mapping, personalized Q&A generation, and user profile integration. Detailed profile attributes include age, gender, educational background, employment status, English proficiency, country of birth, and marital status. This structure supports supervised learning, pairwise preference learning, and evaluation of generated model answers against user-conditioned value preferences.

Evaluation proceeds in two stages. In Stage 1, the model under test receives a profile and question and produces a free-form model_answer. In Stage 2, a judgment model receives the user profile, value preferences, question, reference answer answer_w, and the model answer, then outputs a binary alignment label, “Yes” or “No.” The primary metric is Preference Alignment Accuracy, defined as the fraction of instances judged aligned. The paper also reports disparity through per-country and per-demographic-group PAA rather than introducing a separate fairness metric.

The empirical analysis shows substantial cross-country and cross-demographic variation. Doubao-1.5-Pro is reported as high and stable across many countries, with country examples including Ireland at 94.2%, Romania at 96.2%, South Korea at 94.7%, and Argentina at 97.2%. GPT-4o shows strong overall alignment but marked regional variability, including Russia at 90.5%, India at 87.2%, Turkey at 97.2%, Vietnam at 100%, Lithuania at 100%, Brazil at 5.6%, and Honduras at 0%. DeepSeek-v3 is likewise uneven, with Romania at 98.1%, China at 96.8%, Indonesia at 100%, Netherlands at 35.8%, Kenya at 11.1%, and Brazil at 0%. The paper interprets these results as evidence of large cross-national disparities.

The same work extends MVPBench into a training resource. Base models LLaMA-2-7b-hf, LLaMA-2-13b-hf, LLaMA-2-7b-chat, and LLaMA-2-13b-chat are fine-tuned with LoRA and DPO. The reported LoRA configuration uses rank $r=16$, scaling factor $\alpha=32$, dropout 0.05, and applies updates to the attention mechanism’s query and value projections; the DPO experiments use $\beta=0.1$, three epochs, learning rate $5 \times 10^{-5}$, bf16 precision, per-GPU batch size 2, gradient accumulation 4, max input length 512, and max output length 1024. Beyond PAA, the paper defines Optimized Preference Alignment and Semantic Preference Matching Rate. Reported in-domain gains are large: for example, Llama2-13b improves from 47.91 to 99.62 in MVP_test OPA and from 5.95 to 31.12 in MVP_test SPMR, while out-of-domain UF-P-4 OPA rises from 51.58 to 56.01.

3. MVPBench for graph-based evaluation of multi-path visual physical CoT

In "Seeing is Not Reasoning: MVPBench for Graph-based Evaluation of Multi-path Visual Physical CoT" (Dong et al., 30 May 2025), MVPBench is a benchmark for multimodal LLMs on visual physical reasoning, with explicit emphasis on visual chain-of-thought and multi-path reasoning. The benchmark is designed around real-world physical scenes and exam-style problems, and each example requires both a final answer and a coherent reasoning path grounded in evolving visual evidence.

The dataset contains 1,211 examples and 4,701 images, with 4,688 unique images. It is organized into four major categories. Physics Experiments contributes 400 questions from real-world experiment videos cut into 3–5 key frames; Physics Problems contributes 311 questions from Chinese Gaokao, International Physics Olympiad, mock exams, and PhysReason-mini; Spatial Relations contributes 400 questions over direction judgment, distance estimation, first-view transformation, and topological relation judgment; and Dynamic Prediction contributes 100 questions adapted from PhysBench video data. Each sample is annotated with key reasoning steps and one or more reasoning chains, and the average statistics reported are 28.01 words per question, 2.93 answer steps, 3.90 Image-CoTs per sample, and 2.67 reasoning paths per sample.

A central design feature is the multi-path reasoning graph. Annotated chains are converted into a directed acyclic graph $G=(V,E)$, where nodes are key reasoning steps and edges denote logical dependencies. The evaluation then aligns model-generated reasoning steps to graph nodes and scores both the correctness of the steps and the structural validity of their order.

The metric suite has three layers. CoT Quality includes Step Accuracy Score, CoT Reasoning Score, and Key Step Coverage. CoT Diversity includes Path Validity Rate, Path Coverage Score, and CoT Match Score, with adjusted variants that account for the number of predicted paths relative to the number of reference paths. CoT Efficiency includes Step Relevance Score and Reflection Validity Rate. The composite quality score is defined as

$$\text{CRS} = \alpha \cdot \text{SAS} + (1-\alpha)\cdot \text{Correct}(s_A),$$

with $\alpha=0.7$ by default. The core structural metric is

$$\text{CMS} = 2 \cdot \frac{\text{PVR}\cdot \text{PCS}}{\text{PVR}+\text{PCS}}.$$

The reported results show that multi-image input nearly always improves performance. For GPT-4o, moving from single-image to multi-image yields gains of +20.30 in SAS, +14.75 in KSC, and +21.41 in CRS; OpenAI o3 gains +15.87 in SAS and +15.83 in CRS. Among closed-source models, OpenAI o3 has the strongest CoT quality and efficiency. Among open-source models, Qwen2.5-VL-72B has the highest CoT diversity, while QVQ-72B has the best quality but is described as verbose and less robust. Human performance on diversity is reported at roughly 95–99% across categories, whereas model PVR and PCS values are generally in the 60–80% range.

The paper’s error analysis separates perception failures from reasoning failures and finds that reasoning errors dominate in physics subsets, whereas Spatial Relations exhibits high perception error. It also argues that RL-based post-training often harms visual-physical reasoning. InternVL2.5-78B-MPO underperforms the base InternVL2.5-78B on several subsets, and analogous degradations are reported for R1-VL-2B relative to Qwen2VL-2B and for MM-Eureka-7B relative to Qwen2.5VL-7B. This is presented as evidence that current post-training objectives can improve conversational behavior while degrading multimodal grounding and structured CoT coherence.

4. MVPBench as a multi-video perception benchmark

In "MVPBench: A Multi-Video Perception Evaluation Benchmark for Multi-Modal Video Understanding" (Bai et al., 24 Mar 2026), MVPBench is an evaluation suite for multi-video understanding rather than single-image or single-video reasoning. Each benchmark instance presents multiple video clips and a textual prompt, and the model must return only a number or sequence of numbers.

The benchmark comprises 2,774 videos, 5,050 QA instances, and 14 subtasks grouped into five domains: Temporal Segment Splicing, Content Assessment, Video Quality Assessment, Video Logic Inference, and Similar Video Pairing. The subtasks include caption-guided temporal splicing, caption-free temporal splicing, daily life action evaluation, professional action evaluation, weather condition evaluation, video clarity evaluation, video brightness evaluation, video forensic detection, multiview visual perception pairing, common sense judgment of physical laws, gait recognition matching, cinematographic style matching, Olympic sports matching, and dance style matching. Data are drawn from YouCook2, EPIC-Skills, AQA-7, CVQAD, SDSD, PEV, CASIA Gait, manually collected web clips, and synthetic videos generated by Kling and Ying from GPT-4o-produced captions.

The evaluation protocol standardizes all tasks as numerical decision problems. A GPT-4-turbo-based extractor parses the model output into a final numerical choice. To compare tasks with different answer-space sizes, the paper defines a chance level $\le 60$0 for subtask $\le 60$1 with $\le 60$2 response options and a normalized proficiency score

$\le 60$3

where $\le 60$4 is the observed accuracy of model $\le 60$5 on subtask $\le 60$6. An overall score is then formed as a weighted average, with weights proportional to $\le 60$7.

The benchmark includes human annotation for reference. Twenty evaluators each answer $\le 60$8 questions, with 10% duplicated questions for inter-evaluator agreement. The paper reports 89.7% inter-evaluator agreement and 88.89% overall human accuracy. There is no train/validation/test split; the benchmark is test-only.

The main results indicate that current multi-modal models are weak at direct multi-video reasoning. Human performance is 88.89, whereas GPT-4o attains 31.10 overall, Gemini-2.5-Flash 29.33, and the strongest open-source model, LLaVA-OneVision-Qwen2-72B, 25.79. Many models have negative overall normalized proficiency, meaning below-random performance after normalization. Reported subtask trends include about 10.39% accuracy for caption-guided temporal splicing, about $\le 60$9 normalized for caption-free temporal splicing, about 18.43% for daily action evaluation, about 7.44% for professional action evaluation, about 11.98% for clarity, about 9.73% for brightness, about 22.40% for forensic detection, about 37.84% for multiview pairing, about 11.54% for physical-law judgment, about 13.06% for gait, about 11.70% for cinematographic style, about 9.48% for Olympic sports, and about 1.46% for dance.

A notable diagnostic result is strong positional bias. For Qwen2.5-VL-72B, first-option selection frequencies exceed random expectation across multiple subtasks, including 21.13% versus 10.56% for caption-guided temporal splicing, 28.30% versus 16.67% for caption-free temporal splicing, 49.00% versus 33.33% for gait, 68.00% versus 33.33% for cinematographic style, 78.00% versus 50.00% for Olympic sports, and 46.25% versus 33.33% for dance. The paper also compares direct multi-video inference with two single-video approximations and finds that direct multi-video processing is substantially better aligned with human judgments.

5. MVP-Bench as a benchmark for multi-level visual perception

In "MVP-Bench: Can Large Vision--LLMs Conduct Multi-level Visual Perception Like Humans?" (Li et al., 2024), MVP-Bench is a vision–language benchmark for hierarchical visual perception, spanning low-level details and high-level semantic and social interpretations. The benchmark is built around natural–manipulated image pairs in human-centric scenes, with the aim of measuring whether models maintain coherent perception across levels and under subtle edits.

The dataset contains 530 natural–manipulated image pairs, or 1,060 images in total, and 1,872 questions. Natural images come from the EMU dataset, while manipulated images are synthetic edits of the natural images. High-level perception is organized into five categories—Behaviour, Role, Identity, Emotion, and Scenario—and low-level perception into 13 categories linked to those high-level interpretations. Of the 1,872 questions, 1,105 are high-level and 767 are low-level; the high-level set contains 460 Yes/No, 418 single-image multiple choice, and 227 cross-image multiple-choice questions, while the low-level set contains 540 Yes/No and 227 cross-image multiple-choice questions.

The construction pipeline has three stages. First, ChatGPT with Chain-of-Thought prompting proposes a high-level semantic change and the corresponding low-level feature edit. Second, manipulated images are produced using one of three edit types: partial component substitution, overall background substitution, or direct alteration. The implementation combines Shikra for grounding the main subject and target object, SAM for subject masks, Stable-Diffusion-Inpaint for localized or background edits, and InstructPix2Pix for direct alteration. Third, ChatGPT generates Yes/No, single-image multiple-choice, and cross-image multiple-choice questions aligned to the selected low-level and high-level categories. Two authors manually review all 3,205 initially generated questions, and a question is retained only if both accept it.

Evaluation distinguishes single-image and cross-image settings. For Yes/No questions, the paper reports question-level accuracy, image-level accuracy, and pair-level multi-question accuracy. For multiple-choice questions, it reports both VanillaEval and CircularEval, the latter rotating options to reduce position bias. The cross-image setting is especially important because the same question can be asked of both the natural and manipulated image, with opposite ground-truth answers.

The main empirical finding is a persistent high-level deficit. Using single-image Yes/No qAcc, GPT-4o reaches 56.09% at the high level and 74.44% at the low level. In cross-image MCQ, GPT-4o reaches 74.01% at low level but only 34.80% at high level. The paper states that all evaluated LVLMs perform worse on manipulated images than on natural images. In low-level Yes/No iAcc, GPT-4o scores 76.92% on natural images and 48.52% on manipulated images; GPT-4V shows a 40.3-point gap between natural and manipulated images. The study also reports that MiniCPM-V-2-3B performs strongly on high-level perception among open-source models, while DeepSeek-VL-7B is strongest on low-level tasks among open-source models.

The qualitative analysis identifies several recurrent failure modes. GPT-4V often refuses to make high-level inferences from strong low-level cues, such as inferring that a person in a doctor’s coat is a doctor. Other models miss subtle but decisive objects, over-rely on dominant scene priors, or reinterpret violent scenes as staged performances. The paper treats these behaviors as evidence that current LVLMs do not yet conduct multi-level visual perception in a human-like manner, especially on manipulated images where small low-level changes should flip high-level interpretation.

6. Comparative interpretation and research significance

Taken together, the various benchmarks named MVPBench define four very different research problems: user-conditioned value alignment, visually grounded physical reasoning, joint reasoning over multiple videos, and hierarchical visual perception over natural–manipulated image pairs (Liang et al., 9 Sep 2025, Dong et al., 30 May 2025, Bai et al., 24 Mar 2026, Li et al., 2024). Their commonality lies less in content than in evaluation philosophy. Each benchmark is explicitly diagnostic: it is designed to expose a capability that standard aggregate leaderboards tend to blur.

The value-alignment MVPBench evaluates whether responses track explicit user profiles and multi-dimensional preferences. The visual-physics MVPBench evaluates whether a reasoning trajectory is structurally valid relative to a graph of physical logic. The multi-video MVPBench evaluates whether a model can integrate several videos in a single decision problem rather than approximate the task through per-video captioning or scoring. MVP-Bench for visual perception evaluates whether low-level recognition and high-level interpretation remain consistent when small visual edits induce large semantic changes. In that sense, all four benchmarks oppose single-number notions of capability and replace them with structured performance criteria.

A plausible implication is that references to “MVPBench” without further qualification are underspecified. For technical discussion, replication, or benchmarking claims, the relevant identifier is not the acronym alone but the exact title and arXiv id. The term therefore functions less as the name of a unified benchmark family than as a reused label attached to multiple independent benchmark programs.