HV-MMBench: Human-Centric Video Benchmark

• HV-MMBench is a specialized benchmark designed to evaluate human-centric video understanding, emphasizing fine-grained human attributes, interactions, and causal reasoning.
• It features 15 tasks, 1,200 videos, and 8,700 Q&A instances across 50 scenarios and 7 core domains, covering video clips from 10 seconds to 30 minutes with high resolutions.
• The benchmark utilizes multiple question formats to diagnose both perceptual and cognitive capabilities, revealing potential overestimations in closed-form evaluations versus open-ended reasoning.

Searching arXiv for HV-MMBench and closely related benchmark papers to ground the article in current literature. arXiv search query: "HV-MMBench human-centric video understanding benchmark MLLMs" HV-MMBench is a benchmark for evaluating multimodal LLMs on human-centric video understanding: the analysis of videos in which human attributes, actions, interactions, intentions, relationships, and causal structure are central to the task formulation. It was introduced to address a gap in existing video evaluation suites, which, according to its authors, tend to emphasize video generation quality, action recognition, or comparatively shallow understanding, and often rely on single-question paradigms with overly simple evaluation metrics. HV-MMBench is therefore designed as a multi-task, multi-format, multi-domain, and temporally broad benchmark, comprising 15 tasks, 1,200 videos, 8,700 question-answer instances, 50 distinct visual scenarios, and video durations ranging from 10 seconds to 30 minutes (Cai et al., 7 Jul 2025).

1. Scope and motivating problem

HV-MMBench targets a specific failure mode in contemporary multimodal evaluation: strong aggregate performance on generic video benchmarks does not necessarily imply robust understanding of people-centered scenes. Human-centric video understanding requires more than recognition of coarse actions. It also entails fine-grained perception of people, modeling of interpersonal interactions, inference of intentions and motivations, prediction of social relations, and reconstruction of causal chains over time. HV-MMBench is organized around this broader conception of competence (Cai et al., 7 Jul 2025).

The benchmark’s motivation is articulated against three limitations in prior work. First, earlier human-centric benchmarks are described as too narrow in task coverage, with emphasis on generation quality or action recognition rather than the full spectrum from perception to cognition. Second, many existing benchmarks use only multiple-choice questions, which can be partially solved through elimination strategies or language priors rather than grounded video understanding. Third, prior benchmarks often concentrate on short clips and a small number of domains, limiting stress tests of temporal reasoning and cross-scenario generalization. HV-MMBench is constructed as a direct response to those limitations by broadening task families, broadening answer formats, and spanning both short-term and long-term video contexts (Cai et al., 7 Jul 2025).

Within the broader benchmark landscape, HV-MMBench occupies a distinct position. Open-domain video benchmarks such as Video-MME, MVBench, Video-Bench, Koala-36M, and OpenVid-1M are characterized as broad but not specifically human-centric, whereas prior human-centric benchmarks such as HumanVid, OpenHumanVid, and HumanVBench are described as focusing more narrowly on generation or limited understanding subtasks. HV-MMBench instead concentrates on human actions, emotions, identity, intentions, relationships, and causal reasoning, thereby framing human-centered video comprehension as a composite perceptual-cognitive problem rather than a single-label classification problem (Cai et al., 7 Jul 2025).

2. Dataset composition and coverage

The corpus is designed to combine breadth of scenario coverage with variation in temporal scale and visual quality. Videos are drawn from public sources including UltraVideo, OpenHumanVid, and Koala-36M, then filtered and annotated. The resulting benchmark spans 50 human-centric visual scenarios grouped into 7 core domains: daily life, professional activities, social interactions, health and medical management, education and learning, transportation, and cultural entertainment (Cai et al., 7 Jul 2025).

A central design choice is explicit temporal diversity. HV-MMBench includes short clips of around 10 seconds as well as long videos up to 30 minutes, permitting evaluation of instantaneous perception, short-horizon action understanding, long-range event tracking, and extended causal reasoning within a single benchmark. Visual quality is also nontrivial: video resolutions range from 720P to 4K, with most videos above 1080P. This suggests that the benchmark is not confined to low-fidelity web snippets and instead attempts to preserve visually rich human detail where tasks such as attribute recognition and face recognition are materially affected by image quality (Cai et al., 7 Jul 2025).

Component	Specification
Videos	1,200
QA instances	8,700
Scenarios	50
Domains	7 core domains
Duration range	10 seconds to 30 minutes
Resolution range	720P to 4K, with most above 1080P

The scenario design is intended to make evaluation less brittle than domain-specific benchmarks. Human behavior in education, transportation, or medical management may involve different cues, different social structures, and different temporal dependencies. By distributing the benchmark across 50 scenarios and 7 domains, HV-MMBench aims to test whether models generalize across fine-grained context variation rather than merely exploiting repeated patterns from a narrow activity distribution (Cai et al., 7 Jul 2025).

3. Task taxonomy and question formats

HV-MMBench is organized around 15 tasks that span what the authors describe as a spectrum from basic attribute perception to advanced cognitive reasoning. Named perceptual tasks include Basic Attribute Recognition, Face Recognition, and Action Recognition. Named higher-order reasoning tasks include Relation Inference, Intention Inference, and Causal Reasoning. The benchmark description also explicitly refers to emotion recognition, social relationship prediction, and motivation inference as part of its broader design space. The task system is therefore best understood as a hierarchical range from low-level human perception to high-level social-cognitive inference rather than a flat collection of unrelated subtasks (Cai et al., 7 Jul 2025).

The benchmark uses four question-answer paradigms: multiple-choice, fill-in-the-blank, true/false, and open-ended questions. This heterogeneity is one of its defining features. Multiple-choice and true/false items permit standardized scoring and are suitable for recognition-oriented tasks, but the benchmark design treats them as insufficient on their own because they are more vulnerable to guessing, option elimination, and language-prior exploitation. Fill-in-the-blank requires direct answer generation without options, and open-ended questions are used for the most demanding cases, especially causal reasoning, where coherent explanation matters in addition to correctness (Cai et al., 7 Jul 2025).

Format	Role in the benchmark	Main evaluation
Multiple-choice	Recognition and structured reasoning	Accuracy
Fill-in-the-blank	Direct generation without options	Precision@1, Recall@1, F1@1
True/False	Coarse recognition and reasoning	Accuracy
Open-ended	Causal explanation and generative reasoning	Composite score

Task frequencies are intentionally uneven. Basic Attribute Recognition is the largest task with 2,517 question-answer pairs. Intention Inference contains 1,938 pairs, and Relationship Inference 1,399 pairs. Causal Reasoning includes 70 open-ended samples, approximately 7.5% of that task’s total. This distribution indicates that HV-MMBench is not exclusively dominated by simple perception, but neither does it flatten the benchmark into a uniform task inventory. Instead, it reflects a pragmatic weighting of practically important subtasks while preserving a nontrivial presence of difficult reasoning cases (Cai et al., 7 Jul 2025).

A recurrent finding associated with this design is that answer format materially changes what the benchmark measures. Closed-form tasks may reward recognition of plausible options, whereas fill-in-the-blank and open-ended variants more directly probe whether a model can generate grounded human-centered interpretations without answer-set scaffolding. This suggests that HV-MMBench is intended not only as a scorecard, but also as a diagnostic instrument for disentangling apparent reasoning from grounded understanding (Cai et al., 7 Jul 2025).

4. Construction and annotation pipeline

HV-MMBench is built through a three-stage pipeline: video collection and preprocessing, automated question-answer annotation, and manual quality review. In the first stage, candidate videos are collected from public datasets and manually filtered to ensure that they match predefined scenario labels, contain no artificial overlays such as subtitles or black borders, and have resolution above 720P. The final dataset contains 1,200 videos after this selection process (Cai et al., 7 Jul 2025).

The second stage decomposes automated annotation into attribute labeling and question-answer generation. For attribute labeling, a two-stage model pipeline is used: Qwen2.5-VL-72B first generates detailed video captions from raw video, and Qwen2.5-72B then infers which task-specific attributes are present. This decoupling is explicitly justified as a way to improve interpretability and reduce error propagation, with the video model responsible for captioning and the LLM responsible for mapping textual descriptions to task assignments (Cai et al., 7 Jul 2025).

For question-answer generation, the benchmark uses manually designed templates. For each secondary task, the authors designed 5–10 templates. Relevant attributes trigger question generation, and localization cues such as “the man wearing a blue shirt” are inserted when needed. Qwen2.5-VL-72B then generates one correct answer and at least three distractors. These distractors are intentionally difficult and may involve slight attribute perturbations, temporal perturbations, or sequence-level changes. The stated purpose is to make the benchmark more diagnostic and less susceptible to trivial pattern matching (Cai et al., 7 Jul 2025).

The third stage is human quality control. Automated review filters out questions whose answer is implicitly contained in the question itself or that do not genuinely require video understanding. Manual review is conducted by two independent reviewers. For multiple-choice and true/false items, both reviewers vote, and a third expert adjudicates in cases of disagreement. For fill-in-the-blank and open-ended questions, candidate answers are checked carefully, and items are retained only if both review rounds validate them. This protocol is intended to preserve both reliability and diversity while constraining annotation noise (Cai et al., 7 Jul 2025).

5. Evaluation protocol and metrics

HV-MMBench uses format-dependent metrics rather than a single universal score. For multiple-choice and true/false questions, the benchmark uses standard Accuracy. For fill-in-the-blank tasks, where multiple answers may be acceptable, it uses Precision@1, Recall@1, and F1@1. This choice reflects the fact that generative answers in human-centric video understanding are often semantically correct without being strictly identical to one canonical string, especially for descriptive attributes or social interpretations (Cai et al., 7 Jul 2025).

For open-ended causal reasoning, HV-MMBench introduces a composite score intended to combine lexical overlap, structural fidelity, and semantic coherence:

$$Score = \alpha \cdot Score_F + \beta \cdot Score_O + \gamma \cdot Score^{\text{norm}}_G$$

with default weights $\alpha = 0.5$, $\beta = 0.3$, and $\gamma = 0.5$ (Cai et al., 7 Jul 2025).

The published equation is reported with a minor typographical issue in the paper’s formatting, but the intended structure is described clearly. The first component, $Score_F$, is a fuzzy step-wise F1 score that measures event-level overlap between predicted and reference causal chains using fuzzy token matching. The second, $Score_O$, is a structural consistency score based on the longest common subsequence:

$Score_O = \frac{\text{LCS}(P,G)}{|G|}$

where $P$ denotes the predicted chain and $G$ the gold chain. The third, $Score_G$, is a semantic coherence score assigned by Qwen2.5-72B on a 0-to-5 scale, where 0 denotes no clear causal link and 5 denotes perfect semantic alignment with logically sound event sequencing; the score is normalized by dividing by 5 (Cai et al., 7 Jul 2025).

This composite scheme is significant because exact-match evaluation is poorly suited to open-ended causal explanations. By combining fuzzy event overlap, event-order consistency, and a semantic judge, HV-MMBench attempts to distinguish between partially correct causal reconstructions, structurally disordered narratives, and semantically coherent explanations that nevertheless diverge lexically from the reference. A plausible implication is that the benchmark seeks to evaluate reasoning traces rather than only final answer tokens, especially in its most demanding causal tasks (Cai et al., 7 Jul 2025).

6. Empirical findings and benchmark implications

HV-MMBench evaluates several open-source multimodal LLM families, including the Qwen2.5-VL series, InternVL2.5 series, LLaVA-OneVision, LLaVA-Video, and VideoLLaMA2. The reported results show a marked divergence between performance on closed-form formats and performance on generative formats. On multiple-choice and true/false tasks, strong models achieve high scores even on higher-level reasoning tasks. Qwen2-VL-7B reaches 96.18% on intention inference in multiple-choice format, and Qwen2.5-VL-32B reaches 94.53% on causal reasoning in multiple-choice format. However, the same evaluation suite also reports weak fine-grained perceptual performance: Qwen2-VL-7B obtains only 42.99% multiple-choice accuracy on Face Recognition (Cai et al., 7 Jul 2025).

The contrast becomes stronger on fill-in-the-blank tasks. Models perform relatively better on Basic Attribute Recognition but degrade sharply on Action Recognition, Intention Inference, Relationship Inference, and especially Causal Reasoning. Qwen2.5-VL-32B, despite strong closed-form performance, is reported at only 0.33 / 0.03 / 0.07 on causal reasoning in fill-in-the-blank format. The benchmark interprets this discrepancy as evidence that multiple-choice and true/false performance can overestimate actual reasoning ability because models may rely on option recognition or pretrained language priors rather than genuine generation of grounded causal explanations (Cai et al., 7 Jul 2025).

Open-ended causal reasoning is the most difficult regime. Across models, lexical overlap remains low, with $\alpha = 0.5$0 roughly in the range 0.14–0.22, and structural alignment is also limited. The best model on the reported open-ended setting is Qwen2.5-VL-32B, with $\alpha = 0.5$1, $\alpha = 0.5$2, $\alpha = 0.5$3, and a final score of 0.59. Qwen2.5-VL-7B follows with a final score of 0.57, while all other models remain below 0.50 overall. This indicates that some models can produce semantically plausible causal narratives, but still struggle to recover the exact event chain or preserve reference structure reliably (Cai et al., 7 Jul 2025).

Several general conclusions follow from these findings. First, current MLLMs are substantially stronger in closed-form evaluation than in open-ended reasoning. Second, fine-grained human perception remains a bottleneck, especially for Face Recognition and subtle attribute analysis. Third, temporal and causal modeling remain weak: even when a model can choose the correct answer from a set of options, it often cannot generate a coherent causal account of the same event sequence. Fourth, the benchmark reveals an apparently paradoxical pattern in which models sometimes score better on high-level multiple-choice reasoning tasks than on low-level perceptual ones. HV-MMBench interprets this not as evidence that causal reasoning is solved, but as a sign that structured formats may inflate measured reasoning competence (Cai et al., 7 Jul 2025).

In the contemporary benchmark ecosystem, HV-MMBench can be situated alongside other efforts to make multimodal evaluation more diagnostic and less vulnerable to superficial success. MMBench-Live emphasizes continuously evolving, distribution-consistent multimodal evaluation to reduce temporal staleness and contamination (Liu et al., 2 Jul 2026), while MMBench-GUI introduces hierarchical capability decomposition and efficiency-sensitive online metrics for GUI agents (Wang et al., 25 Jul 2025). HV-MMBench is aligned with this broader trend in its own domain: it replaces one-format, one-metric testing with task diversity, answer-format diversity, temporal diversity, and explicit causal evaluation, thereby turning human-centric video understanding into a benchmarked research program rather than a single aggregate score (Cai et al., 7 Jul 2025).