Dr.V-Bench: Video Hallucination Benchmark

• Dr.V-Bench is a benchmark dataset that diagnoses video model hallucinations by decomposing errors into perceptive, temporal, and cognitive levels.
• It uses 10,000 annotated QA instances from nearly 5,000 videos across diverse domains to evaluate spatial-temporal grounding and answer consistency.
• The hierarchical design and robust evaluation protocols enable precise root-cause analysis, guiding improvements in visual and temporal reasoning models.

Dr.V-Bench is a benchmark dataset for diagnosing hallucination in large video models through fine-grained spatial-temporal grounding. It was introduced as one of two key components of the broader Dr.V framework, alongside the satellite video agent Dr.V-Agent, and is organized around a hierarchical decomposition of hallucination into perceptive, temporal, and cognitive levels. The benchmark contains 10,000 annotated QA instances drawn from 4,974 unique videos, spans three task formats, and is designed to evaluate whether model outputs remain consistent with the underlying video evidence rather than merely producing plausible text (Luo et al., 15 Sep 2025).

1. Concept and scope

Dr.V-Bench targets a specific failure mode in video understanding systems: hallucinations that produce content conflicting with the input videos. Within the Dr.V framework, the benchmark serves as the evaluative substrate, while Dr.V-Agent applies a step-by-step diagnostic pipeline that uses fine-grained spatial-temporal grounding at the perceptive and temporal levels, followed by cognitive-level reasoning. This coupling is intended to mirror human-like video comprehension and to make hallucination diagnosis more interpretable and reliable (Luo et al., 15 Sep 2025).

The benchmark is explicitly hierarchical. Its 14 fine-grained hallucination types are organized into three levels—perceptive, temporal, and cognitive—so that errors can be localized to different strata of video understanding rather than collapsed into a single end-task score. This organization matters because a model may recognize visible entities while failing to track temporal order, or may preserve short-range temporal consistency while still producing unsupported higher-level inferences.

2. Dataset composition and coverage

Dr.V-Bench contains 10,000 annotated QA instances from 4,974 videos. The videos range from 0 to 600 seconds in duration, with an average of approximately 120 seconds, and are sampled at the original frame rates of their source datasets. The collection aggregates material from 15 public video collections spanning approximately 50 domains, including ActivityNet-QA, NExT-QA, Causal-VidQA, CLEVRER, YouCook2, TempCompass, MMWorld, MSR-VTT, and VATEX. At the content level, it covers 18 high-level domains such as daily life, sports, cooking, scripted scenes, and animation, with explicit emphasis on complex scenarios including multiple scene cuts, overlapping actions, and long-range temporal dependencies, which account for approximately 25% of videos (Luo et al., 15 Sep 2025).

The task inventory is divided across multiple-choice, yes/no, and caption-generation formats.

QA format	Instances	Notes
MCQ	6,000	Includes object 720, number 300, knowledge-based explanation 360
Yes/No	3,000	Binary QA
Caption	1,000	Caption-generation QA items

The consolidated summary also reports a video-level split of training 70% (approximately 3,482 videos, 7,000 instances), validation 15% (approximately 746 videos, 1,500 instances), and test 15% (approximately 746 videos, 1,500 instances), while noting that this split is not explicitly in the main paper. Hallucination-type coverage is described as roughly uniform across the 14 types, with each type having 150 to 900 examples depending on task format; the smallest type is reported as “Sequence” in captions, with 20 examples per domain.

3. Annotation schema and construction protocol

The core annotation unit in Dr.V-Bench is a spatio-temporal segment linking an object or event label to a temporal interval and spatial evidence. For each target object, annotators record a start frame and end frame together with bounding boxes on all key frames. Coordinates are normalized to the $[0,1]$ range relative to $(x_{\min}, y_{\min}, x_{\max}, y_{\max})$, and annotations are stored in JSON. Temporal indices correspond to raw video frames, and timestamps may be added as $$\text{start\_time} = \text{start\_frame} / \text{fps}$$ (Luo et al., 15 Sep 2025).

The annotation pipeline proceeds in five steps: extract target objects or events from the QA; annotate start and end frames; select key frames that aid in answering; draw bounding boxes on every key frame; and perform cross-validation. The start and end boundaries are defined as the first and last frame where at least 30% of the object’s contour is visible. Annotators are instructed that objects must be visible and unambiguous, OCR text must be legible and recorded only as displayed alphanumeric strings, scene-relation labels must describe static spatial relations such as “to the left of” or “behind,” and temporal events must be annotated at verb-level granularity such as “open,” “slide,” or “pour.”

Quality control is built into the construction process. Two independent annotators are used, inter-annotator agreement is measured via average IoU of at least 0.85, and conflicts are resolved by a senior annotator or discarded. At the granularity level, bounding boxes are provided on each key frame, with typical key-frame spacing of at most one second, and spatial annotations are derived from the original video resolution before normalization for storage.

4. Hierarchical hallucination diagnosis

The benchmark’s defining design choice is its decomposition of hallucination into perceptive, temporal, and cognitive levels. Although the full list of 14 fine-grained hallucination types is not enumerated in the provided summary, the available subtype counts show that the task space includes categories such as object, number, knowledge-based explanation, and sequence. This hierarchical structure supports root-cause analysis of hallucination types at three reasoning levels, rather than treating hallucination as a monolithic phenomenon (Luo et al., 15 Sep 2025).

The perceptive level targets failures of direct visual grounding, such as errors about visible entities or attributes. The temporal level targets failures involving ordering, persistence, or event progression across frames. The cognitive level targets higher-order reasoning errors that can remain undetected if evaluation only checks local object recognition. This stratification is operational rather than rhetorical: the benchmark reports separate perceptive, temporal, and cognitive accuracies for baseline models, making it possible to distinguish models that are visually competent but temporally weak from models that maintain some temporal coherence but still hallucinate at the reasoning layer.

Segment-level statistics reinforce the fine-grained nature of the design. The summary reports an average segment duration of approximately 2.5 seconds with $\sigma \approx 1.8$ seconds and an average bounding-box size of approximately 0.40 of frame area with $\sigma \approx 0.15$. Rare-class upsampling and GPT-4o-augmented answer foils are used to mitigate imbalance.

5. Evaluation protocol

Dr.V-Bench evaluates both grounding quality and answer quality. For spatial grounding, it uses intersection over union:

$$\mathrm{IoU}(B_{\mathrm{pred}}, B_{\mathrm{gt}})=\frac{|B_{\mathrm{pred}} \cap B_{\mathrm{gt}}|}{|B_{\mathrm{pred}} \cup B_{\mathrm{gt}}|}.$$

For temporal grounding, it uses temporal IoU:

$$\mathrm{tIoU}=\frac{|[s_{\mathrm{pred}}, e_{\mathrm{pred}}] \cap [s_{\mathrm{gt}}, e_{\mathrm{gt}}]|}{|[s_{\mathrm{pred}}, e_{\mathrm{pred}}] \cup [s_{\mathrm{gt}}, e_{\mathrm{gt}}]|}.$$

These frame- and interval-level measures are aggregated into composite statistics. The benchmark reports $m\_tIoU$, defined as the mean over all temporal segments of tIoU; $m\_vIoU$, defined as the mean over all videos of average per-frame IoU over the intersection frames; and $vIoU@R$, defined as the fraction of instances with $(x_{\min}, y_{\min}, x_{\max}, y_{\max})$0 (Luo et al., 15 Sep 2025).

For QA scoring, yes/no and MCQ items are evaluated by strict match or GPT-4o semantic match. Caption items are judged by GPT-4o, with the judgment procedure reported as validated at 98.5% against humans. This dual emphasis on grounding and answer correctness is central to the benchmark’s purpose: it is not limited to whether a model arrives at a plausible answer, but also assesses whether that answer can be supported by localized video evidence.

6. Baseline performance and Dr.V-Agent

Baseline results indicate that open and proprietary models remain substantially below human performance, especially when accuracy is decomposed by reasoning level. The benchmark reports the following accuracies for selected models and a human oracle (Luo et al., 15 Sep 2025).

Model	Overall accuracy	Perceptive / Temporal / Cognitive
Qwen2-VL (7B)	72.67%	81.13 / 65.61 / 71.27
GPT-4o	77.29%	85.22 / 68.96 / 76.50
Gemini-1.5-Pro	79.68%	87.29 / 73.61 / 78.04
Human oracle	95.25%	—

These numbers show a consistent pattern: perceptive accuracy is the strongest among the three levels, temporal accuracy is lower, and cognitive accuracy remains well below human performance. The gap between perceptive and temporal performance is especially relevant because it indicates that local frame understanding does not guarantee robust video-level reasoning.

Dr.V-Agent is reported to improve these results substantially by enforcing a stepwise grounding-and-reasoning procedure. The reported gains are: VideoChat2 from 36.28% to 53.43% (+17.15), Qwen2-VL from 72.67% to 82.64% (+9.97), GPT-4o from 77.29% to 88.36% (+11.07), and Gemini-1.5-Pro from 79.68% to 91.12% (+11.44). In the benchmark’s framing, these gains support the claim that fine-grained spatial-temporal grounding is not only diagnostic but can materially improve reliability when incorporated into inference-time reasoning.

7. Relation to adjacent benchmarks, applications, and limitations

Dr.V-Bench belongs to a broader class of structured evaluation suites, but its target problem is distinct. It is separate from VBench, which evaluates video generative models through 16 dimensions of video generation quality and video-condition consistency, and validates those metrics against human preference annotations with Spearman correlations exceeding 0.8 across all dimensions (Huang et al., 2023). It is also separate from DrVD-Bench, which evaluates whether vision-LLMs reason like human doctors in medical image diagnosis through modules for Visual Evidence Comprehension, Reasoning Trajectory Assessment, and Report Generation Evaluation over 7,789 image-question pairs (Zhou et al., 30 May 2025). A plausible implication is that the similarity of names can obscure a sharp methodological difference: Dr.V-Bench is a hallucination-diagnosis benchmark for large video models, not a clinical reasoning benchmark and not a video generation benchmark.

In practical terms, Dr.V-Bench is presented as useful beyond QA evaluation. The benchmark provides gold-standard spatial-temporal grounding for end-to-end spatio-temporal alignment and supports related tasks including video captioning, grounded video retrieval, and temporal action localization. It is also positioned as a diagnostic benchmark for guiding architectural improvements, such as better temporal encoders or fact modules (Luo et al., 15 Sep 2025).

Its limitations are equally explicit. The benchmark relies on costly manual annotations, and bounding-box and frame-rate scaling are described as nontrivial. Evaluation of free-form generation beyond QA remains an open challenge. Even with these constraints, the summary characterizes Dr.V-Bench as the most comprehensive public benchmark for evaluating and diagnosing video hallucination in large video models, owing to its combination of 10,000 instances, 14 fine-grained hallucination types, rich spatial-temporal grounding, and three QA formats (Luo et al., 15 Sep 2025).