Dr.V-Agent: Hierarchical Diagnosis of Video Hallucinations

• Dr.V-Agent is a training-free diagnostic component that identifies hallucination errors in large video models through a hierarchical, adaptive grounding process.
• It employs a six-step pipeline across perceptive, temporal, and cognitive levels to extract evidence and generate structured feedback for answer refinement.
• Evaluated on Dr.V-Bench, the system outperforms self-reflection baselines, demonstrating robust performance across diverse hallucination types and complex video scenarios.

to=arxiv_search.search 北京赛车有 json {"query":"(Luo et al., 15 Sep 2025) Dr.V Agent video hallucination", "max_results": 5} to=arxiv_search.search 天天中彩票不中返 json {"query":"video hallucination grounding Self-PEP large video models Dr.V", "max_results": 10} to=arxiv_search.search 大发时时彩计划 北京赛车如何 json {"query":"(Lou et al., 24 Oct 2025) CXRAgent (Feng et al., 26 May 2025) DoctorAgent-RL (Park et al., 10 Nov 2025) Surgical Agent Orchestration Platform (Xu et al., 17 Mar 2025) DAgent", "max_results": 10} Dr.V-Agent is the diagnosis-and-feedback component of the Dr.V framework for analyzing hallucination in large video models (LVMs). It is designed to determine whether a model’s answer is faithful to a video by applying fine-grained spatial-temporal grounding in a hierarchical sequence that progresses from perception to temporality to cognition, and then converting the resulting diagnosis into structured feedback for answer refinement. In the formulation introduced with Dr.V-Bench, Dr.V-Agent is explicitly training-free, modular, and adaptive: it operates independently of the target LVM, inspects the video-question-answer triple, and routes each case through a variable-depth chain rather than a fixed monolithic procedure (Luo et al., 15 Sep 2025).

1. Conceptual scope and task definition

Within the Dr.V framework, hallucination is treated not as a single undifferentiated error class but as a hierarchy of failures distributed across three levels: perceptive hallucination, temporal hallucination, and cognitive hallucination. Perceptive failures include errors about objects, color, count, location, or OCR text; temporal failures concern actions, motion, event order, or dynamic relations; cognitive failures concern causal, counterfactual, contextual, or knowledge-based reasoning (Luo et al., 15 Sep 2025).

The core task is defined over a video $V$, a textual prompt $T$, and a target LVM answer $A$. Dr.V-Agent analyzes whether $A$ is consistent with the raw video and prompt, and, if hallucination exists, it generates a rationale that localizes the inconsistency and can be used for correction. The extracted evidence categories are formalized as

$$O = \{o_1, \dots, o_n\}, \quad E = \{e_1, \dots, e_m\}, \quad C = \{c_1, \dots, c_k\},$$

where $O$ denotes objects, $E$ events, and $C$ causal claims. This organization is central to the design: Dr.V-Agent does not directly judge answer validity in one pass, but first decomposes the answer into evidence-bearing components that can be checked at different semantic levels (Luo et al., 15 Sep 2025).

A notable implication of this design is that Dr.V-Agent is not itself a general-purpose video answerer. Its role is post hoc diagnosis and correction. The target model produces an answer first; Dr.V-Agent then audits that answer against grounded evidence. This distinguishes it from systems that directly generate task outputs.

2. Hierarchical reasoning and adaptive execution path

The guiding principle of Dr.V-Agent is that reliable video hallucination diagnosis should proceed in the order perception $\rightarrow$ temporal understanding $\rightarrow$ cognition. The paper presents this as a chained hierarchical reasoning process intended to mirror human-like video comprehension: first determine what objects are present, then determine what happens over time, and only then evaluate higher-level causal or commonsense claims (Luo et al., 15 Sep 2025).

This hierarchy is operationalized through an adaptive six-step chain. The full chain is available, but not every instance requires every stage. The routing policy is explicit:

• Perceptive hallucinations use Steps 1, 2, 5, 6
• Temporal hallucinations use Steps 1, 2, 3, 5, 6
• Cognitive hallucinations use Steps 1, 2, 3, 4, 5, 6

This adaptive path is significant because it makes the system hierarchical rather than merely sequential. Easy cases terminate earlier, while cognitively demanding cases trigger deeper evidence collection. The paper characterizes this as both efficient and interpretable, since each added stage corresponds to a necessary dependency: cognitive assessment is deferred until the relevant spatial and temporal evidence has already been grounded (Luo et al., 15 Sep 2025).

3. Six-step diagnostic-feedback pipeline

Dr.V-Agent’s six-step pipeline combines LLM-based routing and reasoning with external open-vocabulary grounding tools. The stages are summarized below.

Stage	Main components	Function
1. Type classification	GPT-4o	Extract $T$0, $T$1, $T$2 and choose reasoning depth
2. Perceptive checking	Grounded SAM 2, YOLO-World	Verify object-level and static claims
3. Temporal checking	CG-STVG, Grounded-VideoLLM	Verify events, motion, and order
4. Cognitive checking	InternVL2, Qwen2-VL	Generate dense causal descriptions over grounded event spans
5. Reasoning	DeepSeek R1	Diagnose inconsistency across evidence levels
6. Feedback generation	Structured output	Produce grounded evidence and revision suggestions

In Step 1, GPT-4o analyzes the question and target response, extracts the relevant objects, events, and causal claims, and determines whether the instance is primarily perceptive, temporal, or cognitive. The paper reports that this routing stage achieves 99.6% accuracy on 1,000 manually labeled examples, which is presented as evidence that hierarchical path selection is highly dependable (Luo et al., 15 Sep 2025).

In Step 2, Dr.V-Agent performs perceptive-level checking with Grounded SAM 2 and YOLO-World. For each object $T$3, it cross-validates the outputs of both tools, taking the intersection of the predicted bounding boxes and averaging the timestamps reported by the two tools. This stage targets hallucinations involving object existence, static attributes, location, counts, and OCR content (Luo et al., 15 Sep 2025).

In Step 3, temporal-level checking is performed with CG-STVG and Grounded-VideoLLM. These modules verify whether an event actually occurs, whether temporal order is correct, and whether dynamic attributes are consistent with the answer. This converts the inspection problem from static grounding to event grounding (Luo et al., 15 Sep 2025).

In Step 4, cognitive-level checking uses InternVL2 and Qwen2-VL to generate dense descriptive captions over the event duration identified earlier. The purpose is to represent cause-and-effect relations explicitly, so that unsupported causal or counterfactual reasoning can be tested against grounded descriptions rather than inferred directly from raw frames (Luo et al., 15 Sep 2025).

In Step 5, DeepSeek R1 reasons over the accumulated perceptive, temporal, and cognitive evidence and identifies mismatches between that evidence and the target answer. In Step 6, Dr.V-Agent outputs structured feedback

$T$4

where $T$5 contains extracted spatial-temporal-causal information and $T$6 contains revision suggestions. This makes Dr.V-Agent a diagnostic-feedback loop rather than only a binary hallucination detector (Luo et al., 15 Sep 2025).

4. Grounding apparatus and benchmark substrate

Dr.V-Agent is paired with Dr.V-Bench, which provides the evaluation substrate for the hierarchical framework. Dr.V-Bench contains 10,000 instances drawn from 4,974 videos, spans 14 hallucination types and 18 domains, and supports three QA formats: yes/no QA, multiple-choice QA, and caption generation QA. Each instance includes fine-grained spatial-temporal annotation, including target object localization, start and end frames, key frames, and bounding boxes (Luo et al., 15 Sep 2025).

The benchmark is explicitly designed to stress grounded reasoning. The paper notes that about 25% of the videos involve substantial complexity such as multiple scene transitions or overlapping events, and that video durations extend to 600 seconds. These properties are relevant because long-horizon temporal reasoning and cluttered scenes are precisely where hallucination diagnosis becomes difficult (Luo et al., 15 Sep 2025).

The grounding submodules are evaluated with spatial-temporal localization metrics. Temporal overlap is measured by

$T$7

with mean temporal IoU reported as $T$8. Spatial localization is measured with

$T$9

and thresholded localization performance is reported as $A$0. For grounding evaluation, the paper samples 2,000 instances from Dr.V-Bench and reports the effectiveness of the external modules used by Dr.V-Agent (Luo et al., 15 Sep 2025).

This benchmark-agent pairing is methodologically important. Dr.V-Agent’s hierarchy is not only proposed conceptually; it is supported by a benchmark whose annotations directly match the evidence structures the agent consumes.

5. Empirical findings and diagnostic behavior

The experimental results support three recurring claims: first, grounding quality of the external modules is adequate for downstream diagnosis; second, hierarchical diagnosis improves over self-reflection baselines; third, task difficulty increases systematically from perceptive to temporal to cognitive reasoning (Luo et al., 15 Sep 2025).

On the grounding side, the paper reports the following results for the expert tools used in Dr.V-Agent:

• Grounded SAM 2: $A$1, $A$2, $A$3, $A$4
• YOLO-World: $A$5
• CG-STVG: $A$6
• Grounded-VideoLLM: $A$7

These numbers are used to justify the reliability of the grounding tools that feed Dr.V-Agent’s later reasoning stages (Luo et al., 15 Sep 2025).

On hallucination diagnosis, Dr.V-Agent is compared against Self-PEP, a self-reflection baseline, across ten evaluated video models. The paper states that Dr.V-Agent consistently outperforms Self-PEP. Representative improvements include VideoChat2 from 38.01 to 53.43 average in one table, or from 36.28 to 54.88 in extended results depending on the evaluation slice; GPT-4o from 77.29 to 88.36; and Gemini-1.5-Pro from 79.68 to 91.12 (Luo et al., 15 Sep 2025).

The gains are reported to be especially strong on hallucination types that require fine-grained grounding, including object, static relation, OCR, and dynamic relation. This is consistent with the architecture: Dr.V-Agent is strongest where explicit external grounding is most informative. The paper also reports systematic difficulty trends: perceptive tasks are easiest, temporal tasks are harder, and cognitive tasks are hardest; similarly, yes/no QA is easiest, multiple-choice is harder, and caption generation is hardest (Luo et al., 15 Sep 2025).

Reliability analyses extend beyond diagnosis accuracy. The paper reports that GPT-4o matches human judgment on caption-generation evaluation with 98.5% accuracy and Cohen’s Kappa = 0.96 on 200 examples. This result is used to support the trustworthiness of the benchmark’s caption-evaluation mechanism (Luo et al., 15 Sep 2025).

6. Interpretation, misconceptions, and relation to adjacent agent systems

A common source of confusion is terminological rather than technical: Dr.V-Agent is a named component of the Dr.V video-hallucination framework, not a generic label for any doctor-like or diagnostic agent. In the cited literature, other systems with superficially similar naming conventions occupy different problem settings. DoctorAgent-RL models multi-turn clinical consultation as a multi-agent reinforcement learning problem (Feng et al., 26 May 2025); SAOP is a voice-directed orchestration platform for multimodal patient-data interaction in da Vinci robotic surgery (Park et al., 10 Nov 2025); CXRAgent is a director-orchestrated system for chest X-ray interpretation (Lou et al., 24 Oct 2025); and DAgent is a relational database-driven analytical report-generation agent (Xu et al., 17 Mar 2025). These systems are related at the level of agentic orchestration, planning, or evidence use, but they are not instances of Dr.V-Agent as defined in the video-hallucination literature.

The distinctive position of Dr.V-Agent is that it performs post hoc grounded diagnosis of another model’s output. It does not primarily optimize consultation policy, surgical interaction, chest X-ray tool coordination, or database report generation. Instead, it diagnoses whether a video answer is faithful, localizes the source of failure across perceptive, temporal, and cognitive levels, and returns correction-oriented feedback (Luo et al., 15 Sep 2025).

The paper frames this design as a practical blueprint for robust video understanding, but it also implies several boundaries. Dr.V-Agent is training-free and built from off-the-shelf experts rather than a jointly optimized end-to-end system; its effectiveness therefore depends on the quality of its external grounding modules. Its hierarchy also encodes a methodological commitment: cognition is assessed only after spatial and temporal evidence has been grounded. This suggests a broader thesis about hallucination diagnosis in multimodal systems—namely, that unsupported high-level reasoning is often downstream of failures in lower-level grounding—while leaving open the question of how such diagnosis should be integrated into future native video reasoning models (Luo et al., 15 Sep 2025).