VideoDeepResearch: Agentic Long Video Framework

• VideoDeepResearch is an agentic framework for long video understanding that iteratively gathers selective evidence to answer complex queries.
• The framework decomposes the task by orchestrating specialized tools such as video clip retrieval, subtitle extraction, and local visual perception.
• It achieves significant performance and efficiency gains by reducing visual token usage and enabling multi-step reasoning over extensive video data.

Searching arXiv for the core paper and closely related benchmark/system papers on video deep research and agentic long-video understanding. VideoDeepResearch is an agentic framework for long video understanding that treats long-video question answering as a problem of iterative evidence gathering rather than brute-force ingestion of all frames into a single multimodal model. In its defining formulation, a text-only large reasoning model orchestrates a modular toolkit of video clip retrieval, subtitle retrieval, subtitle extraction, local visual perception, and global video browsing, progressively constructing a small informative subset of the original video and using that subset to answer the task. This design challenges the assumption that long video understanding necessarily requires a monolithic multimodal LLM with very long context, strong built-in visual perception, and broad domain expertise. Instead, VideoDeepResearch argues that reasoning, perception, and retrieval can be distributed across specialized components and coordinated through agentic tool use (Yuan et al., 12 Jun 2025).

1. Definition and conceptual scope

VideoDeepResearch denotes, in the narrow sense, the framework introduced in “VideoDeepResearch: Long Video Understanding With Agentic Tool Using” (Yuan et al., 12 Jun 2025). In a broader research sense, it also names an emerging problem setting in which a system must inspect video, identify task-relevant visual or subtitle-based clues, search selectively over internal or external evidence sources, and synthesize a grounded answer through multi-step reasoning. This broader interpretation is supported by subsequent benchmarks and systems that explicitly study “video deep research” as an open-web or agentic reasoning problem rather than a conventional closed-context video question answering task (Liu et al., 11 Jan 2026).

The core problem addressed by VideoDeepResearch is long video understanding under severe context constraints. The paper frames a raw video as a frame sequence $\mathcal{X} = \{X_1, \cdots, X_n\}$ and notes that, for hour-long videos at standard frame rates such as 25 fps, $n$ can be around 90,000 natural frames. Current multimodal LLMs cannot process anywhere near that many frames directly; the paper notes that state-of-the-art systems typically process at most about 1,000 frames, and many practical settings use even fewer (Yuan et al., 12 Jun 2025). Uniform downsampling to a much smaller subset $\mathcal{X'} = \{X_1, \cdots, X_m\}$, where $m \ll n$, is therefore lossy by construction, because task-critical evidence may be sparse and temporally localized.

This motivates a shift from monolithic long-context modeling to selective access. VideoDeepResearch argues that long video understanding should be reframed as progressive reasoning over informative video segments. A plausible implication is that the central resource is not raw context length alone, but the quality of the agent’s search policy over video evidence. That interpretation is reinforced by VideoDR, which defines video deep research as a setting where the video supplies localized visual anchors while the answer is distributed across the open web, requiring cross-frame clue extraction, interactive retrieval, and multi-hop reasoning (Liu et al., 11 Jan 2026). It is also consistent with the Vision-DeepResearch benchmark, which argues that realistic multimodal deep-research systems must perform genuine visual search rather than rely on whole-image lookup or textual leakage (Zeng et al., 2 Feb 2026).

2. System architecture and tool modules

The architecture of VideoDeepResearch is organized around a text-only large reasoning model $\mathcal{M}$ connected to a multimodal toolkit. The major modules are a video clip retriever $\mathcal{R}_v$, a subtitle retriever $\mathcal{R}_s$, a visual perceiver $\mathcal{P}_c$, a subtitle extractor $\mathcal{E}_s$, and a video browser $\mathcal{P}_b$, together with an evolving context or memory state $n$0 that stores the task instruction, user question, and accumulated tool outputs (Yuan et al., 12 Jun 2025).

The inference loop begins by segmenting the video into short clips:

$n$1

The context is initialized as

$n$2

with answer $n$3 (Yuan et al., 12 Jun 2025). At each iteration, the reasoning model produces a thought and an action, and if the action is not “answer,” the system parses tool calls, executes them, merges tool outputs into $n$4, and updates the context:

$n$5

The loop continues until the answer is judged sufficient (Yuan et al., 12 Jun 2025).

The toolkit is deliberately specialized. The video clip retriever operates over pre-segmented clips $n$6, where $n$7 is the clip duration, and retrieves top-$n$8 clips in response to a text-only or multimodal query:

$n$9

The subtitle retriever is analogous:

$\mathcal{X'} = \{X_1, \cdots, X_m\}$0

The visual perceiver performs local clip-based visual QA:

$\mathcal{X'} = \{X_1, \cdots, X_m\}$1

while the subtitle extractor maps explicit time ranges to subtitle spans:

$\mathcal{X'} = \{X_1, \cdots, X_m\}$2

For high-level tasks such as identifying a theme, the video browser performs coarse whole-video browsing and directly answers broad questions:

$\mathcal{X'} = \{X_1, \cdots, X_m\}$3

These modules embody a coarse-to-fine access pattern: broad localization through retrieval, then detailed perception over selected segments (Yuan et al., 12 Jun 2025).

The concrete instantiation reported in the paper uses LanguageBind-large (428M) as the video and subtitle retrievers, DeepSeek-r1-0528 as the text-only large reasoning model, and either Qwen2.5VL-7B or Seed1.5VL-Pro as the visual perceiver. Videos are pre-segmented into non-overlapping 10-second clips, and the visual perceivers operate on short clips represented by up to 32 frames (Yuan et al., 12 Jun 2025). This suggests a deliberately modular engineering strategy: each component is independently pretrained and used through orchestration rather than joint end-to-end optimization.

3. Reasoning, retrieval, and evidence accumulation

The central procedural claim of VideoDeepResearch is that long video understanding benefits from a thought-action loop in which reasoning determines what evidence to access next. Rather than retrieve everything at once, the system incrementally constructs a subset

$\mathcal{X'} = \{X_1, \cdots, X_m\}$4

through iterative reasoning steps

$\mathcal{X'} = \{X_1, \cdots, X_m\}$5

with the design goal that $\mathcal{X'} = \{X_1, \cdots, X_m\}$6 (Yuan et al., 12 Jun 2025). The problem is therefore not merely retrieval-augmented generation in the static sense, but a staged search-and-reason loop where early evidence shapes later search.

This mechanism differs from a one-shot retrieval baseline because the relevant retrieval query may not be obvious before some intermediate evidence has been observed. The paper explicitly argues that video RAG is often insufficient for complex long video understanding tasks requiring multi-step reasoning, dynamic exploration, or iterative clarification of what evidence is actually needed (Yuan et al., 12 Jun 2025). A plausible implication is that retrieval in this setting is query-dependent in a stronger sense than standard nearest-neighbor search: the agent must discover what to ask before it can retrieve the right evidence.

This interpretation is echoed by benchmark work. VideoDR formalizes video deep research as a setting where models must extract cross-frame visual anchors from a video and then preserve those anchors while interacting with the open web (Liu et al., 11 Jan 2026). Vision-DeepResearch Benchmark makes a parallel argument on the image side, showing that realistic systems must perform entity-level visual retrieval under non-idealized conditions and that intermediate entity recall is a major bottleneck (Zeng et al., 2 Feb 2026). Together, these results suggest that the distinctive difficulty is not generic chain-of-thought alone, but grounded control over a search trajectory.

The limitations of this strategy are also explicit. VideoDeepResearch acknowledges that retrieval failure is a primary failure mode, especially on tasks such as EgoQA and SportsQA where the retrieval module struggles to localize relevant segments (Yuan et al., 12 Jun 2025). VideoDR reaches a related conclusion in a more controlled benchmark setting: Agentic performance gains depend on whether a model can maintain the initial video anchors over long retrieval chains, and the major bottlenecks are goal drift and long-horizon consistency rather than abstract reasoning errors per se (Liu et al., 11 Jan 2026). This suggests that memory and control policies are at least as important as raw perceptual accuracy.

4. Empirical performance and task profile

VideoDeepResearch is evaluated on four benchmarks: MLVU, LVBench, Video-MME, and LongVideoBench. The paper reports that VideoDeepResearch(Qwen2.5VL-7B) achieves 55.9 on MLVU, 50.7 on LVBench, 72.4 on VideoMME-L, and 64.1 on LongVideoBench, for an average of 60.8, compared with 52.0 for base Qwen2.5VL-7B and 53.2 for Qwen2.5VL-7B + RAG (Yuan et al., 12 Jun 2025). The stronger variant, VideoDeepResearch(Seed1.5VL-pro), achieves 64.5 on MLVU, 55.5 on LVBench, 76.3 on VideoMME-L, and 70.6 on LongVideoBench, for an average of 66.7 (Yuan et al., 12 Jun 2025).

The abstract highlights improvements over the previous state of the art of 9.6% on MLVU (test), 6.6% on LVBench, and 3.9% on LongVideoBench (Yuan et al., 12 Jun 2025). The task-level analysis on MLVU is particularly revealing: the method outperforms the best baselines by 5.0% on NeedleQA, 12.2% on action count, 28.2% on action order, and 17.9% on TutorialQA, and also surpasses the best baseline by 8.8% on Anomaly (Yuan et al., 12 Jun 2025). These are precisely the task types where decomposition and segment localization are likely to matter more than uniform sampling.

The duration analysis reinforces the same point. On very long videos, conventional large-context MLLMs degrade more sharply. GPT-4o and Gemini-1.5 Pro each lose around 13 points when comparing very long videos $\mathcal{X'} = \{X_1, \cdots, X_m\}$7 seconds to short videos $\mathcal{X'} = \{X_1, \cdots, X_m\}$8 seconds, with reported drops of 13.2 and 12.7 points, respectively. VideoDeepResearch drops only 4.9 points under the same comparison (Yuan et al., 12 Jun 2025). This supports the authors’ claim that selective processing scales better with duration than global downsampling.

The efficiency analysis further reports that VideoDeepResearch reaches 73.4% on LongVideoBench for medium-long videos (180–600 seconds), outperforming the best baseline, GPT-4o at 69.1%, and 68.0% on longer videos (900–3600 seconds), outperforming the best baseline by 7.1 points. At the same time, it uses 48,932 visual tokens for shorter videos and 53,920 for longer videos, reductions of 25.0% and 17.4% compared to GPT-4o and Gemini-1.5-Pro respectively, while achieving the best performance (Yuan et al., 12 Jun 2025). This suggests that agentic retrieval can improve both accuracy and token efficiency when the retrieval/perception interface is strong enough.

Yet the empirical picture is not unambiguously in favor of fully end-to-end agentic reasoning. VideoDR compares Workflow and Agentic paradigms and shows that Agentic is not consistently superior: Gemini-3-pro-preview improves from 69% to 76%, but GPT-5.2 remains at 69%, Qwen3-Omni-30B-A3B remains at 37%, and MiniCPM-V 4.5 drops from 25% to 16% (Liu et al., 11 Jan 2026). This suggests that the success of VideoDeepResearch-style systems depends not just on adding tools, but on whether the model can preserve and exploit the relevant anchors throughout the trajectory.

5. Relation to adjacent systems and benchmarks

VideoDeepResearch sits within a rapidly expanding ecosystem of video-agent systems and deep-research benchmarks. A useful comparison is DIVE, an iterative video reasoning system that won the Complex Video Reasoning & Robustness Evaluation Challenge 2025. DIVE organizes question answering as a six-step loop of intent estimation, question breakdown, agent-based answering of sub-questions, refinement, continuation judgment, and final answer generation, and reached 81.44% test accuracy on CVRR-ES (Kamoto et al., 27 Jun 2025). Unlike VideoDeepResearch, DIVE is centered on iterative decomposition over a single input video rather than the long-video selective retrieval problem, but both frameworks share a controlled reason-then-search structure.

VideoDR extends the conceptual frame further by making open-web search part of the benchmark definition. It defines video deep research as video-conditioned open-domain question answering where the answer is distributed across the web and the model must use the video to derive and preserve the right search target (Liu et al., 11 Jan 2026). Its central finding is that goal drift and long-horizon consistency are the main bottlenecks, implying that future systems may need explicit anchor memory, video revisitation, or external state representations.

The Vision-DeepResearch Benchmark makes a closely related point in the static-image regime. It shows that prior multimodal deep-research benchmarks often permit shortcut solutions through textual leakage or near-exact whole-image retrieval and proposes a multi-round cropped-search workflow that substantially improves performance under realistic visual retrieval scenarios (Zeng et al., 2 Feb 2026). This suggests a natural video analogue: spatiotemporal cropping, clip proposal, and keyframe-centric search may become standard design patterns in future VideoDeepResearch systems.

Infrastructure papers broaden the picture. Deep Research Comparator provides a platform for side-by-side comparison and fine-grained annotation of deep research agents, including inspection of intermediate steps and span-level report feedback (Chandrahasan et al., 7 Jul 2025). DeepResearchGym contributes a reproducible retrieval-and-evaluation sandbox for deep research systems, separating alignment with user information needs, retrieval faithfulness, and report quality (Coelho et al., 25 May 2025). ViDR argues that multimodal deep research reports should ground claims in source visual evidence objects rather than remain text-centered, treating source figures as retrievable, interpretable, routable, and verifiable evidence (Shi et al., 13 May 2026). A plausible implication is that a mature VideoDeepResearch system will need not only better long-video reasoning, but also better interfaces for evidence storage, annotation, and verification.

6. Limitations, misconceptions, and future directions

A common misconception is that VideoDeepResearch demonstrates that long video understanding no longer requires strong multimodal models. The paper makes a narrower claim. It does not eliminate the need for visual perception; rather, it relocates that capability into specialized tools while keeping planning in a text-only large reasoning model (Yuan et al., 12 Jun 2025). The overall system still depends critically on the quality of the retrievers and visual perceivers. When those tools fail to localize relevant segments, planning alone cannot recover the answer.

A second misconception is that retrieval alone explains the gains. The paper explicitly compares base MLLMs, MLLM + RAG, and the full agentic framework, and reports that retrieval alone provides only modest gains relative to the larger improvements obtained by iterative reasoning plus selective tool use (Yuan et al., 12 Jun 2025). This suggests that the contribution is not reducible to adding a retriever in front of a standard model.

The main limitations are tool dependence, lack of end-to-end optimization, and trajectory brittleness. VideoDeepResearch is training-free at the system level: the reasoning model, retrievers, and perceivers are independently pretrained components connected procedurally (Yuan et al., 12 Jun 2025). This makes the system practical, but also means coordination errors cannot be corrected through joint learning. VideoDR further indicates that long-horizon consistency is fragile when models must carry anchors through many search steps without revisiting the video (Liu et al., 11 Jan 2026). A plausible implication is that future work will need explicit memory structures, anchor verification, or retrieval-conditioned replanning.

Another open issue is benchmark realism and evidence granularity. VideoDR is only 100 samples, albeit carefully curated (Liu et al., 11 Jan 2026), while Vision-DeepResearch Benchmark is image-based and therefore omits the temporal complications that distinguish video evidence (Zeng et al., 2 Feb 2026). Future benchmarks will likely need finer-grained annotations of anchors, search trajectories, and timestamped evidence chains. Systems such as VDCook, which frame video data construction as a configurable operating system for retrieval, synthesis, metadata enrichment, and provenance tracking, suggest that the infrastructure for such work is beginning to emerge (Wu, 4 Mar 2026). DataCube similarly shows how large video collections can be turned into semantically searchable corpora through natural-language profiling, hybrid retrieval, and deep semantic matching (Ju et al., 18 Feb 2026).

The long-term significance of VideoDeepResearch is therefore less a final solution than a reframing. It suggests that long video understanding is an evidence-gathering problem in which selective retrieval, modular perception, and controllable reasoning can outperform brute-force context expansion under realistic resource constraints (Yuan et al., 12 Jun 2025). Subsequent work suggests that the next stage will require explicit control of visual anchors, stronger open-web verification, and richer multimodal evidence objects rather than purely textual summaries (Liu et al., 11 Jan 2026).