Video Intelligence via Tool-Augmented Learning

• The paper presents VITAL as an end-to-end multimodal framework that interleaves text-based reasoning with on-demand tool calls to enhance video understanding.
• It employs a visual toolbox—chiefly a video clipping tool—to dynamically retrieve densely sampled frames, improving temporal grounding and answering accuracy.
• Reinforcement learning with Difficulty-aware GRPO optimizes tool usage, yielding significant performance gains on benchmarks for long-video question answering and grounding.

Video Intelligence via Tool-Augmented Learning (VITAL) is an end-to-end, agentic video reasoning framework for multimodal LLMs (MLLMs) designed for video question answering, temporal grounding, and grounded video question answering. Its central premise is that long-video understanding should not rely on a single pass over sparsely sampled frames: instead, the model is equipped with a visual toolbox—primarily a video clipping tool—and trained to perform multimodal Chain-of-Thought (CoT) that interleaves textual reasoning with on-demand acquisition of new video evidence. In this formulation, temporal grounding and question answering are treated as mutually beneficial rather than separate competencies, and reinforcement learning is used to optimize tool use and temporal precision in a unified multi-task setting (Zhang et al., 6 Aug 2025).

1. Conceptual foundation

VITAL arises from a specific diagnosis of long-video failure modes in contemporary MLLM pipelines. Standard systems typically pre-sample a small, fixed set of frames or compress long videos into limited visual tokens. This weakens temporal coverage, makes critical events easy to miss, and encourages text-only CoT to continue reasoning without re-checking visual evidence. In long videos, these properties manifest as weak temporal coverage, hallucination, and limited coupling between temporal grounding and answer generation (Zhang et al., 6 Aug 2025).

The framework therefore treats video reasoning as an active evidence-seeking process. Rather than assuming that all relevant visual information must be present in the initial context, VITAL allows the model to request densely sampled frames from any temporal interval of the source video. The model can then refine its hypotheses by alternating between reasoning and tool calls, producing a multimodal trajectory rather than a purely textual derivation. This design is explicitly intended for long-video reasoning tasks such as video question answering, temporal grounding, and grounded VQA (Zhang et al., 6 Aug 2025).

A common misconception is that tool augmentation in video reasoning is simply a matter of attaching additional modules to a frozen MLLM. VITAL’s formulation is stricter: the model is trained to use tools within its reasoning trajectory, and its reward structure directly incorporates temporal grounding quality and answer quality. The result is not merely auxiliary perception, but a learned policy over when and how to request new visual evidence (Zhang et al., 6 Aug 2025).

2. System architecture and tool interface

VITAL is built on a standard Visual Encoder + LLM MLLM backbone, instantiated in the paper with Qwen2.5-VL-7B. The visual encoder processes video frames or dense clips into visual tokens, a merger or projector aligns those tokens with the LLM token space, and the LLM autoregressively generates >, <tool_call>, and <answer> segments. There is no separate explicit planner: the LLM itself serves as the controller over reasoning and tool use (Zhang et al., 6 Aug 2025).

The visual toolbox contains three configurable tool types, although the principal tool used in training and experiments is the video clipping tool. Its role is to return densely sampled frames from a queried time interval:

$$\mathcal{V}_{k+1} = g_{\text{clip}}(\mathcal{V}_0, t_{\text{start}}, t_{\text{end}})$$

where $\mathcal{V}_0$ is the original video and $\mathcal{V}_{k+1}$ is the returned clip as visual tokens. Two additional tool types, clip captioning and clip QA, appear in ablations rather than the default training setup (Zhang et al., 6 Aug 2025).

The multi-round reasoning process is formalized through iterative generation and parsing:

$$\mathcal{O}_k = f_{\text{MLLM}}\big(\{\mathcal{T}_i,\mathcal{C}_i,\mathcal{V}_i\}_{i=0}^k\big)$$

$$(\mathcal{T}_{k+1}, \mathcal{C}_{k+1}) = p(\mathcal{O}_k)$$

If $\mathcal{C}_{k+1}$ is a valid tool call, the toolbox executes it and returns either visual tokens or text; if instead the model produces an answer sequence, the trajectory terminates. The full multimodal CoT is written as

$$\tau = \{\mathcal{T}_1, \mathcal{C}_1, \mathcal{V}_1, \mathcal{T}_2, \mathcal{C}_2, \mathcal{V}_2, \dots, \mathcal{T}_n, \mathcal{A}_n\}.$$

This representation makes tool calls part of the same autoregressive sequence as reasoning and answering (Zhang et al., 6 Aug 2025).

The default invocation protocol uses JSON-like content inside <tool_call> blocks, with fields for tool name and arguments such as "start" and "end". During inference, the framework uses up to 2 tool rounds, so the deployed system preserves the same agentic protocol used in training while keeping runtime bounded (Zhang et al., 6 Aug 2025).

3. Multimodal Chain-of-Thought and data construction

VITAL is trained with two dedicated multi-task datasets. MTVR-CoT-72k is the supervised fine-tuning corpus, split into MTVR-CoT (54k), which contains text-based CoT without tools, and MTVR-CoT-Tool (18k), which contains long-video samples with multimodal CoT including tool calls. MTVR-RL-110k is the reinforcement-learning corpus, split into MTVR-RL (94k) and MTVR-RL-Tool (16k) (Zhang et al., 6 Aug 2025).

The data covers three task families: temporal grounding, reasoning VQA, and grounded VQA. Sources include Charades-STA, ActivityNet-MR, VidChapters-7M, Video-R1, LongVideo-Reason, ReXTime, and NExT-GQA. The generation pipeline first samples multiple rollouts from the base MLLM, measures reward dispersion to remove samples that are too easy or too hard, and then uses a stronger reasoning MLLM to synthesize either text-only CoT or tool-augmented multimodal CoT. For temporal grounding, tool ranges are generated by adding approximately 20% random noise to ground-truth boundaries; for VQA, the reasoning model selects tool parameters directly (Zhang et al., 6 Aug 2025).

The supervised objective is standard teacher-forcing cross-entropy over sequences that include reasoning text, tool calls, and final answers:

$$\mathcal{L}_{\text{CE}} = -\sum_{t=1}^{T} \log p_\theta(y_t \mid y_{<t}, x)$$

where the target sequence can contain <think>, <tool_call>, and <answer> tokens. This means answer prediction, temporal grounding, CoT generation, and tool usage are all encoded in the same next-token training objective rather than separated into task-specific heads (Zhang et al., 6 Aug 2025).

This training strategy is significant because it operationalizes multimodal CoT as a supervised object. VITAL is not merely given answers and asked to infer tool use implicitly; it is shown full trajectories where the reasoning process explicitly decides to retrieve new evidence, inspects that evidence, and only then answers. This suggests that in VITAL, the tool interface is part of the model’s learned language of reasoning rather than an external inference-time heuristic.

4. Reinforcement learning and Difficulty-aware GRPO

The reinforcement-learning component addresses what the paper terms difficulty imbalance in multi-task video RL. Task-wise imbalance arises because some tasks saturate under exact-match rewards while temporal grounding progresses through continuous IoU improvements. Sample-wise imbalance arises because easy samples become solved early, leaving the learning signal dominated by harder, lower-reward cases. VITAL addresses both through Difficulty-aware Group Relative Policy Optimization (DGRPO) (Zhang et al., 6 Aug 2025).

For each rollout $\tau$, three base rewards are defined: an accuracy reward $\mathcal{R}_{\text{acc}}(\tau)$, a format reward $\mathcal{V}_0$0, and, in tool stages, a tool reward $\mathcal{V}_0$1. The intermediate reward is

$\mathcal{V}_0$2

where $\mathcal{V}_0$3 is the task-wise scaled accuracy. For temporal grounding tasks,

$\mathcal{V}_0$4

with different $\mathcal{V}_0$5 choices for short and long temporal grounding tasks (Zhang et al., 6 Aug 2025).

Sample-wise difficulty is then estimated as the mean reward over a rollout group:

$\mathcal{V}_0$6

and converted into a difficulty weight

$\mathcal{V}_0$7

The final reward is

$\mathcal{V}_0$8

This softly downweights easy samples while keeping them in the training distribution (Zhang et al., 6 Aug 2025).

The policy objective follows GRPO form:

$\mathcal{V}_0$9

with the DGRPO reward substituted into the group-relative advantage computation. The overall training schedule uses four one-epoch stages: SFT without tools, DGRPO without tools, SFT with tools, and DGRPO with tools (Zhang et al., 6 Aug 2025).

In grounded VQA, the coupling between answer quality and localization quality is explicit. The accuracy reward is

$\mathcal{V}_{k+1}$0

so correct answering without correct grounding cannot maximize reward. This is one of the clearest formal expressions of the VITAL thesis that temporal grounding and question answering are mutually beneficial rather than independent subproblems (Zhang et al., 6 Aug 2025).

5. Empirical performance and ablation structure

VITAL is evaluated on 11 video understanding benchmarks spanning long-video QA, long-video temporal grounding, short-video temporal grounding, grounded VQA, and complex video reasoning. On LongVideo-Reason, VITAL-7B reaches 79.3 accuracy, compared with 70.2 for VITAL-7B without tools and 60.1 for Qwen2.5-VL-7B. On Video-MME long, VITAL-7B reaches 54.0, compared with 51.2 for VITAL-7B without tools and 51.0 for Qwen2.5-VL-7B (Zhang et al., 6 Aug 2025).

The temporal grounding gains are especially pronounced on long-video settings. On VidChapters-7M, VITAL-7B achieves 34.7 on $\mathcal{V}_{k+1}$1, compared with 25.8 for its text-CoT counterpart. On VUE-TR-Vision, it reaches 35.3 in $\mathcal{V}_{k+1}$2/AUC, compared with 31.6 for VITAL-7B without tools and 12.6 for Qwen2.5-VL-7B. On short-video grounding benchmarks it remains competitive or state of the art, reaching 72.0 on Charades-STA $\mathcal{V}_{k+1}$3 and 50.8 on ActivityNet-MR $\mathcal{V}_{k+1}$4 (Zhang et al., 6 Aug 2025).

On grounded VQA, the joint formulation also yields gains. VITAL-7B improves NExT-GQA from 37.2/77.5 to 43.0/78.7 in mIoU/Acc, and ReXTime from 40.9/79.1 to 47.6/80.5. On complex video reasoning benchmarks, it reaches 41.8 on VSI-Bench, 54.2 on Video-MMMU, and 68.7 on MMVU multiple-choice, matching or exceeding several specialized video reasoning baselines (Zhang et al., 6 Aug 2025).

The ablation results clarify what the framework is and is not. Combining temporal grounding and reasoning VQA improves over training on either alone, and adding grounded VQA improves further: the reported average rises from 45.4 for TG only and 42.0 for RQA only to 55.5 for TG+RQA and 57.1 for TG+RQA+GQA. DGRPO also improves over SFT alone and over SFT plus vanilla GRPO, especially on LongVideo-Reason and VidChapters-7M (Zhang et al., 6 Aug 2025).

A further misconception concerns tool choice. In zero-shot ablations with GPT-4.x and Gemini, clip captioning and clip QA can harm localization performance, while video clipping is the only tool that does not harm and can help when trained. This indicates that VITAL’s improvement is not simply the presence of more tool endpoints; it depends on training the model around a tool whose interface preserves temporal precision (Zhang et al., 6 Aug 2025).

6. Position within agentic video reasoning research

VITAL is best understood as one influential point in a broader design space of tool-augmented video intelligence. The original paper defines it as a multimodal tool-augmented reinforcement-learning framework with a visual toolbox and multimodal CoT (Zhang et al., 6 Aug 2025). Subsequent work makes this status explicit. AVI, for example, directly compares itself to VITAL and characterizes VITAL as a multimodal tool-augmented RL agent for long-video reasoning, while proposing a training-free alternative based on a three-phase Retrieve–Perceive–Review protocol, a structured video knowledge base with entity graphs, and an open-source model ensemble (Gao et al., 18 Nov 2025).

Several later systems preserve VITAL’s core intuition while modifying the training or control mechanism. ViaRL treats frame selection itself as a learnable tool, optimized with rule-based reinforcement learning using downstream answer accuracy as reward; it reports a nearly 15\% improvement on Needle QA and frames the selector as a temporal grounding tool within a tool-augmented learning loop (Xu et al., 21 May 2025). LongVT makes temporal cropping a native tool through crop_video(start_time, end_time), trains with a three-stage SFT/RL/RFT pipeline, and introduces VideoSIAH for segment-in-a-haystack long-video QA (Yang et al., 25 Nov 2025). VideoTIR emphasizes efficient long-video understanding with comprehensive multi-level toolkits and Toolkit Action Grouped Policy Optimization, while VideoThinker trains an agentic VideoLLM entirely from synthetic tool-interaction trajectories first generated in caption space and then grounded back to video frames (Gao et al., 26 Mar 2026, Li et al., 22 Jan 2026).

Other branches of the literature explore adjacent control questions. STAR imposes a training-free spatiotemporal schedule over a Video Toolkit, reporting an 8.2\% gain on VideoMME and 4.6\% on LongVideoBench by enforcing alternating temporal and spatial tool use (Fan et al., 11 Dec 2025). AdaTooler-V identifies blind tool-use as a failure mode and introduces AT-GRPO and a Tool Benefit Score so that the model learns whether a visual problem truly requires tools (Wang et al., 18 Dec 2025). ReTool-Video scales the tool space to 134 registered tools and uses recursive grounding to translate abstract actions into executable tool chains, while VIoTGPT provides an earlier ReAct-style example of learning to schedule vision tools for video-centric IoT settings (Liu et al., 13 May 2026, Zhong et al., 2023).

These developments suggest that VITAL is both a specific framework and a broader research program: agentic video models that learn to allocate perceptual bandwidth dynamically, rather than consuming a fixed frame budget in one shot.

7. Limitations, misconceptions, and future directions

The original VITAL study notes several limitations. Very long videos beyond the training range may still be challenging, especially when important events are extremely rare. Complex questions requiring external knowledge or domain expertise remain difficult. Multi-round tool-augmented RL is also significantly slower than single-pass text RL, because each tool call introduces additional visual encoding cost (Zhang et al., 6 Aug 2025).

A second limitation concerns tool granularity. VITAL’s default tool is deliberately narrow: a video clipping interface that preserves temporal control. This yields strong gains, but also means that spatial grounding, audio inspection, OCR, tracking, and other multimodal operations remain outside the default toolbox. Later work points toward these extensions: AVI adds structured memory and graph retrieval, ReTool-Video introduces meta tools and recursive grounding, LongVT extends native tool calling over long videos, and VideoTIR emphasizes internal multi-level retrieval chains (Gao et al., 18 Nov 2025, Liu et al., 13 May 2026, Yang et al., 25 Nov 2025, Gao et al., 26 Mar 2026).

A plausible implication is that the long-term trajectory of VITAL-like systems lies in combining three elements that the literature has so far explored separately: first, VITAL’s explicit coupling of answer quality and grounding quality; second, adaptive or selective tool policies that avoid blind tool use, as in AdaTooler-V; and third, richer structured environments and recursive tool composition, as in AVI and ReTool-Video (Wang et al., 18 Dec 2025, Gao et al., 18 Nov 2025, Liu et al., 13 May 2026). This suggests a transition from single-tool temporal cropping toward general video agents that can reason over clips, frames, entities, transcripts, OCR, and derived intermediate results within one coherent policy.

The broader significance of VITAL is therefore methodological. It established a concrete formulation in which long-video reasoning becomes an interleaved sequence of thought, tool invocation, and evidence review, with temporal grounding embedded directly in the reward and supervision structure. Subsequent systems differ in whether they use supervised synthetic trajectories, rule-based RL, training-free orchestration, adaptive tool gating, or recursive tool grounding, but they continue to operate in the conceptual space that VITAL helped formalize: video intelligence as learned control over external perceptual tools rather than static consumption of pre-sampled frames (Zhang et al., 6 Aug 2025).