ReTool-Video: Recursive Tool-Using Video Agents with Meta-Augmented Tool Grounding

Abstract: Video understanding requires active evidence seeking, motivating tool-augmented video agents for temporal reasoning, cross-modal understanding, and complex question answering. Existing video agents have improved video reasoning with retrieval, memory, frame inspection, and verifier tools, but they still face two limitations: (1) a coarse tool space that lacks fine-grained operations for compositional reasoning; and (2) a flat action space that forces high-level video intents into primitive executable tool calls. In this paper, we address these challenges with two complementary designs. First, we construct a MetaAug-Video Tool Library (MVTL), an extensible tool library with 134 registered tools, including 26 base tools for general multimodal signal processing and 108 meta tools for filtering, aggregation, reranking, formatting, and other intermediate-result operations. MVTL supports dual-level access to both structured video information and raw modal evidence, enabling diverse video reasoning scenarios. Second, we propose ReTool-Video, a recursive tool-using method that grounds high-level video intents into executable tool chains. In ReTool-Video, matched actions are executed directly, while unmatched intents are delegated to a resolver for parameter repair, tool substitution, or decomposition. This allows abstract actions such as temporal merging, cross-modal verification, or repeated-event aggregation to be progressively translated into concrete multimodal operations at runtime. Experiments on MVBench, MLVU, and Video-MME w/o sub. show that ReTool-Video consistently outperforms strong baselines. Further analysis demonstrates that recursive grounding and fine-grained meta tools improve the stability and effectiveness of complex video understanding.

• The paper introduces a recursive tool-grounding mechanism that decouples intent from direct tool execution, enabling robust multi-step video reasoning.
• The paper details the MetaAug-Video Tool Library (MVTL) with 134 tools, providing fine-grained compositional integration of visual, textual, and temporal evidence.
• The paper demonstrates state-of-the-art performance on benchmarks like MLVU and Video-MME using an RL-trained planner to optimize tool call precision and efficiency.

ReTool-Video: Recursive Tool-Using Agents for Compositional Video Understanding

Introduction

"ReTool-Video: Recursive Tool-Using Video Agents with Meta-Augmented Tool Grounding" (2605.13228) addresses the persistent bottlenecks in tool-augmented video agents, emphasizing two main deficiencies: the absence of both a diverse, composable tool space and a mechanism for flexibly grounding high-level video reasoning intents to executable tool chains. The work introduces the MetaAug-Video Tool Library (MVTL) and the ReTool-Video agent, which, when integrated, markedly enhance multi-step video understanding and question answering (QA) across complex temporal and multimodal scenarios.

MetaAug-Video Tool Library (MVTL)

A major contribution is the construction of MVTL, which integrates 134 registry-level tools subdivided into 26 base tools and 108 meta tools, thereby offering agents fine-grained compositional execution capabilities. Base tools focus on core evidence acquisition across modalities (retrieval/search, visual analysis, ASR, transcript, event graph, etc.), while meta tools are dedicated to intermediate-result processing: filtering, aggregation, reranking, formatting, post-processing, temporal merging, counting, computation, and group operations.

Figure 1: Functional categories of base tools and meta tools in MVTL, illustrating the registry’s extensibility and fine-grained structure.

MVTL supports dual-level evidence access: (i) fast retrieval and manipulation of structured signals (captions, ASR segments, event graphs, knowledge graphs), and (ii) raw video access (timestamped frames, clips) as the final evidence authority. The registry structure—incorporating tool descriptions, modality tags, input/output schemas, and execution constraints—enables context-aware routing and reliable interface expansion (supporting new, domain-specific tools without breaking compatibility).

Recursive Tool Grounding in ReTool-Video

ReTool-Video’s core advance is the separation of intent expression from direct tool executability. The planner is permitted to output both executable actions as well as abstract, potentially under-specified intent actions. The runtime system then either executes or recursively decomposes these actions using the "resolver" module. This enables strategies such as parameter repair, tool substitution, or compositional decomposition until only executable, registry-matching tool chains remain.

Figure 2: Overall framework of ReTool-Video, with the planner, MVTL tool library, and recursive resolver. Primitive actions are executed directly; abstract actions are recursively resolved into chains of tool calls.

This recursive grounding mechanism allows agents to flexibly operate across a broad action space, supporting compositional tasks like temporal merging, multi-segment aggregation, and longitudinal event verification. In practical terms, this leads to robust handling of cases where video reasoning requires aligning multi-step observations from different modalities or timescales, circumventing the "flat action" limitation prevalent in prior agents.

Additionally, ReTool-Video employs controlled parallel execution for cases where several independent information probes can be conducted simultaneously, improving efficiency without introducing context confusion.

Reinforcement Learning for Planner Policy

The planner within ReTool-Video is trained via group-wise reinforcement learning (RL), focusing update signals on high-level decisions (action selection, evidence sufficiency, and termination) while freezing the resolver and tool-execution components. The RL reward combines final-answer correctness, structural validity of the output, and a cost penalty for ineffective or unnecessarily extended trajectories. Training utilizes group-relative advantage estimation for more robust policy optimization over long-horizon reasoning sequences. Only planner tokens are updated, ensuring that credit assignment is not diluted by lower-level action repair or schema normalization operations.

Empirical Evaluation

ReTool-Video was evaluated on MVBench, MLVU, and Video-MME$_{\mathrm{w/o\ sub.}}$, benchmarks targeting temporal video understanding and long-range, multimodal QA. The reported results demonstrate decisive superiority over both open and closed-source baselines. For instance, ReTool-Video (9B parameters) achieves 81.5% on MLVU and 76.6% on Video-MME$_{\mathrm{w/o\ sub.}}$, surpassing InternVL3.5-30B-A3B (73.0% and 68.7% respectively) as well as competitive closed-source models at a smaller scale.

This margin is even more pronounced on long-video datasets, underlining that the architecture’s recursive intent-grounding and meta-tool orchestration directly target the dominant error modes of existing approaches—namely, brittle parameterization and inappropriate tool selection for multi-turn, long-horizon tasks.

Ablation studies confirm the contributions of each system component: omitting meta tools, recursion, or RL planner policy each yields significant accuracy drops, demonstrating that both flexible action grounding and sophisticated intermediate-result operations are necessary for robust video reasoning.

Figure 3: Final accuracy as a function of tool-use statistics. Success correlates more strongly with tool-call quality and success rate than with raw call count or trajectory length.

Tool behavior analysis shows that accuracy correlates with the precise, selective use of tools, rather than indiscriminate invocation. Meta tools, though fewer in calls, are critical for scenarios requiring evidence integration, such as deduplicating temporal overlaps, aggregating distributed evidence, or post-processing candidate windows.

Qualitative Analysis

The MLVU case study highlights the recursive grounding process: coarse retrieval provides noisy or ambiguous candidates, the planner expresses an abstract need for visual verification, and the resolver decomposes this into controlled Video_Clip_QA tool chains. Meta tools then operate to deduplicate temporally adjacent detections and aggregate/merge evidence across event boundaries—a process that avoids the over-counting and brittle failure prevalent in existing flat-action agents.

Figure 4: Case study on MLVU; initial retrieval identifies candidate intervals, followed by recursive verification, temporal sorting, and meta-tool-based merging for robust event counting.

Implications and Future Directions

ReTool-Video substantively expands the granularity and compositionality of tool-augmented agents for video understanding, demonstrating empirically that such advances translate into substantially improved performance on long-range, multi-step video QA.

The extensibility of MVTL, coupled with recursive resolver mechanisms, enables continual capability expansion and domain adaptation without core system redesign. The registry and normalization protocols facilitate compositional reasoning compatible with future large multimodal agents.

The adoption of planner-focused RL suggests that credit assignment should be prioritized at the orchestration (rather than execution) layer, and that compositional agent architectures can flexibly scale to new benchmarks and instruction paradigms with limited additional training.

Future directions include: automating tool-discovery and meta-tool synthesis; exploring hierarchical planner-resolver architectures with explicit semantic memory; and integrating causal and counterfactual reasoning modules leveraging MVTL’s fine-grained tool schemas and dual-level access.

Conclusion

ReTool-Video establishes a new state-of-the-art in agentic video QA, combining an extensible tool library (MVTL) with a recursive grounding mechanism to address both action- and tool-level bottlenecks in prior video agents. The architecture demonstrates that compositionality, meta-operations on intermediate results, and explicit abstract-intent handling—underpinned by RL-driven action selection—are essential for robust, generalizable video understanding. These results provide an actionable blueprint for the next generation of multimodal tool-augmented artificial agents.