Multimodal Chain-of-Tool-Thought

• The paper introduces MM-CoTT, which interleaves linguistic reasoning with explicit tool manipulation to dynamically update and refine multimodal contexts.
• MM-CoTT frameworks utilize a reasoning stack with precise tool parameterization, leveraging both supervised and reinforcement learning for improved inference.
• Empirical results show marked performance gains in visual and geometric tasks, while challenges remain in tool sequencing and computational efficiency.

A multimodal chain-of-tool-thought (MM-CoTT) is a paradigm in which a machine learning model interleaves linguistic reasoning with explicit, stepwise manipulation of visual or other modality-specific tools throughout the course of problem solving. Unlike classical chain-of-thought (CoT) approaches that operate purely in text or reflect vision-LLMs limited to static image interpretation, MM-CoTT augments each stage of reasoning by invoking specialized operations—such as masking, cropping, sketching, or calling cognitive sub-modules—enabling dynamic, context-sensitive information extraction and transformation. This framework is realized through both reinforcement learning finetuning and supervised learning, as well as modular architectures that organize and track the sequence of tool-based interventions and their effects within a single reasoning trajectory.

1. Architectural Foundations and General Mechanisms

Central to MM-CoTT is the extension of LLMs or vision-LLMs (VLMs) with the capacity to (a) select and parameterize external tools, (b) update context with new multimodal artifacts (e.g., edited images, sketched diagrams), and (c) condition subsequent reasoning steps on the evolving multimodal state. This is typically instantiated as a loop or stack: at each iteration, the model generates a thought (text plan), emits a tool call (often as explicit code or JSON), the tool executes and produces an observable result, and the context is updated for the next step.

For example, in Visual Sketchpad, the autoregressive loop involves generating a plan (text), action (code for visual manipulation), observing the tool's output (new image or mask), and continuing until termination (Hu et al., 13 Jun 2024). VTool-R1 attaches Python-based visual-editing functions (e.g., focus_on_columns_with_highlight, draw bounding boxes) to a VLM and explicitly prompts the model with system instructions, tool APIs, and contextual data, such that the model self-determines at each step whether and how to invoke a tool (Wu et al., 25 May 2025). The Simple o3 framework exposes tool call primitives (focus_area, zoom_in, reuse) and structures the reasoning as observe–reason–act cycles, driven by interleaved visual tokens and text (Wang et al., 16 Aug 2025). In agentic variants (e.g., VICoT), the stack formalizes each step as a tuple of (reasoning, tool, evidence), preserving full provenance and allowing explicit multi-round tool-based inference (Wang et al., 25 Nov 2025).

2. Tool Specification, Parameterization, and Integration

Tool integration is operationalized at both the prompt level and within the autoregressive decoding space. Tools are typically defined as Python functions, image operations, or expert "roles" that a model invokes based on the current context. The selection mechanism varies: in some frameworks, the policy head explicitly scores tool tokens against text tokens (Hu et al., 13 Jun 2024), while in others (e.g., VICoT), an MCP (Model Context Protocol) XML schema registers tools and standardizes invocation (Wang et al., 25 Nov 2025). The inputs are fully parameterized (e.g., bounding boxes for cropping, scale factors for zooming, column/row identifiers for masking), and the outputs are programmatically appended to the model's context as both visual (modified image, sketch, mask) and textual (evidence, plan) artifacts.

The toolset spans:

System	Tool Types	Invocation Mechanism
VTool-R1 (Wu et al., 25 May 2025)	Column/row highlight, mask, bbox draw	Python code blocks
Simple o3 (Wang et al., 16 Aug 2025)	focus_area (crop), zoom_in, reuse	JSON function tags
Visual Sketchpad (Hu et al., 13 Jun 2024)	Auxiliary lines, bounding boxes, masks, depth	Python/vision modules
VICoT (Wang et al., 25 Nov 2025)	Detection, crop, super-resolution, denoise, RAG	MCP XML interface

Tools may be applied to either manipulate the current observation (e.g., cropping an image for a focused view) or to create new intermediate reasoning artifacts (e.g., sketching out geometric constructs). The specification enforces strict input formats, and tool call errors are handled either neutrally (no extra penalty, as in VTool-R1) or with explicit ablation logic.

3. Training Objectives and Data Generation Pipelines

Training MM-CoTT models requires datasets where multimodal reasoning steps, tool calls, and their effects are explicitly represented. Supervised fine-tuning approaches use automatically synthesized or verified reasoning chains: for example, Simple o3's TWI-Tools-146K dataset is generated by sampling interleaved reasoning and tool call sequences from an MLLM, then validating geometric and semantic correctness of each step (Wang et al., 16 Aug 2025). The masking of training loss ensures that only text tokens (i.e., reasoning, function tags) contribute gradients; visual tokens serve as context only.

Reinforcement learning frameworks such as VTool-R1 employ an outcome-based reward—where only the correctness of the final answer determines the reward function—and regularization via KL divergence from a reference policy. The GRPO objective is used to stabilize policy updates group-wise, with no intermediate rewards to avoid reward hacking (Wu et al., 25 May 2025). Other approaches, such as VICoT, support stack-level distillation, with teacher–student losses ensuring that compact student agents can reproduce full reasoning trajectories (Wang et al., 25 Nov 2025).

Data pipelines emphasize both diversity (chart QA, perceptual VQA, logical QA, diagram understanding) and validation: e.g., Simple o3's two-stage tool call verification assesses both the geometric validity of crops and the alignment between textual plan and invoked tool (Wang et al., 16 Aug 2025).

4. Reasoning Stack, Context Management, and Control Flow

A key organizing principle in MM-CoTT is the explicit management of the evolving multimedia context. Agentic architectures, such as VICoT and Visual Sketchpad, formalize the reasoning process as a dynamically growing stack or sequence, where each frame records the reasoning statement, tool used, and the evidence produced (Wang et al., 25 Nov 2025, Hu et al., 13 Jun 2024). This enables models to reason over their own prior modifications: auxiliary drawings, segmentations, or cropped regions directly update the context for subsequent steps.

Control flow may entail single- or multi-round tool use. While VTool-R1 currently supports only one tool invocation per query, agentic frameworks and Visual Sketchpad allow for arbitrarily long chains, limited only by context window and inference cost. Stack-based designs not only support forward chaining but permit explicit push/pop/backtrack, affording interpretable trajectories and more robust error recovery (Wang et al., 25 Nov 2025). Linear context growth relative to chain length is maintained for tractability.

5. Empirical Performance, Evaluation, and Ablations

Extensive benchmarks demonstrate that MM-CoTT architectures consistently improve reasoning, particularly on tasks requiring fine-grained perception, visual attention, or geometric manipulation. For instance, Simple o3 achieves an increase of +49.6 on MME(R), +12.9 on VStarBench, and +3.1 on COCO Caption (Rouge-L) over its base model, with the addition of each new tool (reuse, zoom_in, focus_area) incrementally boosting performance (Wang et al., 16 Aug 2025). Visual Sketchpad yields an average gain of 12.7% on math tasks and 8.6% on vision tasks over baseline LMs, setting state of the art on V*Bench (80.3%), BLINK spatial (83.9%), and visual correspondence (80.8%) (Hu et al., 13 Jun 2024).

Ablation studies reveal that disabling tool use, reducing the interleaving of vision-language, or omitting stack/context management can sharply degrade performance: VICoT's removal of the reasoning stack results in –18% accuracy (Wang et al., 25 Nov 2025); Simple o3 demonstrates that omitting focus_area cropping produces the largest regression in fine-grained visual tasks (Wang et al., 16 Aug 2025).

Reward shaping, process-based penalties, or extrinsic tool rewards can introduce undesirable behaviors—such as models spamming tools or ceasing to invoke them—highlighting the necessity of pure outcome-based or carefully balanced objectives (Wu et al., 25 May 2025).

6. Methodological and Domain-Specific Extensions

MM-CoTT frameworks generalize beyond canonical VQA to domains such as remote sensing, 3D reasoning, diagram understanding, and science QA. VICoT, for example, demonstrates scalability to ultra-high-resolution data, multi-turn tool use, and plug-and-play tool extensibility via standard interfaces (MCP), with domain-specific toolsets for denoising, segmentation, web search, and retrieval augmentation (Wang et al., 25 Nov 2025). In 3D vision–language alignment, hierarchical CoT annotations (object, function, interaction) embedded during contrastive and adaptation training yield robust multi-step understanding, with layered evaluations distinguishing improvements in both the intermediate reasoning quality and final inference accuracy (Chen et al., 8 Mar 2025).

Cantor formalizes cognitive experts within a single MLLM, allocating sub-tasks to role-prompted modules (TextIntel, ObjectQuant, VisionIQ, ChartSense) without the need for model fine-tuning or explicit gating, yielding compositional chains of toolthought (Gao et al., 24 Apr 2024).

7. Limitations, Open Challenges, and Future Directions

Current MM-CoTT systems remain constrained by factors including toolset coverage (most toolkits are limited to simple image operations and masking), single-round invocation in some frameworks, and the lack of human-annotated ground truth for stepwise tool use (Wu et al., 25 May 2025). Compute cost and tool reliability are especially acute in multi-step agentic architectures where each cycle may incur vision module inference and code execution overhead (Hu et al., 13 Jun 2024).

A persistent open problem is the dynamic selection and sequencing of tools without overfitting to spurious chains or succumbing to hallucinations in tool invocation (Gao et al., 24 Apr 2024). There is also a need for further formalization and standardization in the representation, exchange, and evaluation of multimodal tool-chains, especially for deployment in edge or real-time scenarios. Stack distillation offers one viable path towards compact, efficient agents (Wang et al., 25 Nov 2025), but broader generalization and more powerful, context-sensitive tool reasoning remain active research frontiers.

---

These developments collectively instantiate the field of multimodal chain-of-tool-thought as a principled framework for stepwise, interpretable, and highly context-sensitive multimodal reasoning, integrating external visual tools directly into the cognitive process of modern language and vision models (Wu et al., 25 May 2025, Wang et al., 16 Aug 2025, Hu et al., 13 Jun 2024, Wang et al., 25 Nov 2025, Gao et al., 24 Apr 2024, Chen et al., 8 Mar 2025).