TWI-Tools-146K: MLLM Reasoning Dataset

• TWI-Tools-146K is a large-scale dataset offering rich annotations and interleaved visual-language reasoning chains.
• It employs an 'observe–reason–act' methodology using explicit tool commands like focus_area, zoom_in, and reuse for systematic evaluation.
• A rigorous two-stage verification process ensures high quality, yielding a 67% retention rate from initial candidate samples.

TWI-Tools-146K is a large-scale, richly annotated dataset designed for training and evaluating Multimodal LLMs (MLLMs) on interleaved vision-language reasoning tasks. Developed in the context of the Simple o3 framework, the dataset systematically encapsulates human-like “observe–reason–act” cycles using executable visual tool commands, enabling studies of chain-of-thought reasoning grounded in both linguistic and visual modalities. The resource is open-source and tailored to fine-tuned MLLMs for tasks including perceptual question answering, knowledge-based reasoning, abstract figure analysis, and mathematical problem-solving (Wang et al., 16 Aug 2025).

1. Dataset Construction and Scope

TWI-Tools-146K is generated through a scalable synthesis pipeline that initially produces approximately 150,000 candidate multimodal samples. After a rigorous two-stage verification, the final dataset comprises approximately 100,000 high-quality interleaved reasoning chains, with the name “TWI-Tools-146K” reflecting its final scale. Each sample documents a sequence of visual observations, reasoning steps, and executable tool commands, culminating in a grounded final answer. The core task types are:

• Perceptual VQA based on real-world imagery from COCO
• Knowledge-based vision-language reasoning
• Chart and abstract figure analysis (including ArXivQA)
• Mathematical and logical reasoning (sourced from MATHV360K, excluding pure geometry)

This design enables the dataset to support experiments requiring nuanced and compositional understanding that intertwines visual inspection and language-based reasoning.

2. Generative Formalism and Dataset Structure

The formal generative process for each sample $S$ is given by:

$S = \{s_0, s_1, \ldots, s_t, A\}$

where, at each time step $t$,

• The current observation image is $I_t$
• The model emits an atomic reason–plan pair $R_t = (r_t, p_t)$
• The model issues an action command $C_t = (T_n, \text{Param})$ via a tool call
• The next image $I_{t+1}$ is generated by executing $C_t$
• The chain terminates when an answer $A$ in <answer>…</answer> tags is produced

The recursive history is

$$H_{t-1} = \begin{cases} \emptyset & t=0 \ \{ (R_i, C_i, I_i) \}_{i=0}^{t-1} & t>0 \end{cases}$$

The annotation format for each sample is a JSON object with fields for the question, initial image path, a sequence of reasoning- and function-labeled steps, and the final answer. Images are referenced via file paths or URLs, and all tool parameters use unified pixel-based coordinates.

3. Tool Specification and Action Semantics

All visual reasoning chains are constructed using three core visual tools:

1. focus_area($S = \{s_0, s_1, \ldots, s_t, A\}$0, $S = \{s_0, s_1, \ldots, s_t, A\}$1, $S = \{s_0, s_1, \ldots, s_t, A\}$2, $S = \{s_0, s_1, \ldots, s_t, A\}$3): Draws a bounding box around a specified region or crops the image if used in pure cropping mode.
2. zoom_in(factor): Linearly magnifies the image by interpolation.
3. reuse(): Re-presents the current image unchanged for further steps.

Each tool output is verified for geometric and semantic validity. Tool usage is serialized via special tags (e.g., <function>…</function>) and tool outputs are explicitly linked to reasoning plans for transparent replay and evaluation.

4. Data Synthesis Pipeline and Quality Control

The synthesis pipeline employs a frozen MLLM (gemini-2.5-Flash-Preview) to produce candidate reasoning and tool steps in “non-thinking” mode. The iterative procedure, detailed in the paper’s Algorithm 1, proceeds as follows:

• Initialize the history as empty.
• For each step $S = \{s_0, s_1, \ldots, s_t, A\}$4, the model samples a reasoning-tool pair $S = \{s_0, s_1, \ldots, s_t, A\}$5 conditioned on the question, image, and prior history.
• The tool command is executed to yield the next observation image.
• Verification is performed:
  • Geometric: focus_area bounding boxes must align with the referenced entity (checked by gemini-lite).
  • Semantic: zoom_in and reuse must match the visual planning step.
• If verification fails, up to two regeneration attempts are permitted; otherwise, the sample is discarded.
• Final answers are validated by comparison with the ground-truth using Qwen3-turbo.
• The overall retention rate, post verification, is approximately 67% (100,000 out of 150,000 candidates).

5. Benchmarking Results and Tool Ablation

Direct ablation studies on TWI-Tools-146K reveal substantial effects of tool composition and reasoning structure:

Tool Setup	MME Reasoning Gain	VStarBench Gain	HR-Bench 4K	Notes
reuse	+31.2	+4.8%	–	vs. Qwen2.5-VL-7B baseline
zoom_in	<1%	Small	–	marginal except VStarBench
focus_area (cropping)	–	+6.9%	+4.9%	largest fine-grained lift

Importantly, cropping with focus_area (injecting only relevant image tokens) outperforms both bounding box drawing and reuse on fine-grained perception tasks. Adjusting input resolution (2048×28×28 vs. 16384×28×28 tokens) balances reasoning and visual acuity, but extreme resolution can overload the model’s attention on complex scenes.

6. Applications and Future Directions

TWI-Tools-146K enables the training, evaluation, and extension of MLLMs for multi-step, executable vision-language reasoning. The dataset’s explicit step tagging and tool command structure facilitate reproducing the “observe–reason–act” chains and augmenting them with additional tool functions. Its cross-domain coverage (COCO, MATHV360K, ArXivQA) supports diverse research on perception, visual understanding, mathematical reasoning, and figure interpretation.

A plausible implication is that the dataset provides a robust testbed for protocol ablation and error analysis in interleaved reasoning, as well as for studying the impact of explicit tool call integration in next-generation MLLMs.

7. Relation to Related Datasets and Impact

Unlike resources focused exclusively on VQA or static image-text retrieval, TWI-Tools-146K is notable for incorporating explicit multistep, executable tool actions within annotated vision-language chains. This structure reflects a shift toward emulating human reasoning patterns—iteratively observing, hypothesizing, executing, and refining—within an MLLM framework. Researchers can directly replay or augment these chains in fine-tuning or evaluation routines, as well as extend the toolset for novel vision-language tasks (Wang et al., 16 Aug 2025).