---
title: 'TWI-Tools-146K: MLLM Reasoning Dataset'
url: https://www.emergentmind.com/topics/twi-tools-146k
type: topic
---

# TWI-Tools-146K: MLLM Reasoning Dataset

TWI-Tools-146K is a large-scale, richly annotated dataset designed for training and evaluating Multimodal Large Language Models (MLLMs) on interleaved vision-language reasoning tasks. Developed in the context of the Simple o3 framework, the dataset systematically encapsulates human-like “observe–reason–act” cycles using executable visual tool commands, enabling studies of chain-of-thought reasoning grounded in both linguistic and visual modalities. The resource is open-source and tailored to fine-tuned MLLMs for tasks including perceptual question answering, knowledge-based reasoning, abstract figure analysis, and mathematical problem-solving [2508.12109].

## 1. Dataset Construction and Scope

TWI-Tools-146K is generated through a scalable synthesis pipeline that initially produces approximately 150,000 candidate multimodal samples. After a rigorous two-stage verification, the final dataset comprises approximately 100,000 high-quality interleaved reasoning chains, with the name “TWI-Tools-146K” reflecting its final scale. Each sample documents a sequence of visual observations, reasoning steps, and executable tool commands, culminating in a grounded final answer. The core task types are:

- Perceptual VQA based on real-world imagery from COCO
- Knowledge-based vision-language reasoning
- Chart and abstract figure analysis (including ArXivQA)
- Mathematical and logical reasoning (sourced from MATHV360K, excluding pure geometry)

This design enables the dataset to support experiments requiring nuanced and compositional understanding that intertwines visual inspection and language-based reasoning.

## 2. Generative Formalism and Dataset Structure

The formal generative process for each sample $S$ is given by:
$$
S = \{s_0, s_1, \ldots, s_t, A\}
$$
where, at each time step $t$,

- The current observation image is $I_t$
- The model emits an atomic reason–plan pair $R_t = (r_t, p_t)$
- The model issues an action command $C_t = (T_n, \text{Param})$ via a tool call
- The next image $I_{t+1}$ is generated by executing $C_t$
- The chain terminates when an answer $A$ in <answer>…</answer> tags is produced

The recursive history is 
$$
H_{t-1} = 
\begin{cases}
\emptyset & t=0 \\
\{ (R_i, C_i, I_i) \}_{i=0}^{t-1} & t>0
\end{cases}
$$

The annotation format for each sample is a JSON object with fields for the question, initial image path, a sequence of reasoning- and function-labeled steps, and the final answer. Images are referenced via file paths or URLs, and all tool parameters use unified pixel-based coordinates.

## 3. Tool Specification and Action Semantics

All visual reasoning chains are constructed using three core visual tools:

1. **focus_area($x_1$, $y_1$, $x_2$, $y_2$):** Draws a bounding box around a specified region or crops the image if used in pure cropping mode.
2. **zoom_in(factor):** Linearly magnifies the image by interpolation.
3. **reuse():** Re-presents the current image unchanged for further steps.

Each tool output is verified for geometric and semantic validity. Tool usage is serialized via special tags (e.g., <function>…</function>) and tool outputs are explicitly linked to reasoning plans for transparent replay and evaluation.

## 4. Data Synthesis Pipeline and Quality Control

The synthesis pipeline employs a frozen MLLM (gemini-2.5-Flash-Preview) to produce candidate reasoning and tool steps in “non-thinking” mode. The iterative procedure, detailed in the paper’s Algorithm 1, proceeds as follows:

- Initialize the history as empty.
- For each step $t$, the model samples a reasoning-tool pair $(R_t, C_t)$ conditioned on the question, image, and prior history.
- The tool command is executed to yield the next observation image.
- Verification is performed:
    - *Geometric*: focus_area bounding boxes must align with the referenced entity (checked by gemini-lite).
    - *Semantic*: zoom_in and reuse must match the visual planning step.
- If verification fails, up to two regeneration attempts are permitted; otherwise, the sample is discarded.
- Final answers are validated by comparison with the ground-truth using Qwen3-turbo.
- The overall retention rate, post verification, is approximately 67% (100,000 out of 150,000 candidates).

## 5. Benchmarking Results and Tool Ablation

Direct ablation studies on TWI-Tools-146K reveal substantial effects of tool composition and reasoning structure:

| Tool Setup           | MME Reasoning Gain | VStarBench Gain | HR-Bench 4K | Notes                     |
|----------------------|-------------------|-----------------|-------------|---------------------------|
| reuse                | +31.2             | +4.8%           | –           | vs. Qwen2.5-VL-7B baseline|
| zoom_in              | <1%               | Small           | –           | marginal except VStarBench|
| focus_area (cropping)| –                 | +6.9%           | +4.9%       | largest fine-grained lift  |

Importantly, cropping with focus_area (injecting only relevant image tokens) outperforms both bounding box drawing and reuse on fine-grained perception tasks. Adjusting input resolution (2048×28×28 vs. 16384×28×28 tokens) balances reasoning and visual acuity, but extreme resolution can overload the model’s attention on complex scenes.

## 6. Applications and Future Directions

TWI-Tools-146K enables the training, evaluation, and extension of MLLMs for multi-step, executable vision-language reasoning. The dataset’s explicit step tagging and tool command structure facilitate reproducing the “observe–reason–act” chains and augmenting them with additional tool functions. Its cross-domain coverage (COCO, MATHV360K, ArXivQA) supports diverse research on perception, visual understanding, mathematical reasoning, and figure interpretation.

A plausible implication is that the dataset provides a robust testbed for protocol ablation and error analysis in interleaved reasoning, as well as for studying the impact of explicit tool call integration in next-generation MLLMs.

## 7. Relation to Related Datasets and Impact

Unlike resources focused exclusively on VQA or static image-text retrieval, TWI-Tools-146K is notable for incorporating explicit multistep, executable tool actions within annotated vision-language chains. This structure reflects a shift toward emulating human reasoning patterns—iteratively observing, hypothesizing, executing, and refining—within an MLLM framework. Researchers can directly replay or augment these chains in fine-tuning or evaluation routines, as well as extend the toolset for novel vision-language tasks [2508.12109].

Source: https://www.emergentmind.com/topics/twi-tools-146k