v1g Dataset for Multimodal Reasoning

• The v1g dataset is a large-scale resource that provides 300K multimodal reasoning traces with explicit, fine-grained visual grounding annotations linking text to image regions.
• It employs an automated three-stage pipeline to decompose reasoning traces, extract visual queries, and align these with specific image regions, enhancing dynamic visual access.
• The dataset enables supervised training and benchmarking of models by integrating pointer tokens for iterative and context-sensitive visual revisitation during sequential decision-making.

The v1g dataset is a large-scale resource designed to support the development and evaluation of selective visual revisitation mechanisms in multimodal LLMs. Consisting of 300,000 multimodal reasoning traces, v1g provides explicit, fine-grained visual grounding annotations that link each text-based reasoning step to precise image regions, enabling the training and benchmarking of models that dynamically re-access visual evidence throughout stepwise reasoning processes.

1. Dataset Structure and Construction

The v1g dataset comprises 300K multimodal reasoning traces, each formulated as a sequential chain of reasoning steps where textual inferences are explicitly tied to visual cues. Each entry includes interleaved grounding annotations specifying which image patch or region substantiates a particular textual step. This structure provides supervision for both natural language generation and visual reference selection.

Dataset construction involves an automated three-stage pipeline:

1. Oversampling Reasoning Traces: Source traces are extracted from an existing MLLM output corpus (e.g., TVC), providing a diverse set of multimodal question-answering and reasoning sequences.
2. Trace Decomposition and Visual Query Extraction: An LLM-guided process decomposes each trace, extracting visual queries and marking retrieval moments within complex stepwise reasoning.
3. Visual Grounding Alignment: Each identified visual reference is aligned to a bounding box in the corresponding image, encoding the mapping as explicit grounding annotations. These annotations may indicate, for example, the input patch IDs corresponding to evidence for each reasoning step.

This sequential, automated alignment ensures coverage across various task domains and enables systematic, large-scale supervision for dynamic visual access operations.

2. Supervised Training Protocols

Models trained on v1g, notably the v1 architecture, learn a dual objective: generating language and dynamically retrieving visual tokens (patch embeddings) as dictated by the reasoning context. Training leverages the interleaved annotations such that, at each reasoning step, the model is signaled when to emit standard text tokens and when to point to a visually grounded input embedding.

At training time:

• Text tokens are generated as usual from the task vocabulary.
• Pointer tokens indicate selection of a specific image patch, with each grounded step labeled as a retrieval target (〈ptr:cₖ〉).

This structure allows the pointer module—a learned attention head over continuous image embedding space—to be directly supervised using ground-truth alignment. The model minimizes cross-entropy on both language and pointer outputs, with standard classification objectives applied over each respective subspace.

3. Selective Visual Revisitation Mechanism

The v1 model, trained on v1g, implements point-and-copy dynamic visual revisitation. Instead of consuming the visual input in a single forward pass, the model extends its output space during decoding from the discrete vocabulary $\mathcal{V}$ to $\mathcal{V} \cup \mathcal{C}$, where $\mathcal{C}$ represents image patch embeddings.

At each generation step $t$:

• The decoder hidden state $h_t$ produces
  • $\text{logits}_{\text{vocab}}$ for the vocabulary,
  • $\text{logits}_{\text{ptr}}$ for $k$ image patches, computed as

  $$\text{logit}_\text{ptr}^{(k)} = \frac{L_{(q)}(h_t) \cdot L_{(k)}(c_k)^{\top}}{\sqrt{D}}$$

  where $L_{(q)}, L_{(k)}$ are learned linear projections and $D$ is latent dimensionality.

If a pointer token is selected, the relevant patch embedding $c_k$ is injected at the next input position, enabling iterative and context-sensitive visual reasoning. This permits the model to revisit image regions multiple times, conditionally based on evolving linguistic context and intermediate hypotheses.

4. Benchmarking and Empirical Findings

v1g provides a testbed for evaluating models’ capability to ground, revisit, and reason with visual evidence stepwise. Experiments on MathVista, MathVision, and MathVerse—benchmarks centered on multimodal mathematical problem solving—demonstrate:

• The v1 model, when trained with v1g, achieves improved accuracy over baselines that process the image only once or lack pointer mechanisms.
• The performance gain is most pronounced for problems requiring fine-grained, multi-step visual grounding and gradual accumulation of evidence.
• Ablation studies show that disabling the dynamic pointing module leads to significant reductions in benchmark accuracy, confirming the effectiveness of supervised pointer annotations.
• Compared to larger LLMs that rely entirely on text-based chain-of-thought, selective visual revisitation notably narrows the accuracy gap on visually demanding tasks.

Quantitative evaluations are contextualized by ablation comparisons (e.g., “w/o pointing” vs. full model) and task subgroup breakdowns highlighting where dynamic grounding provides the largest marginal benefit.

5. Theoretical and Practical Implications

The v1g dataset demonstrably prevents “visual grounding decay”—where information from visual input is lost as reasoning chains lengthen—by facilitating continual grounding through pointer annotations. This is particularly advantageous in domains where reasoning requires repeated or recursive reference to spatial details (e.g., diagrammatic math, data charts, scientific or clinical images).

The interleaved annotation design and point-and-copy training protocol can be generalized beyond mathematical problem solving to other modalities requiring sequential visual access, such as multi-turn dialogue referencing diagrams or event localization in videos. A plausible implication is that if grounding granularity were increased (e.g., via segmentation masks or more precise queries), further gains in reliability and interpretability of multimodal models could be realized.

6. Availability and Future Directions

The v1g dataset, along with code and trained model checkpoints, is slated for public release to support further research. The explicit and modular structure of v1g enables extension to more sophisticated grounding targets (e.g., hierarchical pointers, non-rectangular regions), alternative retrieval modalities (audio, video), or weakly supervised settings.

Potential future directions suggested include:

• Integration with reinforcement or curriculum learning to refine pointer selection strategies under weak or partial supervision.
• Extension to diverse domains (scientific illustration, medical image question answering) where multi-hop visual reference is critical.
• Exploration of train-time augmentation to further diversify reasoning traces and test transfer to out-of-domain or cross-modal benchmarks.

In sum, the v1g dataset constitutes a foundational resource for research on dynamic multimodal reasoning, providing explicit supervisory signals for both language and visual access that facilitate improved accuracy, interpretability, and robustness in models requiring iterative grounding.