M4-Instruct Dataset

• The dataset is a comprehensive, unified collection of interleaved image–text samples from multi-image, video, 3D, and single-image patch domains.
• It supports diverse tasks like 'Spot the Difference', image editing, visual storytelling, and VQA with robust in-domain and out-domain evaluations.
• Integration with LLaVA-NeXT models yields enhanced cross-modal generalization, emergent transfer learning, and robust performance on varied visual-linguistic tasks.

The M4-Instruct dataset is a comprehensive, large-scale multimodal instruction dataset specifically designed to facilitate the training and evaluation of large multimodal models (LMMs) capable of handling complex multi-image, video, and 3D understanding tasks. Developed as a foundational element for the LLaVA-NeXT-Interleave project, M4-Instruct emphasizes instruction tuning in a unified interleaved image–text format, systematically covering diverse real-world domains and visual-linguistic scenarios. The dataset contains approximately 1,177,600 samples and addresses critical gaps in prior multimodal datasets by enabling simultaneous training across four major visual modalities: multi-image (still images), multi-frame (video), multi-view (3D), and multi-patch (single-image subregions) (Li et al., 10 Jul 2024).

1. Dataset Composition and Structure

M4-Instruct is organized around four principal domains, each corresponding to a distinct modality:

• Multi-image: Collections of real-world images presented in interleaved order, suitable for tasks involving comparative reasoning, difference detection, and story composition.
• Multi-frame: Segmented video data, where input consists of multiple temporal frames, enabling motion or event understanding.
• Multi-view: Sets of images from varying perspectives of the same 3D scene or object, geared toward 3D spatial reasoning.
• Multi-patch: Subregions extracted from single high-resolution images; this incorporates standard single-image scenarios.

Across these, M4-Instruct is systematically partitioned into 14 tasks sourced from 41 heterogeneous datasets. Task categories include "Spot the Difference" (with real-world, synthetic, surveillance, and bird-to-word variants), "Image Edit Instruction" (drawing from HQ-Edit, MagicBrush, IEdit), "Visual Story Telling" (including cartoon-based datasets like AESOP, Flintstones, Pororo, and realistic VIST images), "Text-rich VQA" (spanning WebQA, textbook-structured VQA, OCR-based, and document-centric queries), as well as multi-image puzzle and low-level visual comparison.

A significant portion—approximately 40% of the stage-2 fine-tuning data—is sampled from the LLaVA-NeXT single-image collection to maintain strong performance in conventional visual question answering.

Domain	Example Tasks	Source Types
Multi-image	Spot the Difference, Image Edit Instruction	Realistic, synthetic, surveillance
Multi-frame	Video-based VQA	Segmented video
Multi-view	3D spatial tasks	Rendered/scanned objects
Multi-patch	Single-image VQA	High-res image subdivision

This diversity ensures that M4-Instruct provides broad coverage of data types, visual styles, and task formulations.

2. Dataset Integration in Model Training

M4-Instruct serves as the core resource for instruction tuning in the LLaVA-NeXT-Interleave model pipeline. The dataset is fully unified under an interleaved image–text format, wherein sequences of image placeholders and text prompts are linearly combined. This design establishes a generic templating mechanism allowing a single LMM architecture to ingest examples from any of the M4 domains and task types with minimal data- or domain-specific preprocessing.

The training protocol commences from a pretrained LLaVA-NeXT-Image checkpoint and continues via heavily multi-image, multi-frame, and 3D instruction tuning using the M4-Instruct samples. The strategy includes mixed data formatting: both "in-the-front" (all image tokens concatenated at the start) and "fully interleaved" (images appearing in natural order with text) layouts are employed, increasing model robustness to input arrangement and enhancing downstream adaptability. This joint training regime—exposing the model to tasks from all four domains within a common batch—compels the formation of internal representations supporting task and modality transfer.

A key technical point is the simultaneous inclusion of tasks from video, 3D, and single-image sources, structuring the learning dynamic for maximal cross-modal synergy.

3. Evaluation Protocol and Benchmarks

To rigorously assess LMM performance post-training on M4-Instruct, the LLaVA-Interleave Bench is curated. This benchmark divides evaluation into two principal groupings:

• In-domain: Tasks derived directly from the training set (e.g., Spot the Difference and multi-image VQA) to measure retention and overfitting.
• Out-domain: Tasks never explicitly seen during fine-tuning, including mathematical and scientific diagram understanding (MathVerse-mv, SciVerse-mv), as well as popular multimodal evaluation sets such as Mantis-Eval and BLINK.

This evaluative structure enables comprehensive measurement of both generalization and memorization. Detailed tables in the original paper enumerate scenarios, task types, and sample counts. By ensuring inclusion of both curated and publicly available datasets, M4-Instruct’s benchmark design establishes high experimental transparency and reproducibility.

4. Emerging Model Capabilities and Task Transfer

Training on the M4-Instruct dataset engenders several documented emergent behaviors in LMMs:

• Skill transfer between domains, such as leveraging single-image humor analysis capacities in multi-image correlation or "fun part" classification, and the ability to adapt Twitter-style captioning learned from single-image tasks to video sequence inputs.
• Zero-shot cross-modal generalization, enabling models to perform previously unseen tasks (e.g., synthesizing key content across multiple document images or identifying nuanced artistic styles) even in test settings under-represented in the supervised training corpus.

These emerging capabilities underline the value of a pooled, richly annotated, and interleaved training architecture for compositional generalization and cross-task learning in LMMs.

5. Technical Implementation Details

The unified M4-Instruct format consists of sequentially interleaved image tokens (or their proxies for multi-frame/multi-view/multi-patch) and text instructions. The underlying model architecture mirrors the core LLaVA-NeXT pipeline, including:

• Vision encoder: SigLIP-400M with input resolution 384×384.
• Intermediate projection: Lightweight two-layer MLP.
• Language backbone: Qwen-based LLM.

In multi-frame (video) contexts, the tokenization strategy includes a "Pooling to 1/4" technique, reducing feature map dimensionality (e.g., "40 × 729 × 1/4 = 10 × 729") for computational efficiency while retaining critical spatial-temporal relationships. During training, 10 frames are sampled per video; raising this to 16 frames at inference has been shown to further enhance model performance.

LaTeX notation present in the original work, such as:

• $40 \times 729 \times \frac{1}{4} = 10 \times 729$ is provided to clarify pooling mechanics and formalize the dimensionality reductions central to efficient LMM computation.

6. Significance and Outlook

The introduction of M4-Instruct marks a significant shift toward highly generalizable, instruction-tuned multimodal models. Its multi-domain coverage, emphasis on interleaved representation, and joint training enable capabilities not achievable with previous single-modality or task-segregated datasets. The role of M4-Instruct in facilitating emergent behaviors, model robustness to input permutations, and task transfer underscores its technical significance for ongoing research in vision-LLMing (Li et al., 10 Jul 2024).

This suggests a trend toward more unified, large-scale multimodal corpora supporting instruction tuning across diverse vision-language domains. A plausible implication is that future LMM development will increasingly rely on M4-Instruct-like datasets, exploiting compositional data synthesis and flexible interface design to drive generalization and scalability in synthetic multimodal intelligence.