IRGL-300K: A Large-Scale Multimodal Dataset

• IRGL-300K is a curated dataset of approximately 300K interleaved text–image samples that advance interleaved reasoning generation in T2I models.
• It interleaves multi-step chain-of-thought reasoning with 1024×1024 image synthesis, supporting both initial and refined output modes.
• Utilizing six decomposed learning modes and specific loss functions, IRGL-300K improves instruction following, semantic fidelity, and visual detail.

IRGL-300K is a curated, large-scale dataset comprising approximately 300,000 interleaved text–image samples specifically designed to advance Interleaving Reasoning Generation (IRG) frameworks in text-to-image (T2I) generation. The corpus is engineered to facilitate joint training of “think-and-generate” and “reflect-and-refine” multimodal models, supporting both pure reasoning and full trajectory learning, and enabling robust improvement in instruction following, semantic fidelity, and visual detail (Huang et al., 8 Sep 2025).

1. Structure and Composition of IRGL-300K

Each IRGL-300K sample consists of a sequence that interleaves natural language reasoning and high-resolution images:

• Prompt ($T_{\text{in}}$): A descriptive natural language instruction (e.g., “A wooden table set for four under a gazebo in a garden”).
• Initial Reasoning ($T_{\text{out}}^{(1)}$): An explicit, multi-step chain-of-thought capturing the initial interpretive process.
• Initial Image ($I_{\text{out}}^{(1)}$): A 1024×1024 image synthesized under the guidance of $T_{\text{out}}^{(1)}$.
• Improving Reasoning ($T_{\text{out}}^{(2)}$): Reflective chain-of-thought reasoning focused on analyzing $I_{\text{out}}^{(1)}$, articulating improvements, and guiding further synthesis.
• Refined Image ($I_{\text{out}}^{(2)}$): An improved image generated by incorporating the guidance from $T_{\text{out}}^{(2)}$.

Samples also include modality embeddings (ViT/VAE) for all images, facilitating mixed-modality and efficient representation learning.

The dataset is partitioned into six “decomposed learning modes” representing distinct supervision and data-construction pipelines, ensuring broad coverage of both reasoning and generative subtasks:

Learning Mode	Input Modalities	Supervision Target
Initial Understanding	Prompt, Image	Initial Reasoning ($T_{\text{out}}^{(1)}$)
Initial Generation	Prompt	Initial Reasoning ($T_{\text{out}}^{(1)}$)
Initial Full	Prompt	$T_{\text{out}}^{(1)}$0, $T_{\text{out}}^{(1)}$1
Improving Understanding	Prompt, Two Images	Improving Reasoning ($T_{\text{out}}^{(1)}$2)
Improving Generation	Prompt, $T_{\text{out}}^{(1)}$3, $T_{\text{out}}^{(1)}$4	Improving Reasoning ($T_{\text{out}}^{(1)}$5)
Improving Full	Prompt, $T_{\text{out}}^{(1)}$6, $T_{\text{out}}^{(1)}$7	$T_{\text{out}}^{(1)}$8, $T_{\text{out}}^{(1)}$9

The balance of pure text-only sequences (≈200K) and full, paired reasoning–image trajectories (≈90K) ensures efficient pretraining and targeted fine-tuning.

2. Data Generation and Annotation Pipelines

IRGL-300K employs a mixture of automated and semi-automated pipelines leveraging state-of-the-art models for both natural language and vision:

• Initial Full Data Construction: Prompts are input to GPT-4o, producing a high-quality image $I_{\text{out}}^{(1)}$0. Large multimodal LLMs (MLLM) generate $I_{\text{out}}^{(1)}$1 chain-of-thought that justifies the image choice.
• Improving Full Data Construction: Prompts are passed to a base model (BAGEL, Qwen3) to generate $I_{\text{out}}^{(1)}$2 and $I_{\text{out}}^{(1)}$3. GPT-4o then produces a higher-quality $I_{\text{out}}^{(1)}$4. An MLLM writes the improvement reasoning $I_{\text{out}}^{(1)}$5.
• Understanding Modes: Feature-encoding (ViT/VAE) representations of images are provided as input, with templates used to elicit explicit “understanding” rationales for both initial and improved images or image pairs.
• Generation Modes: The model is prompted to generate chain-of-thought reasoning given natural language or earlier outputs, supporting generalization during inference.

All image outputs are at 1024×1024 resolution for consistency across modalities. No official held-out splits are provided; training/validation partitioning is done by random subsampling (≈10% for validation).

3. Statistical Characteristics and Modalities

IRGL-300K provides coverage of a range of prompt genres and output complexities:

• Reasoning step counts: Initial chains average 4 steps; improvement chains span 4–6 steps.
• Text lengths: Reasoning outputs are 3–7 sentences (20–60 tokens).
• Modality distribution: ~200K text-only reasoning samples, ~90K paired text–image trajectories with full testable interleaving.
• Feature storage: All images are accompanied by ViT (Vision Transformer) and VAE (Variational Autoencoder) embeddings to facilitate flexible multimodal training and efficient sampling.

Each sample encodes full “think, generate, reflect, regenerate” trajectories, encouraging rich, compositional learning and detailed semantic control.

4. Supervision Objectives and Training Regime

Training with IRGL-300K applies distinct loss functions to each output modality:

• Text supervision: All reasoning outputs are subject to next-token cross-entropy (CE), $I_{\text{out}}^{(1)}$6.
• Image supervision: All generated images use mean-square error (MSE) against distilled VAE image latents, $I_{\text{out}}^{(1)}$7.
• Overall objective: For a full IRGL-300K sample, $I_{\text{out}}^{(1)}$8 with $I_{\text{out}}^{(1)}$9 empirically.

Training proceeds in two stages:

1. Reasoning-Focused Fine-Tuning (Stage 1): 2,000 optimizer steps over all six learning modes, emphasizing robust, generalizable chain-of-thought production and baseline image synthesis.
2. End-to-End IRG Tuning (Stage 2): 30,000 steps on the two full modes (Initial Full, Improving Full). This stage targets the accurate realization of guided visual refinements and coherent two-step pipelines.

5. Coverage of Learning Paradigms

Each of the six decomposed modes targets a unique subcomponent:

• Understanding Modes (Initial, Improving): Model learns to “read” images or image pairs and articulate corresponding rationales or improvement strategies.
• Generation Modes (Initial, Improving): Model learns to generate chain-of-thought from prompts and prior outputs, including reflecting without access to oracle-improved images.
• Full Modes: End-to-end trajectories map prompt (and intermediates) to final high-quality images, embedding both interpretive and generative expertise.

This modularity supports data-efficient learning. For instance, text-only modes allow the model to generalize reasoning steps prior to heavy compute on image synthesis.

6. Release, Licensing, and Availability

IRGL-300K, all associated model weights, and training code are released under a liberal open-source license (Apache 2.0 or MIT), with no embargo. All assets are accessible at https://github.com/Osilly/Interleaving-Reasoning-Generation. The release follows the paper’s availability on arXiv and/or acceptance to ICLR (Huang et al., 8 Sep 2025).

7. Context, Evaluation Utility, and Implications

IRGL-300K was developed as a response to observed limitations in contemporary unified T2I models, particularly in instruction following and detail retention relative to systems tightly integrating chain-of-thought reasoning with generative modeling (e.g., GPT-4o).

Empirical results indicate that models trained with IRGL-300K achieve state-of-the-art performance, with absolute gains of 5–10 points on benchmarks such as GenEval, WISE, TIIF, GenAI-Bench, and OneIG-EN. Substantial improvements are observed in visual quality and fine-grained semantic fidelity.

A plausible implication is that the dataset’s paradigm of decomposing T2I into explicit interleaved reasoning and iterative refinement is a scalable path for next-generation multimodal generative systems, supporting robust instruction following, nuanced visual composition, and interpretable pipeline dynamics at scale.