UnifiedVisual-240K: Unified Vision-Language Dataset

• The dataset unifies multimodal understanding and generation by integrating reasoning-rich text with image outputs under a common instruction-response paradigm.
• It employs a balanced mix of 120K understanding and 120K generation samples, using tasks like image generation, editing, and correction to enhance visual-text alignment.
• Empirical results demonstrate that models trained on UnifiedVisual-240K significantly outperform conventional mixes in both understanding scores and generation quality.

UnifiedVisual-240K is a 240K-sample unified vision-language dataset built under the UnifiedVisual framework for unified vision LLMs (VLLMs) that perform both multimodal understanding and generation within a single autoregressive paradigm. It combines 120K multimodal generation/reasoning samples newly constructed under UnifiedVisual with 120K understanding samples drawn from 60K LLaVA-CoT and 60K CoT-Collection, and it is explicitly designed to make understanding and generation mutually reinforcing rather than competitive during training (Wang et al., 18 Sep 2025).

1. Conceptual position and design rationale

UnifiedVisual-240K addresses a specific failure mode in unified VLLM training: understanding datasets are typically abundant and reasoning-rich but terminate in text outputs, whereas generation datasets are often limited to simple prompt-to-image mappings with weak explicit reasoning and weak image-text alignment. Under naive mixing, adding generation data can degrade understanding performance. UnifiedVisual was introduced to counter that imbalance by constructing data in which visual generation, multimodal reasoning, and interleaved image-text processing share a common instruction-response interface and a common autoregressive training regime (Wang et al., 18 Sep 2025).

The framework assumes four broad task types: Text $\rightarrow$ Text, Image/Text $\rightarrow$ Text, Text $\rightarrow$ Image, and Text/Image $\rightarrow$ Text+Image. This is not merely a notation choice. It encodes the central claim of UnifiedVisual: the same model should process textual rationales, visual inputs, visual outputs, and interleaved multimodal explanations without task-specific heads. A plausible implication is that the dataset is aimed less at isolated benchmark optimization than at reducing the architectural and data-format discontinuities that traditionally separate VQA-style systems from text-to-image systems.

A second design principle is that generation samples should themselves contain reasoning. Accordingly, many UnifiedVisual instances separate Instruction from Thought (Rationale), so that text guides image creation and generated images can re-enter the reasoning chain. This suggests that UnifiedVisual-240K is not simply a collection of paired examples, but a dataset engineered around multimodal chain-of-thought as a training substrate for unified autoregressive models.

2. Construction framework and task families

UnifiedVisual organizes its generation-side data along three axes: Visual Generation, Multimodal Reasoning, and Multimodal Internet Data (Wang et al., 18 Sep 2025).

Within Visual Generation, the framework includes Image Generation, Image Editing, and Image Correction. Image generation itself has two paradigms. In topic- and scene-based generation, a topic is converted into an implicit caption, similar captions are filtered for diversity with text-embedding-ada-002, GPT-4 produces a rationale, and that rationale is turned into a DALL-E-3 prompt that yields the image. In category- and image-based generation, the starting point is a real image from OpenImages; GPT-4 writes indirect instructions that describe the semantics without naming the category, then generates a rationale that explains how the image should be inferred from the description and category. In both cases the output is effectively a text-to-image sample augmented with explicit reasoning.

Image Editing extends basic before/after editing corpora such as MagicBrush by rewriting simple commands into more nuanced instructions and then using GPT-4o to analyze the original image, the edited image, and the revised instruction in order to generate a rationale describing the required changes. Image Correction is more adversarial. Starting from captioned images in ShareGPT4V, the caption is perturbed, Stable Diffusion 3.5 generates an intentionally inconsistent image from the modified caption, and GPT-4o diagnoses the mismatches and proposes corrections that are then used to obtain a corrected image. This produces supervision for detecting and repairing image-text inconsistency.

The Multimodal Reasoning branch contains three variants. MM Reasoning (O) performs reasoning over the original image, in a Visual-CoT style, with rationales that attend to cropped or described regions of the input image. MM Reasoning (MM) uses multimodal input and multimodal rationales: images are collected from COCO, deduplicated via CLIP clip-vit-large-patch14, and GPT-4o generates reasoning questions and rationales interleaving text with [image: description] snippets; GPT-4 turns those descriptions into keywords for Bing/Google retrieval, and CLIP selects the best-aligned supporting images. MM Reasoning (T) begins with text-only questions that require visual aids; GPT-4 writes a chain-of-thought containing [image: ...] descriptions, and DALL-E-3 renders those descriptions into images. This trains the model to produce visual supports even when the input is purely textual.

The MM Internet subset is constructed from a large interleaved web image-text corpus inspired by CoMM. Pre-trained VLLMs filter for sentence-image coherence and semantic consistency, and each cleaned interleaved document is converted into a QA-style sample whose answer is the original interleaved content with <image_placeholder> markers. This brings naturally occurring multimodal documents into the same instruction-response format.

3. Dataset composition, representation, and quality control

UnifiedVisual-240K consists of an approximately balanced 120K understanding / 120K generation mixture. The generation subset is internally heterogeneous:

Generation subset	Count
MM Internet	29,399
Image Editing	9,024
Image Generation	22,755
Image Correction	20,000
MM Reasoning (O)	21,000
MM Reasoning (T)	7,276
MM Reasoning (MM)	17,761

The understanding half is sampled as 60K LLaVA-CoT plus 60K CoT-Collection (Wang et al., 18 Sep 2025). Across both halves, all data are converted into a common instruction-response schema. In practice, instances may take forms such as $(x_{\text{text}}, y_{\text{text}})$, $$(x_{\text{image}}, x_{\text{text}}, y_{\text{text}})$$, $$(x_{\text{text}}, y_{\text{text}}, y_{\text{image}})$$, or $$(x_{\text{image}}, x_{\text{text}}, y_{\text{text}}, y_{\text{image}})$$. For interleaved outputs, images are serialized into the response using [BOI] and [EOI] markers.

The paper also specifies a quality-control stack. Text deduplication uses text-embedding-ada-002 embeddings and cosine similarity,

$$\text{sim}(e_i,e_j)=\frac{e_i^\top e_j}{\|e_i\|\|e_j\|},$$

to remove near-duplicate captions and questions. When retrieved images are needed for multimodal rationales, CLIP similarity is computed as

$s_k=\text{CLIP}(t, I_k),$

and the highest-scoring images are retained. MM Internet documents are filtered by pre-trained VLLMs for coherence and semantic alignment. In Image Correction, caption perturbation followed by inconsistency diagnosis forces rationales to reference concrete mismatches such as colors, positions, and counts. This suggests that the dataset’s notion of quality is oriented toward alignment and reasoning structure, not only toward visual aesthetics.

4. Unified autoregressive modeling and training interface

The main experimental model used with UnifiedVisual-240K is Anole, a 7B-scale autoregressive transformer-based unified multimodal model. Its pipeline includes a vision encoder / visual tokenizer that converts images into discrete visual tokens, a single LLM backbone that consumes mixed text and image-token sequences, and an image decoder that reconstructs images from generated visual tokens. [BOI] and [EOI] mark image spans in the token stream, allowing arbitrary interleavings of image and text in both input and output (Wang et al., 18 Sep 2025).

Training reduces all tasks to next-token prediction over a multimodal sequence $\rightarrow$0. The generic loss is

$\rightarrow$1

but UnifiedVisual applies modality-selective loss to avoid conflicts between text-token and image-token supervision. If $\rightarrow$2 indicates whether token $\rightarrow$3 belongs to the supervised modality at step $\rightarrow$4, the objective becomes

$\rightarrow$5

During text segments, only text-token loss is retained; during image generation segments between [BOI] and [EOI], only visual-token loss is retained.

The reported training configuration uses 64 $\rightarrow$6 NVIDIA H100 80G, batch size 512, maximum sequence length 4096, and AdamW with 5% warmup, a cosine decay learning-rate scheduler, and peak LR $\rightarrow$7. The model is trained for 2 epochs over UnifiedVisual-240K, and decoding is greedy before image tokens are passed to Anole’s image decoder. The practical significance of this setup is that the same transformer objective is used for VQA, textual reasoning, text-to-image generation, editing, correction, and multimodal rationale generation, differing only in sequence layout and masking.

5. Empirical behavior and the claim of mutual reinforcement

The core empirical claim associated with UnifiedVisual-240K is that a carefully constructed mixture of understanding and generation data improves both capabilities simultaneously. On multimodal understanding, the paper reports the following average scores:

Model	Avg. understanding score
Anole	33.4
Anole-NormalData	39.9
Anole-UnifiedVisual$\rightarrow$8	43.7
Anole-UnifiedVisual$\rightarrow$9	40.9
Anole-UnifiedVisual	46.3

These comparisons are central. Anole-NormalData, which mixes the same understanding subset with Laion-derived generation data converted following Metamorph, underperforms Anole-UnifiedVisual$\rightarrow$0, showing that conventional generation data can hurt understanding. By contrast, Anole-UnifiedVisual$\rightarrow$1, trained only on the 120K UnifiedVisual generation subset, achieves 40.9, exceeding Anole-NormalData despite having no explicit understanding data. The full Anole-UnifiedVisual model reaches 46.3, outperforming all partial-data variants on the understanding benchmarks RealWorldQA, MMVP, ScienceQA, VStar, MME, and POPE (Wang et al., 18 Sep 2025).

On generation, the paper reports that Anole-UnifiedVisual outperforms Anole-NormalData on overall GenEval and COCO CLIPScore, with especially large gains in single/double object generation, color correctness, and quantity correctness. The interpretation given is that richer multimodal understanding improves control over generated attributes. A hybrid model using the same understanding data plus 50% Laion and 50% UnifiedVisual generation data outperforms both the pure NormalData and pure UnifiedVisual settings, which suggests that high-aesthetic imagery and reasoning-rich generation data are complementary rather than exclusive.

The paper further states that Anole-UnifiedVisual achieves the highest AlpacaEval win rate against Anole-NormalData, indicating benefits even for pure text instruction-following and reasoning. The same dataset design transfers across model families: retraining Emu3-Stage1 and Liquid on UnifiedVisual-240K improves both understanding and generation relative to NormalData. Ablations that vary one side of the dataset while holding the other fixed show monotonic gains in generation as understanding data increases, and monotonic gains in understanding as generation data increases; the reported curves do not plateau.

6. Relation to nearby 240K-scale resources, limitations, and use

The name UnifiedVisual-240K specifically refers to the dataset introduced with the UnifiedVisual framework. It should not be conflated with other 240K-scale multimodal resources. In particular, Ultimate3D is a distinct dataset of 240K VQA pairs for camera-object relations, and its paper explicitly states that it does not use the name UnifiedVisual-240K (He et al., 11 Jul 2025). Likewise, ADARI is a separate large-scale vision-language dataset for creative practice containing 241,982 images and 264,028 raw sentences, aimed at subjective description and multimodal word sense disambiguation rather than unified understanding-generation training (Guevara et al., 2020).

Within its own scope, UnifiedVisual-240K has several stated limitations. At 240K samples, it is small relative to multi-billion-scale pretraining corpora. Some generation samples, especially those involving Stable Diffusion, may be lower in image quality than high-aesthetic Laion data. The dataset also inherits biases from its sources—COCO, OpenImages, ShareGPT4V, web data, and GPT-4 prompts—and these biases are not exhaustively audited. The paper does not report a formal fairness analysis (Wang et al., 18 Sep 2025).

The announced use of UnifiedVisual-240K is correspondingly twofold. First, it functions as a template and seed for larger unified datasets built under the same framework. Second, it serves as a pretraining / instruction-tuning corpus and a benchmarking bed for research on multimodal chain-of-thought, multimodal reasoning, and text-to-image alignment. The code and datasets are announced at https://github.com/fnlp-vision/UnifiedVisual, although the paper does not specify a license and indicates that source-dataset constraints should be checked in the repository. A plausible implication is that UnifiedVisual-240K is best understood not only as a fixed dataset release but as an existence proof for a particular data-construction doctrine: unified VLLMs benefit when generation examples are made reasoning-rich and when reasoning examples are allowed to produce images.