--- title: 'UnifiedVisual-240K: Unified Vision-Language Dataset' url: https://www.emergentmind.com/topics/unifiedvisual-240k type: topic --- # UnifiedVisual-240K: Unified Vision-Language Dataset UnifiedVisual-240K is a 240K-sample unified vision-language dataset built under the UnifiedVisual framework for unified vision large language models (VLLMs) that perform both multimodal understanding and generation within a single autoregressive paradigm. It combines **120K multimodal generation/reasoning samples** newly constructed under UnifiedVisual with **120K understanding samples** drawn from **60K LLaVA-CoT** and **60K CoT-Collection**, and it is explicitly designed to make understanding and generation *mutually reinforcing* rather than competitive during training [2509.14738]. ## 1. Conceptual position and design rationale UnifiedVisual-240K addresses a specific failure mode in unified VLLM training: understanding datasets are typically abundant and reasoning-rich but terminate in text outputs, whereas generation datasets are often limited to simple prompt-to-image mappings with weak explicit reasoning and weak image-text alignment. Under naive mixing, adding generation data can degrade understanding performance. UnifiedVisual was introduced to counter that imbalance by constructing data in which visual generation, multimodal reasoning, and interleaved image-text processing share a common instruction-response interface and a common autoregressive training regime [2509.14738]. The framework assumes four broad task types: **Text $\rightarrow$ Text**, **Image/Text $\rightarrow$ Text**, **Text $\rightarrow$ Image**, and **Text/Image $\rightarrow$ Text+Image**. This is not merely a notation choice. It encodes the central claim of UnifiedVisual: the same model should process textual rationales, visual inputs, visual outputs, and interleaved multimodal explanations without task-specific heads. A plausible implication is that the dataset is aimed less at isolated benchmark optimization than at reducing the architectural and data-format discontinuities that traditionally separate VQA-style systems from text-to-image systems. A second design principle is that generation samples should themselves contain reasoning. Accordingly, many UnifiedVisual instances separate **Instruction** from **Thought (Rationale)**, so that text guides image creation and generated images can re-enter the reasoning chain. This suggests that UnifiedVisual-240K is not simply a collection of paired examples, but a dataset engineered around multimodal chain-of-thought as a training substrate for unified autoregressive models. ## 2. Construction framework and task families UnifiedVisual organizes its generation-side data along three axes: **Visual Generation**, **Multimodal Reasoning**, and **Multimodal Internet Data** [2509.14738]. Within **Visual Generation**, the framework includes **Image Generation**, **Image Editing**, and **Image Correction**. Image generation itself has two paradigms. In **topic- and scene-based generation**, a topic is converted into an implicit caption, similar captions are filtered for diversity with `text-embedding-ada-002`, **GPT-4** produces a rationale, and that rationale is turned into a **DALL-E-3** prompt that yields the image. In **category- and image-based generation**, the starting point is a real image from **OpenImages**; **GPT-4** writes indirect instructions that describe the semantics without naming the category, then generates a rationale that explains how the image should be inferred from the description and category. In both cases the output is effectively a text-to-image sample augmented with explicit reasoning. **Image Editing** extends basic before/after editing corpora such as **MagicBrush** by rewriting simple commands into more nuanced instructions and then using **GPT-4o** to analyze the original image, the edited image, and the revised instruction in order to generate a rationale describing the required changes. **Image Correction** is more adversarial. Starting from captioned images in **ShareGPT4V**, the caption is perturbed, **Stable Diffusion 3.5** generates an intentionally inconsistent image from the modified caption, and **GPT-4o** diagnoses the mismatches and proposes corrections that are then used to obtain a corrected image. This produces supervision for detecting and repairing image-text inconsistency. The **Multimodal Reasoning** branch contains three variants. **MM Reasoning (O)** performs reasoning over the original image, in a Visual-CoT style, with rationales that attend to cropped or described regions of the input image. **MM Reasoning (MM)** uses multimodal input and multimodal rationales: images are collected from **COCO**, deduplicated via **CLIP** `clip-vit-large-patch14`, and **GPT-4o** generates reasoning questions and rationales interleaving text with `[image: description]` snippets; **GPT-4** turns those descriptions into keywords for **Bing/Google** retrieval, and CLIP selects the best-aligned supporting images. **MM Reasoning (T)** begins with text-only questions that require visual aids; **GPT-4** writes a chain-of-thought containing `[image: ...]` descriptions, and **DALL-E-3** renders those descriptions into images. This trains the model to produce visual supports even when the input is purely textual. The **MM Internet** subset is constructed from a large interleaved web image-text corpus inspired by **CoMM**. Pre-trained VLLMs filter for sentence-image coherence and semantic consistency, and each cleaned interleaved document is converted into a QA-style sample whose answer is the original interleaved content with `` markers. This brings naturally occurring multimodal documents into the same instruction-response format. ## 3. Dataset composition, representation, and quality control UnifiedVisual-240K consists of an approximately balanced **120K understanding / 120K generation** mixture. The generation subset is internally heterogeneous: | Generation subset | Count | |---|---:| | MM Internet | 29,399 | | Image Editing | 9,024 | | Image Generation | 22,755 | | Image Correction | 20,000 | | MM Reasoning (O) | 21,000 | | MM Reasoning (T) | 7,276 | | MM Reasoning (MM) | 17,761 | The understanding half is sampled as **60K LLaVA-CoT** plus **60K CoT-Collection** [2509.14738]. Across both halves, all data are converted into a common instruction-response schema. In practice, instances may take forms such as $(x_{\text{text}}, y_{\text{text}})$, $(x_{\text{image}}, x_{\text{text}}, y_{\text{text}})$, $(x_{\text{text}}, y_{\text{text}}, y_{\text{image}})$, or $(x_{\text{image}}, x_{\text{text}}, y_{\text{text}}, y_{\text{image}})$. For interleaved outputs, images are serialized into the response using `[BOI]` and `[EOI]` markers. The paper also specifies a quality-control stack. Text deduplication uses `text-embedding-ada-002` embeddings and cosine similarity, $$ \text{sim}(e_i,e_j)=\frac{e_i^\top e_j}{\|e_i\|\|e_j\|}, $$ to remove near-duplicate captions and questions. When retrieved images are needed for multimodal rationales, CLIP similarity is computed as $$ s_k=\text{CLIP}(t, I_k), $$ and the highest-scoring images are retained. MM Internet documents are filtered by pre-trained VLLMs for coherence and semantic alignment. In Image Correction, caption perturbation followed by inconsistency diagnosis forces rationales to reference concrete mismatches such as colors, positions, and counts. This suggests that the dataset’s notion of quality is oriented toward alignment and reasoning structure, not only toward visual aesthetics. ## 4. Unified autoregressive modeling and training interface The main experimental model used with UnifiedVisual-240K is **Anole**, a 7B-scale autoregressive transformer-based unified multimodal model. Its pipeline includes a **vision encoder / visual tokenizer** that converts images into discrete visual tokens, a single **LLM backbone** that consumes mixed text and image-token sequences, and an **image decoder** that reconstructs images from generated visual tokens. `[BOI]` and `[EOI]` mark image spans in the token stream, allowing arbitrary interleavings of image and text in both input and output [2509.14738]. Training reduces all tasks to next-token prediction over a multimodal sequence $(x_1,\dots,x_T)$. The generic loss is $$ \mathcal{L}=-\sum_{t=1}^{T}\log p_\theta(x_t\mid x_{