--- title: 'DuoGen: Interleaved Multimodal Generation' url: https://www.emergentmind.com/topics/duogen type: topic --- # DuoGen: Interleaved Multimodal Generation Searching arXiv for DuoGen and closely related interleaved multimodal generation papers for citation support. DuoGen is a modular framework for **general-purpose interleaved multimodal generation** in which a model produces mixed sequences of text and images in alternating order, with each new output conditioned on the full preceding multimodal context. Introduced in "DuoGen: Towards General Purpose Interleaved Multimodal Generation" [2602.00508], it targets tasks such as step-by-step instructional guides, visual planning, image-conditioned generation, and image editing, and is defined by a design in which a pretrained multimodal large language model (MLLM) governs textual continuation and modality switching while a pretrained diffusion transformer (DiT) synthesizes images when invoked. ## 1. Definition and task formulation DuoGen addresses a setting in which the output is not restricted to a single modality. The objective is to generate a continuation of an interleaved context that may contain both text and images, with the system itself deciding when a visual output is required. In the paper’s operational description, the interleaved prefix before a new image is represented as \[ \{T_1, I_1, T_2, I_2, \cdots, T_N\}, \] where \(T_i\) are text chunks and \(I_i\) are images [2602.00508]. This formulation differs from conventional text-only assistants, standard text-to-image pipelines, and visual-understanding systems. A general-purpose interleaved model must jointly support instruction following, visual understanding of user-supplied images, coherent language generation across steps, image generation aligned with earlier text and images, continuation of partially given multimodal sequences, and editing or manipulation where previous images remain active context. DuoGen is explicitly designed for this broader problem class rather than for isolated text generation or isolated image synthesis alone [2602.00508]. A central conceptual distinction is that DuoGen is not an early-fusion model pretrained from scratch on mixed multimodal corpora. Instead, it asks whether interleaved alignment can be built directly on top of strong pretrained modules. This suggests a different systems view of multimodal generation: rather than collapsing understanding and rendering into a single pretraining regime, it separates control, reasoning, and visual synthesis while preserving a unified interaction loop. ## 2. Data curation and instruction-tuning corpus DuoGen’s data strategy is organized around two regimes: a curated instruction-tuning dataset for training the MLLM to follow interleaved multimodal instructions, and a much larger context-alignment corpus for teaching the image generator to remain consistent with preceding text and images [2602.00508]. The instruction-tuning set contains **298k interleaved conversation samples**, comprising **268k** conversations rewritten from curated websites and **30k** high-quality synthetic interleaved samples. The website source includes **StoryBird**, **Instructables**, **eHow**, and raw data reused from **CoMM**. The curation pipeline removes pages containing only text and discards invalid images such as **QR codes**, **icons**, and **advertisements**. Main webpage content is converted into Markdown, then processed through four stages: rewrite and split raw interleaved content; caption and categorize all images; remove duplicate images and reorder text-image chunks; and convert the cleaned sequence into a user–assistant conversation [2602.00508]. The synthetic subset is introduced because website images are often inconsistent in quality, resolution, style, and step granularity. To address this, the authors build a hierarchical prompt pool over **8 broad domains**—**Sports**, **Outdoor & Survival**, **DIY & Crafting**, **Vehicle & Transportation**, **Personal Care & Health**, **Farm, Pet, and Animals**, **Home & Living**, and **Office & Productivity**—which human annotators refine into **151 subcategories** with around **10 seed questions per subcategory**, for a total of **1,500 seed prompts**. These are expanded using **OpenAI O3** into **15,270 diverse instructions**. The paper also reports a cooking-focused synthetic branch using **15k dish images** sampled from **MM-Food-100k**, with invalid images and invalid dish names filtered out [2602.00508]. For the second training stage, DuoGen uses a substantially larger alignment mixture. The video-derived component begins from **5 million raw videos**, segmented into **5-second clips**, filtered with scene detection for temporal consistency, and annotated with **Qwen2.5-VL-32B** to describe object motion, human actions, and camera movements. The supplement further enumerates **939k** text-to-image alignment samples, **5,586k** image editing samples, and **5,657k** interleaved context alignment samples. The paper is explicit that the clearest core instruction-tuning set is the **298k** high-quality conversation corpus, whereas the second-stage alignment mixture is broader and more heterogeneous [2602.00508]. A concise summary of the principal data layers is as follows: | Data component | Scale | Role | |---|---:|---| | Rewritten website conversations | 268k | Stage-1 instruction tuning | | Synthetic interleaved conversations | 30k | Stage-1 instruction tuning | | Raw videos for transition mining | 5 million | Stage-2 context alignment | | Text-to-image alignment data | 939k | Stage-2 image alignment | | Image editing data | 5,586k | Stage-2 image alignment | | Interleaved context alignment data | 5,657k | Stage-2 image alignment | This curation pipeline is one of DuoGen’s defining features. The empirical ablation in the paper indicates that data cleaning and conversation rewriting are not auxiliary conveniences but major determinants of final performance. ## 3. Architecture and multimodal interface DuoGen combines two pretrained components: an MLLM for understanding images, following instructions, generating text, and deciding when an image should be produced; and a DiT, initialized from a video generation model, for generating the next image conditioned on the full prior interleaved context. In the reported implementation, the backbones are **Qwen2.5-VL 7B** for the MLLM and **Cosmos Predict 2.5 (2B)** for the DiT [2602.00508]. The modality-switching mechanism is the special token **``** (“Begin-of-Vision”). The MLLM autoregressively predicts text tokens until it emits ``, at which point control passes to the image generator. The newly synthesized image is then appended back into the multimodal history, after which text generation resumes. This yields a recurrent loop: **user input → text generation → `` → image generation → append image → continue text** [2602.00508]. Conditioning of the DiT is bimodal. First, all previous images in the context—both user-provided and previously generated—are stacked as condition frames, encoded by the VAE encoder, and concatenated with the noisy latent of the target image along the temporal axis, following **Cosmos Predict 2.5**’s strategy. Second, the model extracts the MLLM hidden states corresponding to all multimodal tokens before ``, concatenates hidden states from **all decoder layers** along the channel dimension, and passes them through a lightweight **connector** that projects them into the DiT conditioning space. At each DiT decoder layer, guidance is injected by **cross-attention** between image latents and these projected language embeddings [2602.00508]. This architecture is explicitly modular. A plausible implication is that the paper treats interleaved generation less as a monolithic pretraining problem than as an interface problem between reasoning and rendering modules. The use of a **video-pretrained DiT** is particularly consequential, because interleaved generation often requires consistency across multiple previous images rather than conditioning on a single prompt or a single reference frame. ## 4. Decoupled training and inference loop DuoGen uses a **two-stage decoupled strategy**. In **Stage 1**, only the MLLM is instruction-tuned, using the **298k** curated interleaved conversations. The objective is standard next-token prediction over assistant outputs: \[ \mathcal{L}_{\text{text}} = - \sum_{t \in \mathcal{A}} \log p_\theta(x_t \mid x_{` is included when it appears in the assistant turn [2602.00508]. In **Stage 2**, the MLLM is frozen, while the **connector** and **DiT** are trained on the broader context-alignment mixture. For each interleaved sequence, one target image is sampled, a random diffusion step is drawn from the scheduler, and the DiT is optimized with a diffusion-style objective, specifically described as permitting **flow matching**: \[ \mathcal{L}_{\text{img}} = \mathbb{E}_{(S,I^*),t}\left[\ell_{\text{diff}}\left(\text{DiT}(z_t^{I^*}, t; c_{\text{vis}}(S), c_{\text{text}}(S)),\, \text{target}\right)\right]. \] The paper does not specify a more detailed target parameterization beyond this description [2602.00508]. At inference time, the system repeatedly feeds the current interleaved context to the MLLM, generates text token by token, halts when `` appears, synthesizes an image with the DiT, appends that image to the context, and resumes text generation. **Classifier-free guidance** is used for image sampling; for the negative velocity, visual conditions are kept fixed while the **final text chunk** is removed from the MLLM hidden-state sequence. Additional implementation details reported in the paper include an **MLLM image cap of 480 px max side length** to avoid OOM and qualitative outputs shown at **768 × 768** resolution [2602.00508]. This decoupled training regime is a defining methodological choice. The paper’s rationale is that instruction-following behavior should be protected from noisier alignment data, while image-context consistency can be learned at scale without further perturbing the MLLM. ## 5. Empirical performance and benchmark profile DuoGen is evaluated on public interleaved generation benchmarks, a newly introduced benchmark, text-to-image evaluation, and image editing. On **CoMM**, it reports **Sty. 9.22**, **Enti. 9.22**, **Tren. 9.24**, **Comp. 9.66**, **ImgQ 9.53**, and **IRS 7.76**, exceeding the listed open-source baselines including MiniGPT-5, SEED-LLaMA, and Emu2. On **InterleavedBench**, it reports **T-Q 4.28**, **I-Q 3.65**, **I-Co 3.70**, **IT-Co 3.69**, **Helpfulness 4.06**, and **Avg. 3.87** [2602.00508]. The paper also introduces a new benchmark with **Cooking-200**, **Cooking-200-Text-Input**, and **How-to-500**. On **Cooking-200**, DuoGen reports **T-Com 3.61**, **I-Com 4.70**, **I-Co 3.92**, **I-Q 4.78**, and **IT-Co 4.75**. On **Cooking-200-Text-Input**, it reports **3.82**, **4.77**, **4.17**, **4.79**, and **4.76** for the same metrics. On **How-to-500**, it reports **T-Com 3.39**, **I-Com 4.22**, **I-Co 4.21**, **I-Q 4.08**, and **IT-Co 4.18**. In these comparisons, the paper states that DuoGen strongly outperforms open-source baselines and approaches Nano Banana on several subsets, while exceeding it on specific metrics such as image completeness in the cooking setting [2602.00508]. DuoGen also performs strongly outside explicitly interleaved benchmarks. On **GenEval**, it reports **Single Object 0.82**, **Two Object 0.99**, **Counting 0.94**, **Colors 0.91**, **Position 0.84**, **Attribute Binding 0.80**, and **Overall 0.88**. On **ImgEdit**, it reports **Add 4.53**, **Adjust 4.33**, **Extract 2.28**, **Replace 4.69**, **Remove 4.71**, **Background 4.61**, **Style 4.51**, **Hybrid 3.85**, **Action 4.67**, and **Overall 4.19**. On **GEdit_EN**, the scores reported are **G_SC 7.68**, **G_PQ 7.76**, and **G_O 7.35** [2602.00508]. The data ablation is especially informative. On **CoMM**, training with **CoMM original** data yields **Sty. 6.14**, **Enti. 6.21**, **Tren. 6.52**, **Comp. 6.45**, **ImgQ. 6.30**, **IRS 4.42**; processing the data with the authors’ engine improves these to **7.85**, **7.76**, **7.22**, **8.15**, **7.79**, **5.91**; and adding synthetic data further improves them to **9.15**, **9.21**, **9.30**, **9.45**, **9.48**, **7.58**. This is one of the clearest empirical demonstrations in the paper that data quality and data form, rather than architecture alone, are central bottlenecks for interleaved generation [2602.00508]. ## 6. Position within multimodal generation research DuoGen occupies a specific place within multimodal generation research. It is neither a pure tokenizer-level unification method nor a fully fused multimodal model pretrained end to end from scratch. Instead, it is a modular interleaved-generation system that uses an MLLM for control and a video-pretrained DiT for rendering, linked through hidden-state projection and cross-attention. In that respect, it is closely related to a broader shift toward reusing specialized pretrained modules rather than rebuilding joint models from the ground up [2602.00508]. A useful contrast is with **DualToken**, which addresses a different bottleneck: the incompatibility between the visual representations required for understanding and those required for generation within autoregressive multimodal models. DualToken resolves that conflict through dual visual vocabularies—semantic and perceptual—inside a unified tokenizer, whereas DuoGen resolves a different systems problem by coupling a pretrained MLLM with a pretrained DiT through a decoupled interface [2503.14324]. This suggests that “unified multimodal generation” now encompasses at least two distinct design lines: tokenizer-centric unification and module-centric interleaved orchestration. The paper also identifies several limitations. It does not provide extensive controlled ablations over MLLM choice, DiT backbone, connector depth, or cross-attention versus **MMDiT-style** conditioning. The MLLM remains frozen during the second stage, which preserves instruction-following behavior but may leave some language–image co-adaptation unrealized. Sequence-level modality decisions are learned through `` token prediction rather than via a dedicated planner. Exact robustness under highly complex or very long interleaved contexts is not fully explored. The authors explicitly list scaling up pretrained components, deeper architectural ablations, and comparison to **MMDiT-style** alternatives as future work [2602.00508]. Within the literature, DuoGen’s principal significance lies in showing that interleaved multimodal generation can be approached as a modular alignment problem rather than solely as a monolithic pretraining problem. Its central claim is not only that one model can both write and draw, but that strong interleaved performance can emerge from the combination of high-quality instruction data, a reusable pretrained MLLM, a reusable pretrained video DiT, and a carefully designed interface between them [2602.00508]. Source: https://www.emergentmind.com/topics/duogen