Concurrent Mixed-Modal Generation

• Concurrent mixed-modal generation is a framework that synthesizes arbitrarily interleaved multi-modal outputs (text, images, audio, video) in a single generative process.
• It leverages diverse architectures such as token-based early fusion, dual-expert decoupling, and non-autoregressive models to enable dynamic modality switching and joint context reasoning.
• Advanced training protocols and evaluation metrics balance modality dominance and enhance coherence, driving applications in open-domain assistants, visual storytelling, and document automation.

Concurrent mixed-modal generation refers to the unified synthesis of multiple data modalities—such as text, images, audio, and video—where output streams are interleaved or produced simultaneously within a single generative process. Unlike unimodal or strictly sequential multi-modal generation paradigms, concurrent mixed-modal generation enables arbitrary interleaving, dynamic modality switching, and joint context reasoning across different forms of content. This ability underpins sophisticated applications in open-domain assistants, visual storytelling, document automation, and reasoning tasks that require seamless transition between symbolic (discrete) and perceptual (continuous) content. The formalization and realization of this capability constitute a major trajectory in modern foundation model research.

1. Foundations and Definitions

Concurrent mixed-modal generation departs from two historical paradigms: (1) unimodal generation (one modality per model/pass), and (2) conditional multi-modal generation (conditioning on one modality to generate another in full, e.g., captioning or text-to-image). Instead, it is defined as the generation of arbitrarily interleaved outputs, $Y = \{(m_1, y_1), …, (m_K, y_K)\}$, where $m_k$ is the modality label (e.g., “text”, “image”, “audio”, “video”) and $y_k$ is the generated segment. The generative model defines a joint distribution over sequences,

$$P_\theta(Y|U) = \prod_{k=1}^K P_\theta(y_k | U, Y_{<k})$$

where $U$ is the user prompt, supporting both variable-length and content-adaptive interleaving (Xing et al., 26 Mar 2026, Team, 2024).

This formalism encompasses models across token-based early fusion (text+image, text+speech), dual-expert decoupling, and multi-expert “mixture-of-transformers” backbones. The output can reflect non-fixed alternation, data-dependent modality orderings, and concurrent updates to multiple modalities (Nguyen et al., 3 Oct 2025, Liao et al., 8 May 2025).

2. Architectural Approaches

A. Token-Based Early Fusion:

Chameleon and Ichigo implement a single decoder backbone that accepts and outputs an interleaved stream of discrete tokens, with text and image (or speech) tokens mapped into a unified vocabulary. Both modalities are projected into shared embedding spaces (e.g., using SentencePiece BPE for text, VQ-VAE or Whisper VQ for images/speech), allowing the transformer to process them indistinguishably at the sequence level. During generation, the model emits either a text token or a block of image or audio tokens based on next-step probabilities (Team, 2024, Dao et al., 2024).

B. Dual-Expert and Mixture-of-Transformers (MoT):

Frameworks such as Wan-Weaver and TV2TV employ a planner/visualizer (or text/video) decoupling within a MoT backbone (Xing et al., 26 Mar 2026, Han et al., 4 Dec 2025). The planner autoregressively emits text and dense prompts, while the visualizer synthesizes pixels or video frames conditioned on planner outputs. Interaction between experts uses explicit “gating” tokens (e.g., <imagine>, BOF) and prompt-context windows. This design enables fine-grained control over interleaving and explicit grounding of visual content.

C. Unified Non-Autoregressive Models:

OneFlow and CoM-DAD advance beyond autoregressive frameworks through concurrent, variable-length non-monotonic generation. OneFlow introduces a discrete insertion-based Edit Flow process for text, combined with continuous Flow Matching for images. The model performs hierarchically scheduled, concurrent insertions and ODE denoising, supporting true simultaneous synthesis (Nguyen et al., 3 Oct 2025, Xu et al., 7 Jan 2026). CoM-DAD further decouples high-level semantic planning (via continuous latent diffusion) from low-level discrete synthesis using absorbing diffusion, with stochastic transport for inter-modal alignment.

D. Multimodal GANs and Retrieval-Augmented Generation:

Joint adversarial architectures, such as those in early work on audio–video generation, couple unimodal generators and discriminators with joint discriminators that enforce inter-modal correlation and synchronization (Kurmi et al., 2021). Retrieval-augmented setups (M²RAG) treat both input and output as interleaved modality sequences and leverage specialized multi-stage LLM or MLLM prompting for grounding outputs in retrieved multimodal evidence (Ma et al., 2024).

3. Training Objectives, Curriculum, and Alignment Strategies

Mixed-modal generative models typically employ staged or multi-objective training protocols:

• Early-fusion AR models (e.g., Chameleon, Ichigo) train under a unified next-token cross-entropy, mixing text-only, modality-pair, and fully interleaved documents. Modality order balancing, upsampling underrepresented categories, and prompt masking regularize against dominance and drift between modalities (Team, 2024, Dao et al., 2024).
• MoT and decoupled strategies (Wan-Weaver, DuoGen, TV2TV) apply separate training of planners (using synthetic or re-written interleaved data) and visualizers (using abundant reference-guided or video-based corpora). Fine-tuning stages specialize on cross-modal context alignment and dense-prompt context window integration (Xing et al., 26 Mar 2026, Shi et al., 31 Jan 2026).
• Policy optimization for modality interleaving: Reinforcement learning-based objectives with hybrid rewards explicitly optimize text–image alignment, structural fidelity, and process-level feedback (e.g., reward for correct alternation of >/<vis> blocks and alignment between generated text/images) (Nie et al., 10 Mar 2026).

  • Diffusion and absorbing processes: In advanced non-AR systems, losses combine mean-squared flows over continuous semantic representations with token-level cross-entropies and cross-modal reconstruction objectives (Xu et al., 7 Jan 2026, Nguyen et al., 3 Oct 2025).

  Curricula typically mix unsupervised next-token prediction, supervised instruction tuning, upsampling of rare modalities, and sometimes process-level hybrid rewards.

  4. Inference and Interleaving Mechanisms

  Generative inference across modalities is handled via:

  • Autoregressive token-stepping: At each generation step, the model samples from the shared vocabulary; upon predicting a special modal token (such as IMG_START_TOKEN or <BOV>), it emits the required number of image or audio tokens, then resumes token sampling (Team, 2024, Dao et al., 2024, Shi et al., 31 Jan 2026).

  • Expert handoffs via gating tokens: MoT or decoupled backbones switch from planner to visualizer upon encountering a gate (such as <imagine> or BOF). The visualizer synthesizes the visual/auditory segment and hands back control (Xing et al., 26 Mar 2026, Han et al., 4 Dec 2025).
  • Hierarchical or concurrent scheduling: OneFlow and CoM-DAD enable variable-length, order-agnostic, or simultaneous updates via insertion chains (text) and ODE/image denoising steps, leveraging a global interleaved schedule to concurrently refine all modalities (Nguyen et al., 3 Oct 2025, Xu et al., 7 Jan 2026).
  • Retrieval and multi-stage prompting: For retrieval-augmented M²RAG, a generator emits structured Markdown, inserting image placeholders based on relevance-scored retrieved elements within a single or multi-stage sequence (Ma et al., 2024).

  These approaches permit both strictly alternating and arbitrary interleaved modality patterns, with some frameworks supporting user interventions or on-the-fly trajectory modifications (Han et al., 4 Dec 2025).

  5. Evaluation Protocols and Empirical Outcomes

  Evaluation of concurrent mixed-modal generation leverages a range of automatic and human metrics, including:

  • Task-specific metrics: CIDEr, FID, ROUGE, BLEU for text/image, Inception Score and human preference wins for video/audio (Nguyen et al., 3 Oct 2025, Liao et al., 8 May 2025, Kurmi et al., 2021).
  • Specialized interleaved benchmarks: OpenING, WeaverBench, InterleavedBench, CoMM dataset, among others, include dimensions for structural correctness, content and image quality, alignment, completeness, narrative coordination, and image count accuracy. Scoring is often performed by GPT-4o or GPT-5 (Xing et al., 26 Mar 2026, Shi et al., 31 Jan 2026).
  • Ablation studies: Demonstrate that curriculum structure (e.g., decoupled training), cross-modal attention, hybrid rewards, and data-mix strategies significantly boost interleaved sequence quality and consistency (Xing et al., 26 Mar 2026, Nie et al., 10 Mar 2026, Zhang et al., 23 Jun 2025).

  Empirical findings show that these models can match or exceed the performance of much larger or specialized models (e.g., Chameleon outperforming Llama-2 on text and achieving state-of-the-art human preference on mixed-modal long-form tasks (Team, 2024); Wan-Weaver outscoring all open models on WeaverBench and rivaling commercial counterparts (Xing et al., 26 Mar 2026); OneFlow outperforming AR and diffusion baselines for both understanding and generation (Nguyen et al., 3 Oct 2025); and CoM-DAD providing stability and parallelism unattainable with masked LLMs (Xu et al., 7 Jan 2026)).

  6. Challenges, Limitations, and Future Directions

  Key challenges and open problems in concurrent mixed-modal generation include:

  • Data scarcity and diversity: Real interleaved datasets are uncommon; frameworks synthesize large-scale proxies with LLM/VLMs, but rare or complex domains (e.g., medical) remain underrepresented (Xing et al., 26 Mar 2026, Shi et al., 31 Jan 2026).
  • Long-range and structural coherence: Maintaining global consistency and correct image–text interplay, particularly for complex layouts, is unresolved; advanced planners and symbolic modules are proposed as solutions (Xing et al., 26 Mar 2026).
  • Scalability and efficiency: Models such as OneFlow achieve significant FLOP and memory improvements over strict AR models but pose implementation complexity for hierarchical concurrent sampling (Nguyen et al., 3 Oct 2025). Sequential inference over long sequences remains a bottleneck (Xing et al., 26 Mar 2026).
  • Modality balance and dominance: Training must mitigate mode collapse (e.g., always generating only text or only images) via explicit batch balancing, curriculum, and alignment losses (Team, 2024, Nie et al., 10 Mar 2026).
  • Generalization to new modalities: Extension to video, 3D, and audio requires refined architectural and tokenization schemes; multimodal RoPE variants, dual encoders, and early fusion prove effective for images, but scaling to more modalities remains an open avenue (Liao et al., 8 May 2025, Han et al., 4 Dec 2025).

  Future research is directed towards memory- and resolution-adaptive models, user-in-the-loop editing, increasing reasoning proficiency, and unified training modules capable of bidirectional multi-modal understanding and generation at scale.

  7. Representative Model and Method Comparison

  Model/Framework	Key Architecture	Supported Modalities	Core Generation Paradigm
  Chameleon (Team, 2024)	AR Transformer, Early Fusion	Text, Images	Unified autoregressive, token-level interleaving
  Ichigo (Dao et al., 2024)	AR Transformer, Early Fusion	Text, Speech	Unified token stream, no gating
  Wan-Weaver (Xing et al., 26 Mar 2026)	MoT (planner/visualizer)	Text, Images	Decoupled AR planner and DiT visualizer
  OneFlow (Nguyen et al., 3 Oct 2025)	Bidirectional Transformer	Text, Images	Non-AR, concurrent insertion and flow matching
  TV2TV (Han et al., 4 Dec 2025)	MoT, Interleaved towers	Text, Video	LM + flow matching, gating between text/video
  DuoGen (Shi et al., 31 Jan 2026)	MLLM + DiT, Decoupled	Text, Images	AR text, DiT vision, prompt hand-off
  M²RAG (Ma et al., 2024)	Prompted LLM/MLLM	Text, Images (Retrieval)	Multi-stage prompting, retrieval grounding
  BiGen (Zhang et al., 23 Jun 2025)	Encoder-Decoder, Cross-Mod. Alignment	Visual, Textual reports	Cross-attn. fusion, concurrent encoding

  This table highlights architectural diversity and modality coverage among leading models, as well as the dominant strategies for joint or concurrent generation.

  ---

  Concurrent mixed-modal generation systematically integrates arbitrarily interleaved multi-modal outputs within a single foundational model, blending autoregressive, diffusion-based, policy-optimized, and retrieval-augmented paradigms. Recent technical advances—early-fusion architectures, mixture-of-experts frameworks, coordinated training with process-level rewards, and scalable interleaved data curation—yield state-of-the-art results across understanding and generation tasks. Remaining challenges point to hybrid modeling of evermore complex modality sequences, scalable and efficient inference, and robust generalization to real-world long-form and high-dimensional multi-modal outputs.