Unified Multimodal LLMs: Principles & Advances

• Unified Multimodal LLMs are neural architectures that integrate diverse signals (text, images, audio) using shared tokenization and Transformer backbones to enable cross-modal synergies.
• They employ self-supervised learning, mixture of experts, and diffusion-based methods to achieve flexible, scalable, and data-efficient generation and reasoning across modalities.
• Applications include vision-language retrieval, personalized image generation, SVG structuring, and audio-visual speech recognition, often achieving state-of-the-art performance.

Unified Multimodal LLMs (MLLMs) represent a class of neural architectures that achieve parameter, representational, and operational unification across text, images, audio, video, structural graphics, and additional data modalities. Unlike isolated modality models or those requiring task-specific adaptation, unified MLLMs employ single-backbone, cross-modal mechanisms enabling both understanding and generation in flexible, scalable, and data-efficient ways. Current research demonstrates that these models not only generalize across tasks, but yield cross-modal synergies that advance state-of-the-art in vision-language, SVG, audio-visual speech recognition, and domain-specialized tasks such as personalized image generation and medical report synthesis (Han et al., 29 May 2025, Liang et al., 15 Aug 2025, Mao et al., 7 Oct 2025, Cappellazzo et al., 10 Nov 2025).

1. Foundations of Unification: Core Principles and Techniques

Unified MLLMs are defined by a single architecture that (i) ingests varied modality signals (e.g., text, image, audio) as input, (ii) processes them using shared Transformer or diffusion-based backbones, and (iii) generates outputs in any supported modality under a consistent computational scheme (Han et al., 29 May 2025). This fundamentally contrasts with disjoint, task-specific models.

Key techniques include:

• Parameter Sharing and Cross-Modal Attention: All inputs—whether textual tokens, visual embeddings (e.g., ViT or VQ-VAE encodings), audio, or 3D/motion representations—are linearly projected into a shared embedding space and concatenated into a unified token sequence. Modality-agnostic self-attention layers permit bidirectional interactions across these tokens in all Transformer blocks.
• Self-Supervised Learning (SSL): Massive cross-modal corpora are leveraged for denoising, masked prediction, or contrastive objectives, providing a foundation for subsequent supervised or RL adaptation (Han et al., 29 May 2025).
• Mixture of Experts (MoE): Sparse-gated expert modules can be shared among or specialized for modalities, enhancing scaling efficiency without sacrificing cross-modal transfer.
• Chain-of-Thought (CoT) Prompting: Multi-step reasoning is formally represented by introducing latent chains $z$ or via explicit token-based reasoning steps (see MM-R1 X-CoT section below) (Liang et al., 15 Aug 2025).
• Diffusion and Discrete Tokenization: For image, video, audio, and 3D, discrete diffusion processes or BPE-style visual tokenization collapse varied modalities into unified sequences amenable to autoregressive or bidirectional modeling (Mao et al., 7 Oct 2025, Zhang et al., 30 Jun 2025, Zheng et al., 2024).
• Unified Codebooks & Embedding Strategies: Models such as UniCode use a shared discrete codebook for both visual and textual signals, allowing joint representation and generation through a single embedding matrix (Zheng et al., 2024).

2. Architectural Patterns: Tokenization, Fusion, and Modality Bridging

A canonical unified MLLM embodies several components:

• Unified Tokenization: Visual information is mapped (via a VQ-VAE, visual BPE, or structured SVG tokens) to discrete indices or embeddings, with or without spatial structure preservation (Zhang et al., 30 Jun 2025, Wang et al., 13 Oct 2025).
  • Audio and other temporal data are analogously tokenized using pre-trained audio encoders and projection heads (Cappellazzo et al., 10 Nov 2025).
• Early Fusion: All modality-specific tokens are concatenated to form the model input. In MM-R1, image patch embeddings ($\{v_1, \ldots, v_M\}$) are joined with text token embeddings ($\{t_1, \ldots, t_T\}$) as $X_0 = [v_1,\ldots,v_M, t_1, \ldots, t_T]$ (Liang et al., 15 Aug 2025).
• Shared Transformer Backbone: This sequence is propagated through layers of shared self-attention, with or without additional cross-modal or modality-specific adapters.
• Diffusion Integration: For text-image bimodalities, e.g., MeDiM, the transformer backbone forgoes causal masking to allow bidirectional diffusion-based generation, further conditioned on timestep embeddings for both text and image tokens (Mao et al., 7 Oct 2025).
• Unified Multi-Output Heads: The output space can be a single vocabulary (as in discrete unified codebooks), type-indicating tokens (as in UnifiedMLLM's “task tokens”), or modality-conditional heads (e.g., diffusion, autoregression, sequence-to-sequence) (Li et al., 2024, Zheng et al., 2024).

3. Cross-Modal Reasoning and Structured Generation

Unified MLLMs integrate advanced reasoning strategies explicitly suited to the challenges of alignment and controllability across modalities:

• Cross-Modal Chain-of-Thought (X-CoT) Reasoning: In MM-R1, personalization is reformulated as multi-step reasoning: (1) grounding a subject by textual description and visual crop, (2) generation conditioned on this “blueprint.” Special tokens indicate reasoning stages, allowing the transformer to interleave image and text generation, and focus attention via interaction peaks (Liang et al., 15 Aug 2025).
• Instructional or Prompt-Based Reasoning: In E5-V, explicit prompts ("Summary above image in one word:") collapse modality gaps for embedding alignment, enabling powerful zero-shot retrieval by leveraging LLM reasoning abilities (Jiang et al., 2024).
• Task and Grounding Tokens: UnifiedMLLM trains the LLM to emit explicit task and grounding tokens, which mark intended output types and spatial referents, providing a mechanism for downstream expert routing and compositional generation (Li et al., 2024).
• SVG Structural Modeling: In InternSVG, SVG code is parsed via special tag/attribute tokens plus numeric tokens, initialized through subword averaging for efficient language-model conditioning of structured vector graphics tasks (Wang et al., 13 Oct 2025).

4. Training Strategies and Optimization Paradigms

Scalable and robust training of unified MLLMs leverages curriculum learning, staged adaptation, and policy optimization techniques:

• Curriculum and Multi-Stage Training: Models such as Being-VL-0.5 and UnifiedMLLM employ stage-wise data scheduling—progressing from foundational perception data to complex multimodal reasoning and multi-turn instruction. Progressive unfreezing of network parameters is commonly observed to improve both stability and final performance (Zhang et al., 30 Jun 2025, Li et al., 2024).
• Parameter-Efficient Adaptation: Low-Rank Adapters (LoRA) and Mixture-of-Experts gating enable selective adaptation of large backbone LLMs, reducing full-model finetuning requirements and permitting task or modality specialization with marginal additional parameters (Cappellazzo et al., 10 Nov 2025).
• Grouped Reward Proximal Policy Optimization (GRPO): MM-R1 innovates on PPO by applying grouped-ranking rewards, eschewing separate value networks and optimizing for samplewise relative advantages in subject fidelity and prompt alignment (Liang et al., 15 Aug 2025).
• Bidirectional Conditioning and Diffusion Awareness: MeDiM enables joint textual and visual reasoning in medical generation by removing causal masks (yielding bidirectional context) and injecting continuous timestep embeddings at each transformer layer (Mao et al., 7 Oct 2025).
• Unified Codebook Synchronization: UniCode enforces periodic exponential-moving-average updates to reconcile the visual encoder codebook with the LLM embedding matrix, harmonizing both text and image token representations through shared discrete indices (Zheng et al., 2024).

5. Applications and Empirical Benchmarks

Unified MLLMs have empirically demonstrated robust performance on an array of cross-modal and specialized tasks, frequently outperforming modular or task-specific architectures:

General Multimodal Understanding and Retrieval:

• E5-V achieves state-of-the-art on text-image, composed, and image-image retrieval, despite being trained in single-modality text-only settings, with a 95% reduction in training cost. Zero-shot Recall@1/5/10 on COCO and Flickr30K is competitive with or superior to CLIP and EVA-02-CLIP 5B (Jiang et al., 2024).

Personalized Image Generation:

• MM-R1 surpasses both unified and modular baselines on DreamBench (DINO: 0.786, CLIP-T: 0.313) and Kontext-Bench for subject fidelity and text alignment, in a zero-shot regime where no per-subject finetuning occurs (Liang et al., 15 Aug 2025).

SVG Structured Graphics:

• InternSVG outperforms proprietary and leading open-source systems on the SArena benchmark across understanding, editing, and (animated) SVG generation (e.g., Icon editing PSNR: 77.3 vs 57.6; Illustration Text→SVG FID: 22.4 vs 27.3) (Wang et al., 13 Oct 2025).

Medical Multimodal Generation:

• MeDiM attains FID 16.60 on MIMIC-CXR (vs. UniDisc 82.54), BLEU-1 of 0.328 on medical report generation, and demonstrates robust joint image-text generation. Adding MeDiM synthetic pairs to downstream report generators in low-data regimes increases BLEU by up to 31.58% (Mao et al., 7 Oct 2025).

Audio-Visual Speech Recognition:

• Omni-AVSR unifies ASR, VSR, and AVSR in one model, achieving WER 1.0% (ASR) and 26.8% (VSR) on LRS3, with robust scaling across model size and compression, and flexible speed/accuracy trade-offs through elastic inference (Cappellazzo et al., 10 Nov 2025).

6. Challenges, Limitations, and Research Frontiers

Despite demonstrated gains, unified MLLMs pose open challenges:

• Modular Specialization: There is no consensus on principled taxonomy or gating for MoE expert specialization in unified MLLMs. Current architectures rely on heuristic allocation or co-training on diverse datasets (Han et al., 29 May 2025).
• Training Efficiency and Scalability: As capacity grows (e.g., thousands of MoE experts), efficient compression, pruning, and curriculum balancing become critical for tractable training (Han et al., 29 May 2025).
• Structured Reasoning for Non-Text Modalities: While CoT is mature in natural language, analogues in vision (e.g., visual planning graphs), motion (skeleton subgoals), and music (harmonic roadmaps) require further research and formalization (Han et al., 29 May 2025, Liang et al., 15 Aug 2025).
• Evaluation and Interpretability: Diverse and evolving benchmarks complicate comparison; interpretability remains limited outside text-centric reasoning, especially for compositional tasks or multi-step visual plans (Han et al., 29 May 2025).
• Domain Specialization: Although models such as MeDiM and MedXChat port general unified MLLM recipes to the medical domain, domain-specific pretraining or in-context adaptation may still be required for expert-level performance (Mao et al., 7 Oct 2025, Yang et al., 2023).

Future directions include SSL objectives encompassing causal dynamics and physical plausibility (esp. for motion and 3D), selective RLHF schemes, ControlNet-style cross-modal controls, and hybrid inference incorporating physics engines or formal structure awareness (Han et al., 29 May 2025).

7. Representative Models and Comparative Characteristics

Model	Tokenization/Fusion	Key Innovation	Empirical Highlight
MM-R1	Early Fusion, X-CoT	X-CoT reasoning, GRPO	SOTA zero-shot subject fidelity
E5-V	Prompted Embeddings	Single-modality training for multi	SOTA text-image retrieval
MeDiM	Discrete Diffusion	No causal mask, timestep embedding	FID 16.60 for CXR generation
InternSVG	SVG-special tokens	Unified SVG-task model, 2-stage	Outperforms Claude-4 on SArena
Omni-AVSR	Elastic Audio/Visual	Multi-granularity, LoRA adapters	Unified AVSR, competitive WER
UniCode	Unified codebook	Language-driven code synch, image decompression	Joint text/image generation
UnifiedMLLM	Unified vector z	Task/grounding tokens, router+experts, MoE LoRA	SOTA image/video/audio gen, flexible expert routing

• "A Survey of Generative Categories and Techniques in Multimodal LLMs" (Han et al., 29 May 2025)
• "MM-R1: Unleashing the Power of Unified Multimodal LLMs for Personalized Image Generation" (Liang et al., 15 Aug 2025)
• "Discrete Diffusion Models with MLLMs for Unified Medical Multimodal Generation" (Mao et al., 7 Oct 2025)
• "E5-V: Universal Embeddings with Multimodal LLMs" (Jiang et al., 2024)
• "InternSVG: Towards Unified SVG Tasks with Multimodal LLMs" (Wang et al., 13 Oct 2025)
• "Omni-AVSR: Towards Unified Multimodal Speech Recognition with LLMs" (Cappellazzo et al., 10 Nov 2025)
• "Unified Multimodal Understanding via Byte-Pair Visual Encoding" (Zhang et al., 30 Jun 2025)
• "UniCode: Learning a Unified Codebook for Multimodal LLMs" (Zheng et al., 2024)
• "UnifiedMLLM: Enabling Unified Representation for Multi-modal Multi-tasks With LLM" (Li et al., 2024)
• "MedXChat: A Unified Multimodal LLM Framework towards CXRs Understanding and Generation" (Yang et al., 2023)