LLaDA2.0-Uni: Unified Multimodal Diffusion LLM

• LLaDA2.0-Uni is a unified discrete diffusion large language model that processes both text and visual inputs through a semantic tokenization and block-wise masked diffusion methodology.
• It leverages a 16B-parameter Mixture-of-Experts backbone combined with a SigLIP-VQ tokenizer and a U-Net style diffusion decoder to support high-fidelity image synthesis and editing.
• The model achieves efficient inference with Sparse Prefix Retention and non-uniform token unmasking, outperforming specialized VLMs in interleaved multimodal reasoning and generative benchmarks.

LLaDA2.0-Uni is a unified discrete diffusion LLM (dLLM) framework that integrates multimodal understanding and generative capabilities within a single architecture. It combines a fully semantic discrete visual tokenizer, a large Mixture-of-Experts (MoE) dLLM backbone, and a U-Net–style diffusion decoder. By discretizing continuous visual inputs for semantic processing and using a single objective—block-level masked diffusion—the model achieves unified learning and inference for both text and vision modalities. LLaDA2.0-Uni attains performance parity with specialized vision–LLMs (VLMs) in understanding and surpasses them in several generative and editing benchmarks, while natively supporting interleaved multimodal reasoning and generation (AI et al., 22 Apr 2026).

1. Architecture and Tokenization

LLaDA2.0-Uni leverages a 16B-parameter MoE dLLM backbone, adhering to block-wise masked diffusion as its central training paradigm. The backbone employs block-wise attention—full bidirectional within blocks of length $L_B$ (usually 128–256 tokens) and restricted across blocks—enabling parallel sampling ($O(B)$ steps) and bidirectional context modeling. Visual inputs are processed using a SigLIP-VQ tokenizer: images of arbitrary resolution are encoded via the pre-trained SigLIP2-g ViT, vector-quantized through a 16,384-size codebook of 2,048-dimensional embeddings, and translated into 1-D sequences of fully semantic “image tokens.” Special <imgsize_H> and <imgsize_W> tokens prefix these sequences to encode dimensions.

The expanded vocabulary incorporates both the original text tokens and the 16,384 visual codes, alongside special tokens supporting image generation and editing. Text embeddings are initialized from pretrained LLaDA2.0 weights and frozen; image-token embeddings are randomly initialized. The model’s output head covers the full enlarged vocabulary with a parallel initialization schema.

The decoder, based on a 6B “Z-Image-Base” text-to-image diffusion model, implements a U-Net with cross-attention to semantic tokens and learned upsampling modules to produce high-fidelity 1024² outputs. It operates without an explicit textual prompt for reconstruction at this stage.

2. Block-wise Discrete Diffusion and Training Objectives

The training objective is a block-wise discrete diffusion process. Input sequences $x_0$ are segmented into $K$ blocks of size $L_B$; at noise time $t$, each block $k$ is corrupted by masking tokens independently with probability $1-\alpha_t$, yielding $x_{t,k}$. The model learns to infer the clean sequence via the following block Diffusion LLM (BDLM) loss (from equation (8) in the source):

$$L_\text{BDLM}(\theta) = -\mathbb{E}_{t, x_0, x_t} \left[ \frac{\alpha_t'}{1 - \alpha_t} \sum_{k=1}^K \sum_{i=1}^{L_B} 1[x_{t,k}^i = \text{MASK}] \log p_\theta( x_0^{i,k} | x_{0,<k}, x_{t,k} ) \right]$$

The reverse (denoising) step occurs in parallel for each block:

$O(B)$0

where $O(B)$1 and $O(B)$2 are projection parameters and $O(B)$3 denotes temperature.

The diffusion decoder’s training employs a flow-matching loss and a distillation loss (Eq. (16)), including a warm-up phase (freezing the semantic processor), an unfreezing generalization stage, and a high-fidelity refinement pass.

3. Training Data and Multistage Pipeline

LLaDA2.0-Uni is trained in a multi-stage regime:

• Stage 0 (Vision–Language Alignment): 100B tokens, mixing image–caption pairs, OCR, grounding/counting, world-knowledge reasoning, and text from Ling2.0/LLaDA2.0. Masking applies exclusively to image or text tokens, with progressive image resolution (256→512) for generation and consistent 800px edge for understanding.
• Stage 1 (Multi-task Pre-training): 210B tokens spanning image captioning, text QA, OCR, grounding/counting, video, multimodal VQA, editing, and interleaved generation. Supports up to 8k token sequences.
• Stage 2 (Supervised Fine-Tuning): ~60M samples (1:5 text:multimodal), including dialogues, chart/table QA, math, chain-of-thought (CoT), editing, and complex multimodal tasks. Mask-token reweighting ($O(B)$4) prevents long-sequence dominance. Complementary masking per sequence doubles effective token coverage.

Image generation corpus: 200M web image–caption pairs (filtered to 140M for quality/aesthetics); captions refined using Qwen3-VL.

Image editing corpus: Open datasets (X2Edit, OmniEdit) and synthetic pairs, with instructions filtered/re-written via Qwen3-VL.

Interleaved video data: 6M high-quality Koala36M clips into 2–6 frame sequences with captions and user instructions.

Reasoning-augmented data: 8M SFT samples from sources such as Flux-6M, Zebra-CoT, and Weave for advanced CoT and interleaved reasoning.

4. Inference Procedures and Efficiency Enhancements

LLaDA2.0-Uni incorporates SPRINT (Sparse Prefix Retention and Inference-time Non-uniform Token Unmasking):

• Sparse Prefix Retention (SPR): For each block’s first denoising pass, SPR calculates a retention score $O(B)$5, with $O(B)$6 the ratio between normed key vectors and $O(B)$7 the maximum softmax confidence. Modality-specific cache pruning retains all text tokens and 80% (or all) image tokens for subsequent steps.
• Non-uniform Token Unmasking: In each diffusion step $O(B)$8, positions $O(B)$9 with $x_0$0 (typically $x_0$1) are unmasked, maintaining a guaranteed minimum rate of unmask for convergence.

Result: This strategy yields up to 1.6× speedup with <0.6 point average multimodal benchmark score reduction.

For the diffusion decoder, after standard 50-step classifier-free guidance (CFG) sampling and distillation, an 8-step CFG-free “Turbo” mode is enabled, accelerating 1024² image synthesis from 32.95 s/img to 2.90 s/img (11.4× faster) with <1% quality reduction across GenEval, DPG, UniGenBench, OneIG, and WISE benchmarks.

5. Benchmark Results and Comparative Performance

Selected Quantitative Results

Benchmark	LLaDA2.0-Uni Score	Comparison	Closest Baseline	Baseline Score
MMStar (General VQA)	64.1	Unified VLM	Qwen2.5-VL-7B	63.9
MMMU (Reasoning)	50.1	Unified VLM	Lumina-DiMOO	58.6 (subtask)
DocVQA (OCR/Doc)	89.5	Unified VLM	Qwen2.5-VL	94.9
GenEval (Image Generation)	0.89 (overall)	Unified VLM	Lumina-DiMOO	0.88
DPG-Bench (Image Generation)	87.76	Unified VLM	LLaDA-o	87.04
OneIG-EN	0.505	Unified VLM	Qwen-Image	0.539
UniGenBench	79.63	Unified VLM	Seedream 3.0	78.41
ImgEdit-Bench (Editing)	3.92	Unified VLM	InternVL-U	3.67
MICo-Bench (Edit, multi-ref)	47.1	Unified VLM	Qwen-Image-Edit	35.9

Across more than 21 understanding benchmarks, LLaDA2.0-Uni matches or exceeds the state of the art among diffusion LLMs (Lumina-DiMOO, LLaDA-o, MMaDA) in all categories. Its editing and compositional generation scores consistently surpass those of other unified and specialized counterparts, closing gaps with autoregressive (AR) VLMs.

In interleaved generation and reasoning (e.g., InterGen 150-sample test), LLaDA2.0-Uni outperforms Emu3.5 in narrative story telling and forecasting, producing coherent multi-turn image–text sequences capable of stepwise visual reasoning.

6. Application Scope and Known Limitations

LLaDA2.0-Uni is deployed for a broad range of multimodal applications: visual question answering (VQA), document understanding, general text-to-image generation, text evacuation, image editing, interleaved multimodal dialogue, and reasoning-informed generation. Its flexible prompting supports any sequence of text and visual frames up to 16k tokens, including video snippets or complex CoT prompting prior to generation.

Known limitations include:

• Occasional loss of fine pixel-level details in editing, due to the semantic abstraction of the SigLIP-VQ tokenizer. Improved or hybrid semantic-pixel tokenizers could potentially address this.
• Limited diversity in interleaved samples and model capacity may impede scaling for multi-turn interleaved reasoning and generation.
• Early-stage reinforcement-learning fine-tuning demonstrates initial benefits in instruction alignment, but requires more stable protocols for further improvements.

A plausible implication is that unification under block-wise discrete diffusion, coupled with efficiency gains from inference optimizations, represents a viable direction for scalable, fully unified multimodal foundation models (AI et al., 22 Apr 2026).