LLaDA2.0: Scalable Discrete Diffusion LLMs

• LLaDA2.0 is a family of discrete diffusion language models that replace autoregressive token generation with masked, parallel denoising.
• It employs a three-phase warmup–stable–decay training protocol to systematically convert pretrained AR models into efficient dLLMs.
• The framework integrates mixture-of-experts and unified multimodal strategies to achieve competitive performance and accelerated inference.

LLaDA2.0 refers to a family of discrete diffusion LLMs (dLLMs) that demonstrate the scalability and unification of masked diffusion architectures for both language and multimodal tasks. Notable variations include LLaDA2.0-mini (16B parameters), LLaDA2.0-flash (100B parameters) (Bie et al., 10 Dec 2025), and LLaDA2.0-Uni, a unified vision-LLM for multimodal understanding and generation (AI et al., 22 Apr 2026). These models establish that dLLMs, constructed via systematic conversion from strong auto-regressive (AR) checkpoints, can match or exceed AR baselines while enabling fully parallel generation and high multimodal fidelity.

1. Discrete Diffusion LLM Formalism

LLaDA2.0 replaces the AR token-by-token left-to-right generation with a masked diffusion process. At each diffusion timestep $t$, tokens are randomly masked according to a noise schedule $\alpha_t$; the model learns to denoise masked positions in parallel, conditioned on the remaining context. The discrete Masked Diffusion LLM (MDLM) training objective is:

$$\mathcal{L}_{\rm MDLM}(\theta) =-\mathbb{E}_{t,\mathbf x_0,\mathbf x_t} \left[\frac{\alpha'_t}{1-\alpha_t}\sum_{i=1}^L \mathbf 1[x^i_t=\text{[MASK]}]\, \log p_\theta\bigl(x^i_0\mid\mathbf x_t\bigr)\right]$$

Block Diffusion LLMs (BDLMs) partition the sequence into $K=L/L_B$ contiguous blocks of size $L_B$, generalizing the objective:

$$\mathcal{L}_{\rm BDLM}(\theta) =-\mathbb{E}_{t,\mathbf x_0,\mathbf x_t} \left[\frac{\alpha'_t}{1-\alpha_t}\sum_{k=1}^{K}\sum_{i=1}^{L_B} \mathbf 1[x^i_{t,k}=\text{[MASK]}]\, \log p_\theta\bigl(x^i_{0,k}\mid\mathbf x_{0,<k},\mathbf x_{t,k}\bigr)\right]$$

This blockwise formulation allows for parallel, bidirectional denoising while preserving conditional dependencies necessary for structured output.

2. Systematic Conversion and Three-Phase Warmup–Stable–Decay Training

LLaDA2.0 demonstrates a practical, stable protocol for converting pretrained AR models into dLLMs at large scales using a three-phase block-level schedule:

1. Warmup: Progressively increase BDLM block size $L_B$, e.g., from $1 \rightarrow 4 \rightarrow 32 \rightarrow 4096$, which incrementally introduces bidirectionality while stabilizing attention patterns.
2. Stable: Fix $L_B$ to full-sequence, training as a standard MDLM at corpus scale for denoising.
3. Decay: Reduce $L_B$ stepwise (e.g., $\alpha_t$0), distilling global semantics into compact blocks that restore AR-style KV-cache reuse for efficient variable-length generation.

Post-phase checkpoint merging (WSM) averages top-performing checkpoints for robust initialization. All network weights—including embeddings and attention—are inherited from the AR pretraining stage, facilitating knowledge retention and minimizing catastrophic forgetting during attention paradigm shifts (Bie et al., 10 Dec 2025).

3. Mixture-of-Experts Architecture and Multimodal Unification

LLaDA2.0 variants employ sparse Mixture-of-Experts (MoE) layers to efficiently scale capacity:

• Mini: 16 experts (mini, 16B), flash: 64 experts (flash, 100B)
• Top-2 expert gating per token, load-balance auxiliary loss, and MoE layers interleaved with standard FFN layers
• Data Parallelism and Expert Parallelism during fine-tuning to maximize efficiency

LLaDA2.0-Uni extends the BDLM paradigm to multimodal settings via a unified architecture:

• SigLIP-VQ Discretization: A pretrained SigLIP2-g ViT backbone encodes images into a 1D sequence of $\alpha_t$1 ($\alpha_t$2), quantized via a learnable codebook $\alpha_t$3 ($\alpha_t$4) to yield integer tokens $\alpha_t$5
• Shared Vocabulary: Merges text tokens, vision codebook tokens, and special tags to support arbitrary-resolution visual and textual inputs
• Diffusion Decoder: Based on Z-Image-Base, conditions exclusively on semantic tokens (not textual prompts); upsampling and high-fidelity image reconstruction employ a distilled flow-matching approach, running in eight denoising steps without classifier-free guidance (AI et al., 22 Apr 2026)

4. Training Pipeline and Post-training Alignment

All LLaDA2.0 models deploy a multi-stage training protocol:

Stage	Data Volume & Type	Objective/Enhancements
S0	100B tokens: image-caption pairs, text	BDLM (single-modality masking only)
S1	210B tokens: image/text, OCR, counting, VQA, video	Full-modal BDLM (standard, multi-task)
S2	80B tokens: 60M samples (text:multimodal = 1:5)	Conditional block diffusion, mask-token re-weighting, complementary masking

During Supervised Fine-Tuning (SFT), the loss incorporates conditional block diffusion, mask ratio bandwidth clipping, and mask-token re-weighting:

$\alpha_t$6

Alignment via Direct Preference Optimization (DPO) uses BDLM ELBO surrogates:

$\alpha_t$7

where $\alpha_t$8 denotes the difference in surrogate evidence lower bounds between preferred and less-preferred outputs.

5. Inference Efficiency Strategies

LLaDA2.0 innovations in inference include:

• SPRINT (Sparse Prefix Retention + Non-uniform Token Unmasking):

Retains only high-scoring KV cache entries per block for subsequent denoising steps, using:

$\alpha_t$9

• NTU (Non-uniform Token Unmasking):

Unmasks all positions exceeding a confidence threshold $$\mathcal{L}_{\rm MDLM}(\theta) =-\mathbb{E}_{t,\mathbf x_0,\mathbf x_t} \left[\frac{\alpha'_t}{1-\alpha_t}\sum_{i=1}^L \mathbf 1[x^i_t=\text{[MASK]}]\, \log p_\theta\bigl(x^i_0\mid\mathbf x_t\bigr)\right]$$0 rather than a fixed number per step.

These enable up to 1.6× throughput gains (tokens per second) with marginal loss ($$\mathcal{L}_{\rm MDLM}(\theta) =-\mathbb{E}_{t,\mathbf x_0,\mathbf x_t} \left[\frac{\alpha'_t}{1-\alpha_t}\sum_{i=1}^L \mathbf 1[x^i_t=\text{[MASK]}]\, \log p_\theta\bigl(x^i_0\mid\mathbf x_t\bigr)\right]$$1 points) on nine benchmarks.

• Diffusion Decoder Turbo:

Compression of the diffusion decoding process to eight steps (from an original 50) via consistency-based distillation, with negligible impact ($$\mathcal{L}_{\rm MDLM}(\theta) =-\mathbb{E}_{t,\mathbf x_0,\mathbf x_t} \left[\frac{\alpha'_t}{1-\alpha_t}\sum_{i=1}^L \mathbf 1[x^i_t=\text{[MASK]}]\, \log p_\theta\bigl(x^i_0\mid\mathbf x_t\bigr)\right]$$2 metric degradation) and speedups exceeding 11× (AI et al., 22 Apr 2026).

6. Empirical Results: Multimodal and Language Performance

Comprehensive benchmarking situates LLaDA2.0 models at or above the state-of-the-art in both unimodal and multimodal regimes:

Task / Metric	LLaDA2.0-Uni (16B)	Closest Baseline(s)	Summary Advantage
General VQA (MMStar)	64.1%	63.9% (Qwen2.5-VL)	Parity with strong specialist VLMs
Math/Reasoning	68.1% (MathVista)	68.2% (Qwen2.5)	Parity; significantly outperforms prior diffusion VLMs
OCR/ChartQA	89.5%/80.1%	94.9%/84.1% (Qwen2.5)	Subpar to best AR baseline, but major gain vs. diffusion
Text-to-Image GenEval	0.89	0.87 (Qwen-Image, gen-only)	Matches or exceeds best unified/AR baselines
Image Editing MICo	47.1%	35.9% (Qwen-Image, gen-only)	Clear lead over unified and gen-only competitors
Inference TPS	1.6× AR	1.0× (AR), 1.3× (block AR)	Significant parallel decoding advantages

LLaDA2.0-flash (100B) attains 73.18% overall (47-benchmark average), on par with AR models Qwen3-30B and Ling-flash-2.0. Long-context (>32k) performance remains robust ($$\mathcal{L}_{\rm MDLM}(\theta) =-\mathbb{E}_{t,\mathbf x_0,\mathbf x_t} \left[\frac{\alpha'_t}{1-\alpha_t}\sum_{i=1}^L \mathbf 1[x^i_t=\text{[MASK]}]\, \log p_\theta\bigl(x^i_0\mid\mathbf x_t\bigr)\right]$$3 on RULER).

7. Unified Generation and Interleaved Reasoning

LLaDA2.0-Uni’s fully discrete sequence enables natively unified operations: text and image blocks are interchangeable in the BDLM diffusion stream. This permits mixed-modality interleaving—such as stepwise chess move visualization, event forecasting, or complex diagram-based multi-hop reasoning—without architectural modification. On the InterGen benchmark, LLaDA2.0-Uni outperforms Emu3.5 under both Gemini and Qwen3 evaluators (e.g., Storytelling: 7.02 vs. 6.83).

A plausible implication is that, by representing both visual and textual information as discrete blocks within a single mask-diffusion process, future large-scale foundation models can eliminate the technical separation between multimodal input, generation, and reasoning tasks (AI et al., 22 Apr 2026).