LLaDA-MoE: Sparse MoE & Masked Diffusion LLM

• LLaDA-MoE is a large language model that integrates masked diffusion and a sparse Mixture-of-Experts Transformer, activating only 1.4B parameters per token to maximize efficiency.
• The model achieves state-of-the-art generative and reasoning performance, demonstrating superior benchmark results on tasks like MMLU, BBH, and HumanEval.
• It employs innovative training techniques and blockwise decoding to reduce feed-forward computation by over 80% compared to dense models.

LLaDA-MoE is a LLM that unifies the masked diffusion modeling (MDM) paradigm with a sparse Mixture-of-Experts (MoE) Transformer architecture. Designed to maximize computational efficiency while maintaining state-of-the-art generative and reasoning performance, LLaDA-MoE leverages 7 billion parameters in total, but activates only 1.4 billion parameters per token during inference. The model was trained on approximately 20 trillion tokens and demonstrates superior or competitive benchmark results versus both dense diffusion LLMs and smaller parameter-count dense models, establishing new baselines across multiple tasks among diffusion LLMs (Zhu et al., 29 Sep 2025).

1. Model Architecture

LLaDA-MoE integrates two principal components: a Masked Diffusion LLM (MDM) and a sparse MoE Transformer. The architecture comprises 16 Transformer layers with hidden size 2048 and 16 attention heads. Each feed-forward sublayer is replaced by a MoE block with 64 experts (dimension 1024), of which only the top 8 are activated per token. This design yields a model with 7B non-embedding parameters but ensures only 1.4B parameters are active at any token position during inference.

Masked Diffusion Process

Let $y\in\{0,\dots,K-1\}^L$ denote a token sequence of length $L$. The forward diffusion step corrupts tokens by sampling a noise level $t\sim \mathcal U[0,1]$ and replacing each token $y^i$ with a mask symbol $\mathbf M$ with probability $t$:

$$q(y_t^i \mid t, y^i) = \begin{cases} 1-t, & y_t^i = y^i \ t, & y_t^i = \mathbf M \ 0, & \text{otherwise} \end{cases}$$

The reverse step recovers cleaner tokens $y_s$ from $y_t$ for $0\leq s < t \leq 1$, using both model predictions $L$0 and explicit schedule coefficients.

Sparse MoE Details

For each token position $L$1, the router projects the hidden state $L$2 to logits $L$3. After softmax, the top-8 experts are chosen:

$L$4

The per-token output is:

$L$5

where $L$6 is expert $L$7's feed-forward network. Auxiliary balancing losses ($L$8 and $L$9) are added to encourage expert utilization diversity and router entropy.

2. Training Procedure and Objective

Pretraining proceeds over approximately 20T tokens split into two 10T phases, using a diverse text corpus. The curriculum involves fixed and then extended context lengths (4k to 8k) and upsampling of math/code in the latter phase. Subsequent annealing stages use 1T high-quality tokens, with later stages extending the RoPE base from 10k to 50k.

Instruction fine-tuning (SFT) uses masked diffusion loss applied only to response segments of curated prompt–response pairs. The final pretraining objective combines the denoising loss with regularizers:

$t\sim \mathcal U[0,1]$0

During SFT, only the response $t\sim \mathcal U[0,1]$1 is corrupted and denoised, with the prompt $t\sim \mathcal U[0,1]$2 unaltered:

$t\sim \mathcal U[0,1]$3

3. Inference and Computational Efficiency

Generation is initialized with a fully masked sequence and proceeds by iterative denoising, optionally organized into blocks supporting semi-autoregressive parallel decoding. Decoding proceeds via a scheduled noise level reduction, gradually unmasks positions, and remasks low-confidence predictions per block.

Computational savings derive from MoE sparsity: only 8 of 64 experts are active per token, making FFN computations $t\sim \mathcal U[0,1]$4—12.5% of the dense equivalent. For LLaDA-MoE, this reduces feed-forward compute over 80% compared to standard dense 7–8B models, while maintaining full representational capacity.

4. Empirical Performance

Benchmark evaluations span knowledge (MMLU, CMMLU, CEval, RACE), reasoning (BBH, DROP, KorBench), mathematics (GSM8K, MATH, OlympiadBench), coding (HumanEval, MBPP, MultiPL-E, CRUX-O, LiveCodeBench, BigCodeBench), agent, and alignment tasks.

Summary of Results

Model	Total Params	Active Params	Avg. Score (Base)	Avg. Score (Instruct)
LLaDA-MoE	7B	1B	46.94%	53.12%
LLaDA-8B (dense)	8B	8B	43.53%	42.65%
Dream-7B (dense)	7B	7B	46.66%	46.51%
Qwen2.5-3B-Instruct (dense)	3B	3B	—	53.51%

LLaDA-MoE surpasses previous diffusion MDMs (including LLaDA, LLaDA 1.5, and Dream) on average, despite activating only about 1B parameters. The instruct-tuned variant (LLaDA-MoE-7B-A1B-Instruct) demonstrates performance comparable to Qwen2.5-3B-Instruct (53.12% vs. 53.51%), and attains state-of-the-art on many individual tasks among diffusion LLMs. This demonstrates the efficacy of integrating sparse MoE within masked diffusion frameworks (Zhu et al., 29 Sep 2025).

5. Design Considerations and Innovations

The integration of sparse MoE and diffusion objectives is enabled by explicit architectural and training choices:

• Sparse MoE scaling: 7B parameter total capacity but <1.5B active per inference step.
• Auxiliary regularizers: Load-balancing and router entropy losses prevent expert collapse and maintain mixture diversity.
• Masked diffusion training: MDM corruption kernel during both pretraining and instruction tuning.
• Blockwise decoding: Parallelizes generation for improved throughput.

This approach opens a path for scaling MDMs further, enabling larger models without incurring prohibitive inference costs.

6. Limitations and Future Directions

The current study is limited by single-model size (7B total parameters). Future work is anticipated in several directions:

• Scaling expert pool size.
• Developing improved routing algorithms for MoE balancing.
• Exploring more efficient block decoding schedules.
• Extending masked diffusion with sparse MoE to multimodal input or retrieval-augmented contexts.

A plausible implication is that these developments may shift the established efficiency–performance Pareto frontier for large diffusion-based language modeling (Zhu et al., 29 Sep 2025). The LLaDA-MoE models are available on Huggingface for further research collaborations and benchmarking.