LLaDA-8B-Base: Diffusion Language Model

• LLaDA-8B-Base is a diffusion-based language model that applies a masked token denoising process to iteratively reconstruct text, offering a bidirectional alternative to autoregressive methods.
• It employs a 32-layer bidirectional Transformer with 8.02 billion parameters, integrating techniques such as SwiGLU activation and RMSNorm for robust token modeling.
• Empirical evaluations demonstrate competitive performance on benchmarks, superior bidirectional reasoning, and exceptional robustness with model compression techniques like 3-bit GPTQ.

LLaDA-8B-Base is a large-scale, discrete diffusion LLM based on a bidirectional Transformer architecture, designed to challenge the dominant paradigm of autoregressive LLMs. It employs a masked diffusion process for pre-training and generation, directly modeling distributions over token sequences by iterative masking and denoising. LLaDA-8B-Base attains empirical performance competitive with state-of-the-art autoregressive models (ARMs) such as LLaMA3-8B, while demonstrating distinctive advantages in bidirectional reasoning tasks, model compression robustness, and prompt processing characteristics (Nie et al., 14 Feb 2025, Conzelmann et al., 7 May 2026, Huang et al., 18 May 2026).

1. Architecture and Diffusion Model Formulation

LLaDA-8B-Base employs a standard, 32-layer bidirectional Transformer with 4,096 model dimension, 12,288-dimensional feed-forward networks (FFN), 32 attention heads, and RMSNorm normalization after each attention and MLP sublayer. Its architecture closely matches LLaMA3-8B to ensure fair comparisons, differing mainly by the use of vanilla multi-head attention (avoiding key/value caching), a 126,464-token vocabulary, and the SwiGLU activation in the FFN. The model totals approximately 8.02 billion parameters, with about 6.98 billion excluding embeddings (Nie et al., 14 Feb 2025).

The hallmark innovation of LLaDA-8B-Base is its discrete masking diffusion model formulation:

• Forward process: For each original token $x_0^i$ in the input $x_0 \in V^L$, masking is applied independently at time $t\in [0,1]$ with probability $t$, producing $x_t$ where unmasked tokens retain their identity and others are replaced by a special mask token $M$.
• Reverse process: Generation starts from a fully masked sequence and iteratively denoises towards the original text. At each discrete time step, masked positions are predicted by the bidirectional Transformer, and a fraction of tokens is unmasked according to either random or low-confidence strategies.
• Training objective: The model is trained to minimize a cross-entropy loss over masked tokens, scaled by $1/t$, which serves as an upper bound to the negative log-likelihood of the implicit denoising process. This yields a principled maximum-likelihood estimator for discrete sequence modeling (Nie et al., 14 Feb 2025, Conzelmann et al., 7 May 2026).

2. Training Regime: Pre-training and Supervised Fine-Tuning

Pre-training for LLaDA-8B-Base utilizes 2.3 trillion tokens drawn from web, code, math, and multilingual sources, filtered for quality using rule-based and LLM-based heuristics. Fixed-length (4,096-token) examples are primarily used, with a small proportion for variable-length handling. AdamW with learning-rate warmup, plateaus, and staged decay is applied, accumulating $1.3\times10^5$ H800-GPU hours.

Supervised fine-tuning (SFT) is performed on 4.5 million prompt–response pairs spanning instructions, code, math, and structured data. SFT retains the masking-diffusion training paradigm but masks only response tokens at each step, leaving the prompt unmasked; the same $1/t$-scaled loss is applied. Crucially, reward modeling or RL alignment is not employed; SFT alone imparts strong instruction-following capabilities (Nie et al., 14 Feb 2025).

3. Empirical Performance and Comparative Evaluation

Zero-/Few-Shot and Scaling Trends

Extensive benchmarking positions LLaDA-8B-Base close to or above strong ARMs in multiple domains. On six standard tasks (MMLU, ARC-C, CMMLU, PIQA, GSM8K, HumanEval), LLaDA exhibits equivalent or superior scaling and matches or exceeds baseline ARMs on math (GSM8K) and Chinese (CMMLU, C-Eval) tasks, despite being trained on fewer tokens than LLaMA3-8B. A sample comparison:

Model	MMLU	ARC-C	GSM8K (4-shot)
LLaDA-8B Base	65.9	47.9	70.7
LLaMA3-8B	65.4	53.1	53.1

Instruction Fine-Tuning and Bidirectional Reasoning

LLaDA-8B-Instruct attains high accuracy without RL alignment, achieving 88.5% on ARC-C (0-shot) and 78.6% on GSM8K (4-shot). Remarkably, on the reversal poem completion task—a benchmark for bidirectional inference—LLaDA maintains nearly symmetric accuracy in both forward and reversed directions (48.8%/42.4%), whereas ARMs suffer severe degradation in the reversed setting (e.g., GPT-4o: 82.7%/34.3%). This reflects the unique bidirectional context integration of diffusion models (Nie et al., 14 Feb 2025).

4. Activation Dynamics and Compression Robustness

LLaDA-8B manifests a distinctive "layer collapse" phenomenon in early and mid Transformer layers. Most channels in these layers exhibit high redundancy, with nearly collinear activations ($\mathrm{sim}(i,j) \approx 0.95$–$x_0 \in V^L$0); however, a single "super-outlier" channel dominates, carrying the vast majority of reconstruction information. Pruning this channel leads to catastrophic token looping and total loss of task accuracy, but pruning or high sparsity in other channels has negligible impact. This collapse arises from overtraining (as measured by low Hill exponents $x_0 \in V^L$1 in early layers), sharply contrasting with ARM "curse of depth" undertraining in deeper layers (Conzelmann et al., 7 May 2026).

Compression studies show LLaDA-8B-Base to be unusually robust:

• 3-bit GPTQ quantization: Only a –1.8% accuracy drop on GSM8K, in contrast to –64.7% for Llama-3.1-8B under identical conditions.
• Optimal sparsity allocation: Early layers can be pruned more aggressively ("earlier-is-sparser", EIS); at 50% average sparsity, this yields $x_0 \in V^L$28.4% accuracy advantage versus the conventional "deeper-is-sparser" (DIS) strategy optimal for ARMs.

Deployment in resource-constrained settings requires only post-training pipelines for pruning and quantization; no further fine-tuning is needed to maintain high practical utility (Conzelmann et al., 7 May 2026).

5. Inference, Likelihood Estimation, and Prompt Compression

Diffusion-Based Generation and Quality Control

Inference in LLaDA-8B uses a uniform diffusion step schedule (typically $x_0 \in V^L$3), gradually unmasking tokens in parallel. Accuracy on GSM8K and HumanEval increases as $x_0 \in V^L$4 approaches sequence length $x_0 \in V^L$5, allowing adjustable trade-offs between speed and quality. While slower than single-pass autoregression, aggressive remasking and semi-autoregressive blocking substantially reduce effective calls per sequence (Nie et al., 14 Feb 2025).

Likelihood estimation for evaluation, including MC-based estimators and classifier-free guidance, supports robust conditional inference in classification and multi-choice setups.

Prompt Compression Effects

Prompt compression, primarily developed for ARMs, shows non-uniform transferability to LLaDA-8B. Application of LLMLingua-2 achieves $x_0 \in V^L$6 compression, but semantic preservation (e.g., BERTScore F1 $x_0 \in V^L$70.94) does not guarantee behavioral consistency:

• Summarization tasks (DUC2004, ShareGPT) remain robust: ROUGE-L F1 $x_0 \in V^L$8, BERTScore F1 $x_0 \in V^L$9.
• Mathematical reasoning (GSM8K) is sensitive: accuracy drops from 52% (original) to 35.6% (compressed); agreement between original and reconstructed prompts is only 35.2%.
• Compression analysis: BERTScore precision consistently exceeds recall, implicating information omission rather than drift; omitted tokens often destabilize denoising, imperiling answer fidelity.

A plausible implication is that prompt compression criteria must explicitly account for the global iterative denoising requirements of diffusion models, contrasting with the localized saliency assumptions underpinning ARM compression (Huang et al., 18 May 2026).

6. Theoretical Implications, Best Practices, and Open Directions

LLaDA-8B-Base demonstrates that masked discrete diffusion can serve as a scalable, bidirectional, and highly compressible alternative to AR sequence modeling. Insights emerging from analysis include:

• Compression best practices: Apply early-layer-pruned sparsity and low-bit GPTQ quantization, preserving super-outlier channels with dedicated scales.
• Diffusion-specific training dynamics: The model’s early layers require careful control to avoid super-outlier dominance and maximize representational depth—methods include regularization, per-layer learning rate balancing ($t\in [0,1]$0-adaptive), and hybrid normalization schemes.
• Prompt handling and context: Future prompt compression strategies should be designed diffusion-aware, incorporating global denoising stability and behavioral recovery metrics rather than relying solely on tokenwise saliency or semantic similarity (Conzelmann et al., 7 May 2026, Huang et al., 18 May 2026).

Open challenges include scaling diffusion LLMs to $t\in [0,1]$1B parameters, achieving inference speed parity with ARMs, exploring multi-modal extensions under the discrete diffusion framework, and investigating the interplay between diffusion training and alignment techniques such as RLHF (Nie et al., 14 Feb 2025).

---

References:

(Nie et al., 14 Feb 2025): Large Language Diffusion Models (Conzelmann et al., 7 May 2026): Layer Collapse in Diffusion LLMs (Huang et al., 18 May 2026): Prompt Compression in Diffusion LLMs: Evaluating LLMLingua-2 on LLaDA