LLaDA: Diffusion Language Models

• LLaDA is a family of masked diffusion language models that iteratively denoise tokens using bidirectional attention, contrasting with traditional autoregressive models.
• Its architecture supports diverse applications, including instruction tuning, vision, speech, and robotic action, offering robust cross-modal performance.
• Empirical studies demonstrate that LLaDA mimics human text characteristics and utilizes efficient decoding and remasking strategies to enhance generative quality.

Searching arXiv for recent LLaDA papers and related follow-ups. LLaDA, short for Large Language Diffusion Model, denotes a family of diffusion-based LLMs that generate discrete token sequences by iterative denoising of masked text rather than left-to-right next-token prediction. In the literature summarized here, the name is used both for a specific 8B-parameter model introduced as a bidirectional transformer trained with a denoising diffusion-style mask process over text, and for an expanding model family that includes instruction-tuned, multimodal, speech, recommendation, robotic, sparse MoE, and frontier-scale descendants (Tarım et al., 14 Jul 2025, You et al., 22 May 2025, Bie et al., 10 Dec 2025). Its defining technical features are bidirectional attention, a forward masking process that progressively replaces tokens with a mask symbol, and a reverse process that fills masked positions over multiple denoising steps; later work treats LLaDA as a representative diffusion LLM for studying detectability, prompt compression, decoding, alignment, efficiency, and multimodal adaptation (Tarım et al., 14 Jul 2025, Huang et al., 18 May 2026, He et al., 30 Jan 2026, Zhu et al., 25 May 2025).

1. Definition and generative formulation

In the text-generation setting, LLaDA is contrasted with autoregressive LLMs such as LLaMA. An autoregressive model factorizes a token sequence $x_{1:N}$ as

$$P(x_{1:N}) = \prod_{t=1}^{N} P(x_t \mid x_{1:t-1}),$$

with causal masking and irreversible left-to-right decoding, whereas LLaDA defines a forward masking process and a reverse denoising process over a sequence $x^{(0)} \to x^{(T)}$, where $x^{(T)}$ is fully masked (Tarım et al., 14 Jul 2025). The learned reverse transition is written as

$q_\theta\bigl(x^{(t-1)} \mid x^{(t)}\bigr),$

and at each reverse step the model predicts all masked positions in parallel, optionally re-masking low-confidence tokens (Tarım et al., 14 Jul 2025).

A more explicit masked-diffusion formulation appears in later analyses of LLaDA as a diffusion LLM. Let $\mathbf{x}_0 = (x_0^1,\dots,x_0^L)$ be a token sequence. The forward process independently keeps or masks each token: $$q_{t \mid 0}(\mathbf{x}_t \mid \mathbf{x}_0) = \prod_{i=1}^L \text{Cat}\left(x_t^i; \alpha_t \delta_{x_0^i} + (1-\alpha_t)\delta_{\text{MASK}}\right),$$ where $\alpha_t$ is a noise scheduler with $\alpha_0 \approx 1$ and $\alpha_T \approx 0$ (Huang et al., 18 May 2026). In a related formulation used for masked diffusion models in the LLaDA-MoE work, the forward process is

$$P(x_{1:N}) = \prod_{t=1}^{N} P(x_t \mid x_{1:t-1}),$$0

and the pretraining objective is

$$P(x_{1:N}) = \prod_{t=1}^{N} P(x_t \mid x_{1:t-1}),$$1

(Zhu et al., 29 Sep 2025).

This family resemblance across papers is substantive: LLaDA is repeatedly characterized as a masked diffusion or discrete diffusion LLM with bidirectional attention and iterative unmasking, rather than a causal decoder (Tarım et al., 14 Jul 2025, Huang et al., 18 May 2026, You et al., 22 May 2025, Bie et al., 10 Dec 2025). This suggests that “LLaDA” functions both as a concrete model name and as an architectural template for diffusion language modeling.

2. Architecture, training, and inference

The core LLaDA-8B model is described as a transformer without causal masking, using bidirectional attention and no KV-cache; to keep costs manageable, feed-forward dimensions are slightly reduced and “vanilla” multi-head attention is used (Tarım et al., 14 Jul 2025). Training proceeds in two stages. First, self-supervised pretraining uses large text corpora with a variable masking ratio $$P(x_{1:N}) = \prod_{t=1}^{N} P(x_t \mid x_{1:t-1}),$$2, so the model learns to reconstruct masked tokens at all masking ratios. Second, supervised fine-tuning uses approximately 4.5M question–answer pairs, masking only the response span while keeping the prompt intact (Tarım et al., 14 Jul 2025).

At inference, LLaDA initializes the target span as fully masked and runs a fixed number of diffusion steps, for example 10–128. Each step predicts tokens at all masked positions and applies a remask strategy; low-confidence predictions can be masked again, or a block-wise semi-autoregressive remask can be used (Tarım et al., 14 Jul 2025). In the detection study, the experimental LLaDA-8B configuration was

$$P(x_{1:N}) = \prod_{t=1}^{N} P(x_t \mid x_{1:t-1}),$$3

with

$$P(x_{1:N}) = \prod_{t=1}^{N} P(x_t \mid x_{1:t-1}),$$4

(Tarım et al., 14 Jul 2025). The prompt-compression study used the same 8B model with 128 denoising steps, maximum generation length 128, block length 32, temperature 0.0, and classifier-free guidance scale 0.0 (Huang et al., 18 May 2026).

Later work generalized this setup. The FourierSampler paper described LLaDA-8B-Instruct and LLaDA1.5-8B as full-bidirectional masked diffusion LLMs that decode in blocks of size $$P(x_{1:N}) = \prod_{t=1}^{N} P(x_t \mid x_{1:t-1}),$$5, typically $$P(x_{1:N}) = \prod_{t=1}^{N} P(x_t \mid x_{1:t-1}),$$6, over multiple denoising steps (He et al., 30 Jan 2026). The dLLM framework paper presented LLaDA as a canonical masked diffusion LLM supported by a standardized trainer, sampler, and evaluation pipeline, emphasizing that LLaDA and Dream share core MDLM components despite having historically fragmented codebases (Zhou et al., 26 Feb 2026). LLaDA2.0 extended the family by converting strong autoregressive Mixture-of-Experts models into diffusion models through a three-phase Warmup–Stable–Decay process that moves from block size $$P(x_{1:N}) = \prod_{t=1}^{N} P(x_t \mid x_{1:t-1}),$$7 to full-sequence diffusion and then back to compact block diffusion (Bie et al., 10 Dec 2025).

A recurring misconception is that diffusion LLMs are merely masked LLMs used at inference time. The literature instead presents LLaDA as a generative model with a formal forward masking process, a reverse denoising process, multi-step sampling, and explicit training objectives tied to those dynamics (Tarım et al., 14 Jul 2025, You et al., 22 May 2025, Zhu et al., 29 Sep 2025).

3. Empirical behavior relative to autoregressive models

A major line of work examines how LLaDA behaves differently from autoregressive LLMs under evaluation and detection. In “Can You Detect the Difference?” the authors compare LLaDA outputs, LLaMA outputs, and human-written text on 500 arXiv papers under two tasks: rephrasing and title-conditioned abstract generation, yielding 3000 examples in total (Tarım et al., 14 Jul 2025). On the rephrase task, perplexity was $$P(x_{1:N}) = \prod_{t=1}^{N} P(x_t \mid x_{1:t-1}),$$8 for human originals, $$P(x_{1:N}) = \prod_{t=1}^{N} P(x_t \mid x_{1:t-1}),$$9 for LLaMA, and $x^{(0)} \to x^{(T)}$0 for LLaDA; the difference between human and LLaDA was marked non-significant, while human versus LLaMA and LLaMA versus LLaDA had $x^{(0)} \to x^{(T)}$1 by Mann–Whitney U (Tarım et al., 14 Jul 2025). On the same task, burstiness was $x^{(0)} \to x^{(T)}$2 for human text, $x^{(0)} \to x^{(T)}$3 for LLaMA, and $x^{(0)} \to x^{(T)}$4 for LLaDA; lexical diversity measured by TTR was 0.770 for humans, 0.745 for LLaMA, and 0.834 for LLaDA (Tarım et al., 14 Jul 2025).

These measurements led to a specific claim about detectability: LLaDA paraphrases exhibit perplexity and burstiness essentially indistinguishable from human text, while LLaMA paraphrases have very low perplexity and are much easier for autoregressive-oriented stylometric detectors to flag (Tarım et al., 14 Jul 2025). The paper argues that relying on any single metric fails to separate diffusion outputs from human writing, and calls for diffusion-aware detectors, hybrid multi-metric classifiers, diffusion-specific stylometric signatures, and robust watermarking (Tarım et al., 14 Jul 2025). This does not amount to a direct benchmark against DetectGPT or GPTZero, because the study uses stylometric metrics as proxies rather than full detector runs; the paper itself identifies direct evaluation against real detectors as an open problem (Tarım et al., 14 Jul 2025).

Prompt sensitivity is another domain where LLaDA diverges from autoregressive assumptions. In “Prompt Compression in Diffusion LLMs: Evaluating LLMLingua-2 on LLaDA,” prompt compression at approximately $x^{(0)} \to x^{(T)}$5 compression ratio transferred unevenly from autoregressive settings to LLaDA (Huang et al., 18 May 2026). On GSM8K, original-prompt answers aligned with ground truth 52.0% of the time, whereas compressed-prompt answers aligned 35.6% of the time (Huang et al., 18 May 2026). Yet prompt reconstruction still achieved BLEU 0.2641, ROUGE-L-F1 0.5152, and BERTScore-F1 0.9363, showing high semantic similarity despite degraded downstream reasoning (Huang et al., 18 May 2026). The paper concludes that semantic preservation does not necessarily imply stable downstream behavior in diffusion models, and that BERTScore recall being consistently lower than precision suggests information omission rather than semantic drift (Huang et al., 18 May 2026). A plausible implication is that LLaDA’s iterative denoising depends more strongly on prompt details that act as global constraints across all denoising steps.

Alignment results further distinguish the family. “LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models” addresses preference optimization for masked diffusion models by proposing Variance-Reduced Preference Optimization, or VRPO (Zhu et al., 25 May 2025). Applied to LLaDA, it yields LLaDA 1.5 with reported gains of +4.7 on GSM8K, +3.0 on HumanEval, +1.8 on MBPP, +4.0 on IFEval, and +4.3 on Arena-Hard relative to the SFT-only predecessor (Zhu et al., 25 May 2025). The paper’s analysis is framed around ELBO-based likelihood estimates for diffusion models rather than exact autoregressive log-likelihoods, and introduces optimal Monte Carlo budget allocation and antithetic sampling as unbiased variance reduction strategies (Zhu et al., 25 May 2025).

4. Decoding, scaling, and efficiency research

LLaDA has also become a platform for studying decoding algorithms and efficient inference in diffusion LLMs. “FourierSampler: Unlocking Non-Autoregressive Potential in Diffusion LLMs via Frequency-Guided Generation” analyzes hidden states in the frequency domain and reports that low-frequency components primarily encode global structural information and long-range dependencies, whereas high-frequency components characterize local details (He et al., 30 Jan 2026). Based on this, FourierSampler uses a sliding frequency window and an adaptive calibrator to guide a “structure-to-detail” decoding schedule. On LLaDA1.5-8B, it improved average benchmark performance from 46.91 to 50.93, with MBPP rising from 42.02 to 50.58, a 20.4% relative improvement; on LLaDA-8B-Instruct, MBPP improved from 41.25 to 47.86, a 16.0% relative gain (He et al., 30 Jan 2026). The paper states that LLaDA1.5-8B plus FourierSampler surpasses Llama3.1-8B-Instruct and Qwen2.5-7B-Instruct on the reported average without retraining (He et al., 30 Jan 2026).

Acceleration research addresses the fact that LLaDA’s bidirectional attention and iterative denoising preclude straightforward reuse of autoregressive KV caching. “Dynamic-dLLM: Dynamic Cache-Budget and Adaptive Parallel Decoding for Training-Free Acceleration of Diffusion LLM” describes diffusion LLM inference as scaling close to $x^{(0)} \to x^{(T)}$6 when the number of denoising steps grows with sequence length, then proposes Dynamic Cache Updating and Adaptive Parallel Decoding to reduce recomputation and finalize confident tokens earlier (Wu et al., 27 May 2026). On LLaDA-8B-Instruct, Dynamic-dLLM with feature-cache and parallel decoding achieved an average throughput of 34.14 tokens per second versus 10.62 for the baseline, a 3.21× speedup; on GSM8K, throughput rose from 8.32 to 37.29 tokens per second with a small accuracy difference, 61.08 versus 60.67 (Wu et al., 27 May 2026). On LLaDA-1.5, the same framework raised average throughput from 8.30 to 37.02 tokens per second, a 3.20× speedup (Wu et al., 27 May 2026). These are training-free inference modifications rather than architectural retraining.

Compression and internal dynamics have revealed unusually strong robustness. “Layer Collapse in Diffusion LLMs” studies LLaDA-8B and identifies a layer-collapse property in which several early layers exhibit highly similar activation patterns dominated by a single super-outlier channel (Conzelmann et al., 7 May 2026). Pruning that outlier collapses LLaDA-8B’s GSM8K accuracy from baseline to 0%, whereas pruning the strongest channel in Llama-3.1-8B reduces accuracy from 83% to 79% in the paper’s setup (Conzelmann et al., 7 May 2026). Yet LLaDA is simultaneously more robust to compression overall: under 3-bit GPTQ quantization, LLaDA’s GSM8K score drops only -1.8%, whereas Llama-3.1-8B drops -64.7% (Conzelmann et al., 7 May 2026). The same study reports that at 50% average sparsity, allocating more sparsity to early layers in LLaDA yields +8.4% over the reverse strategy, while the same allocation costs Llama -8.4%, implying an inversion of sparsity heuristics relative to autoregressive models (Conzelmann et al., 7 May 2026).

Scaling work culminates in LLaDA2.0. “LLaDA2.0: Scaling Up Diffusion LLMs to 100B” introduces LLaDA2.0-mini and LLaDA2.0-flash, built by converting auto-regressive models into diffusion models with block-level WSD training, then aligning them with SFT and DPO (Bie et al., 10 Dec 2025). On 47 benchmarks, LLaDA2.0-flash reaches an average score of 73.18, compared with 73.60 for Qwen3-30B-A3B-Instruct-2507 and 72.15 for the AR Ling-flash-2.0 baseline (Bie et al., 10 Dec 2025). The same paper reports 535 tokens per second for LLaDA2.0-flash-CAP versus roughly 237–256 tokens per second for the compared autoregressive systems on selected benchmarks, yielding up to 2.1× faster inference (Bie et al., 10 Dec 2025). This suggests that LLaDA-style diffusion is no longer confined to small or purely academic prototypes.

5. Multimodal and domain-specific extensions

LLaDA has been extended across several modalities and application domains. The first major multimodal derivative is “LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning,” which combines LLaDA-8B-Instruct with a SigLIP2 vision encoder and a two-layer MLP projector (You et al., 22 May 2025). Trained on LLaVA-Pretrain, MAmmoTH-VL, and VisualWebInstruct-derived reasoning mixtures, LLaDA-V was reported to be highly competitive with LLaMA3-V when trained on the same instruction data and to show better data scalability on several multimodal benchmarks (You et al., 22 May 2025). For example, on MMMU it achieved 48.6 versus 45.4 for LLaMA3-V; on MMMU-Pro standard, 35.2 versus 28.3; on MMStar, 60.1 versus 56.5 (You et al., 22 May 2025). The same paper states that LLaDA-V achieves state-of-the-art multimodal understanding among existing hybrid autoregressive-diffusion and purely diffusion-based multimodal LLMs (You et al., 22 May 2025).

Biomedical adaptation appears in “LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding,” which uses LLaDA-8B-Instruct, SigLIP2, and a projector in a three-stage biomedical vision instruction tuning pipeline (Dong et al., 3 Aug 2025). On an open-ended biomedical visual conversation task, LLaDA-MedV reports relative performance gains of 7.855% over LLaVA-Med and 1.867% over LLaDA-V (Dong et al., 3 Aug 2025). On closed-form biomedical VQA subsets, it reports 84.93% on VQA-RAD, 92.31% on SLAKE, and 95.15% on PathVQA (Dong et al., 3 Aug 2025). The paper also notes that explicit response-length control in LLaDA-MedV can produce reasonably longer and more informative outputs than LLaVA-Med (Dong et al., 3 Aug 2025).

LLaDA-V has also been adapted for robotic manipulation. “LLaDA-VLA: Vision Language Diffusion Action Models” presents the first Vision-Language-Diffusion-Action model built on pretrained diffusion VLMs (Wen et al., 8 Sep 2025). Its two core additions are a localized special-token classification strategy, which restricts action prediction to a dedicated set of action tokens rather than the full vocabulary, and a hierarchical action-structured decoding strategy that respects dependencies within and across actions (Wen et al., 8 Sep 2025). The paper states that LLaDA-VLA significantly outperforms state-of-the-art VLAs on both simulation and real-world robots (Wen et al., 8 Sep 2025). Because the excerpted details are partial, this conclusion should be read as the paper’s reported result rather than a benchmark table reproduced here.

Speech work shows that the LLaDA paradigm extends beyond text and vision. “Audio-Conditioned Diffusion LLMs for ASR and Deliberation Processing” adapts LLaDA-8B-Instruct to automatic speech recognition by conditioning it on Whisper-derived acoustic embeddings (Wang et al., 20 Sep 2025). In a cascade setup with Whisper-LLaMA, the best system reaches 2.25%/4.94% WER on LibriSpeech test-clean/test-other, which the paper describes as a 12.3% relative improvement over the Whisper-LLaMA baseline on test-other (Wang et al., 20 Sep 2025). It also reports that a plain-text LLaDA without acoustic features fails to improve accuracy, underscoring that audio conditioning is essential in this setting (Wang et al., 20 Sep 2025).

A related speech-generation adaptation, “LLaDA-TTS: Unifying Speech Synthesis and Zero-Shot Editing via Masked Diffusion Modeling,” replaces the autoregressive LLM of an LLM-based TTS system with a masked discrete diffusion decoder (Fan et al., 27 Mar 2026). Using only 50 hours of fine-tuning data, it reports 0.98% CER on Chinese and 1.96% WER on English on Seed-TTS-Eval at 64 steps, matching the original CosyVoice 3 baseline while delivering a 2× LLM-stage speedup (Fan et al., 27 Mar 2026). The same paper argues that bidirectional masked diffusion enables zero-shot word-level insertion, deletion, and substitution without additional training (Fan et al., 27 Mar 2026).

6. Variants, family members, and research significance

Several papers characterize LLaDA less as a single model than as a research family. “LLaDA-MoE: A Sparse MoE Diffusion LLM” introduces a sparse Mixture-of-Experts diffusion model trained from scratch on approximately 20T tokens, with 7B total non-embedding parameters but only 1.4B active parameters per token (Zhu et al., 29 Sep 2025). On the paper’s aggregated benchmark average, LLaDA-MoE-7B-A1B-Base scores 46.94 versus 43.53 for LLaDA-8B-Base and 46.66 for Dream-v0-Base-7B; the instruct model scores 53.12 versus 42.65 for LLaDA-8B-Instruct, 45.56 for LLaDA-1.5, 46.51 for Dream-v0-Instruct-7B, and 53.51 for Qwen2.5-3B-Instruct (Zhu et al., 29 Sep 2025). The paper presents this as evidence that sparse MoE architectures retain their strengths under masked diffusion objectives (Zhu et al., 29 Sep 2025).

In multimodal efficiency, “Efficient Token Pruning for LLaDA-V” analyzes attention patterns in LLaDA-V and reports delayed semantic aggregation: strong cross-modal alignment emerges in middle-to-late layers rather than early ones (Wan et al., 28 Jan 2026). Based on this, the authors propose LLaDA-FastV, pruning visual tokens at later layers of the first denoising step and persisting the reduced token set across subsequent steps (Wan et al., 28 Jan 2026). Their recommended configuration, $x^{(0)} \to x^{(T)}$7 and $x^{(0)} \to x^{(T)}$8, retains 98.7% average relative performance while leaving 51% of baseline FLOPs, a 49% reduction; a more aggressive $x^{(0)} \to x^{(T)}$9 retains 95.3% of performance with 37% of baseline FLOPs, a 63% reduction (Wan et al., 28 Jan 2026). This supports the claim that diffusion-based multimodal models require pruning strategies different from those used in autoregressive decoders.

Infrastructure work has treated LLaDA as a reference model class. “dLLM: Simple Diffusion Language Modeling” introduces an open-source framework that reproduces, fine-tunes, deploys, and evaluates open-source diffusion LLMs such as LLaDA and Dream through a standardized pipeline (Zhou et al., 26 Feb 2026). The framework includes masked diffusion trainers, samplers, and evaluation recipes matching official LLaDA settings, and reports near-reproduction of official LLaDA metrics on benchmarks such as MMLU, GSM8K, MATH, HumanEval, and MBPP (Zhou et al., 26 Feb 2026). This suggests that LLaDA has matured enough to motivate ecosystem-level standardization rather than isolated model releases.

A broader significance emerges across these papers. First, LLaDA systematically challenges autoregressive assumptions about decoding, calibration, compression, and stylometric detection (Tarım et al., 14 Jul 2025, He et al., 30 Jan 2026, Conzelmann et al., 7 May 2026). Second, it functions as a transferable backbone architecture across text, vision, speech, TTS, recommendation, and action modeling (You et al., 22 May 2025, Dong et al., 3 Aug 2025, Wang et al., 20 Sep 2025, Fan et al., 27 Mar 2026, Shi et al., 9 Nov 2025, Wen et al., 8 Sep 2025). Third, its later variants indicate that diffusion language modeling is no longer confined to small dense models; it now includes sparse MoE systems and converted 100B-scale frontier models (Zhu et al., 29 Sep 2025, Bie et al., 10 Dec 2025).

This suggests that LLaDA occupies a position analogous to early GPT-family models in autoregressive language modeling: simultaneously a concrete set of architectures and a template around which a broader methodological literature is being organized. Whether diffusion LLMs will displace autoregressive systems in general-purpose deployment remains unsettled, but the published record now supports a narrower conclusion: LLaDA established a viable masked-diffusion alternative for large language modeling, and subsequent work has shown that the paradigm scales, adapts across modalities, and exposes qualitatively different behaviors in generation, inference, and representation learning (Tarım et al., 14 Jul 2025, You et al., 22 May 2025, Bie et al., 10 Dec 2025).