Large Language Diffusion Models (LLDMs)

• Large Language Diffusion Models (LLDMs) are diffusion-based language models that iteratively denoise masked token sequences using bidirectional attention to improve reasoning and structured generation.
• They integrate masked token systems with block-wise, hybrid, and multimodal decoding strategies to achieve enhanced output coherence and efficiency.
• LLDM research tackles unique challenges in alignment, safety, and long-context scalability, presenting promising alternatives to traditional autoregressive LLMs.

Large Language Diffusion Models (LLDMs) are large-scale diffusion-based LLMs that replace left-to-right next-token prediction with iterative denoising over discrete token sequences. In the contemporary literature, they are often framed as large discrete diffusion LLMs (dLLMs), especially in masked-token systems such as LLaDA, Dream, SDAR, and iLLaDA, and are distinguished from autoregressive LLMs by parallel token refinement, fully bidirectional attention, and denoising-based generation rather than causal factorization (Yu et al., 16 Jun 2025). Recent work presents LLDMs not as a single architecture but as a family spanning pure full-sequence diffusion, block-wise and hybrid decoding, and multimodal extensions, with growing evidence that they can be competitive on reasoning, coding, and structured-generation tasks while introducing distinct alignment, efficiency, and safety problems (Bertolani et al., 17 Jun 2026).

1. Formal framework and model taxonomy

The standard mathematical template for LLDMs is a discrete diffusion process over token sequences. In the general D3PM-style formulation, the forward and reverse processes are written as

$$q(x_{1:T} \mid x_0) = \prod_{t=1}^{T} q(x_t \mid x_{t-1}), \qquad p_\theta(x_{0:T}) = p(x_T)\prod_{t=1}^{T} p_\theta(x_{t-1}\mid x_t).$$

For LLMs, the dominant practical instantiation is absorbing-state masked diffusion, where tokens are progressively replaced by a special mask state. A commonly used simplified corruption process is

$$q(x_t \mid x_0)=\mathrm{Cat}(x_t;\alpha_t x_0 + (1-\alpha_t)m),$$

with $m$ denoting the mask token and $\alpha_t$ a noise schedule (Yu et al., 16 Jun 2025).

A major conceptual distinction in the literature separates continuous diffusion in embedding space from discrete diffusion over tokens. Continuous DLMs preserve smooth Gaussian-style corruption and tractable intermediate states, but operate in non-symbolic state spaces; discrete DLMs preserve token discreteness and explicit transition kernels, but their corruption is coarse and stepwise. The paper on discreteness formalizes this tension through five properties: diffusion-side properties D1 smooth corruption, D2 tractable intermediate states, and D3 iterative reverse generation, and language-side properties L1 discreteness and L2 structural dependency. Its core claim is that current LLDM families satisfy these only partially: continuous systems are more faithful to diffusion mechanics, whereas discrete systems are more faithful to tokenized language (Jin et al., 27 Dec 2025).

This taxonomy is significant because many empirical and theoretical questions about LLDMs reduce to how these trade-offs are resolved. Masked discrete systems align well with symbolic generation and foundation-model scaling, but they inherit challenges that do not arise in classical image diffusion, especially the uneven informational effect of masking across positions and the difficulty of maintaining joint coherence when multiple tokens are denoised in parallel (Jin et al., 27 Dec 2025).

2. Training objectives, pretraining recipes, and architectural inheritance

The dominant training objective in modern LLDMs is a weighted masked reconstruction loss over randomly corrupted sequences. In iLLaDA, a clean sequence $x_0$ is corrupted by sampling a mask ratio $t \sim U[0,1]$ and masking each token independently with probability $t$, yielding

$$\mathcal{L}(\theta) \triangleq -\mathbb{E}_{t, x_0, x_t} \left[ \frac{1}{t} \sum_{i=1}^{L} 1[x_t^i = \textrm{M}] \log p_{\theta}(x_0^i \mid x_t) \right].$$

A notable design choice in iLLaDA is that the same masked diffusion objective is retained in both pretraining and supervised fine-tuning; prompt tokens, response tokens, and EOS can all be masked during SFT. The model is an 8B masked diffusion LM trained from scratch with fully bidirectional attention, scaling pretraining to 12T tokens and instruction fine-tuning to a 25B-token corpus for 12 epochs (Nie et al., 24 Jun 2026).

The training literature also includes a second trajectory in which diffusion models are derived from autoregressive backbones. AR-MAP makes this architectural inheritance explicit by treating a preference-aligned autoregressive LLM as an implicit teacher for a homologous DLLM. It defines a preference task vector

$$T_{\text{pref}} = W^{\text{aligned}}_{\text{AR}} - W_{\text{AR}},$$

and a diffusion adaptation vector

$$T_{\text{diffusion}} = W_{\text{DLLM}} - W_{\text{AR}},$$

then approximates the aligned diffusion model as

$$q(x_t \mid x_0)=\mathrm{Cat}(x_t;\alpha_t x_0 + (1-\alpha_t)m),$$0

The method relies on homologous architectures such as Qwen3-8B-Base $$q(x_t \mid x_0)=\mathrm{Cat}(x_t;\alpha_t x_0 + (1-\alpha_t)m),$$1 SDAR-8B-Instruct and Qwen2.5-7B $$q(x_t \mid x_0)=\mathrm{Cat}(x_t;\alpha_t x_0 + (1-\alpha_t)m),$$2 Dream-7B-Instruct, and interprets alignment transfer as model editing or task arithmetic in a shared parameter space. The same paper argues that direct diffusion preference optimization is statistically brittle because ELBO-based likelihood ratios are doubly stochastic and induce high variance in DPO-style objectives (Lin et al., 2 Feb 2026).

A third line of work modifies LLDM attention rather than only its loss. C$$q(x_t \mid x_0)=\mathrm{Cat}(x_t;\alpha_t x_0 + (1-\alpha_t)m),$$3DLM argues that standard DLM attention is fully connected and causally blind, then adds concept-level causal supervision through a teacher-derived causal graph and a V-aware Re-attention mechanism. Its training objective augments the standard DLM loss with causal attention constraints, motivated by the claim that fully connected denoising attention can dilute reasoning structure in long chain-of-thought supervision (Han et al., 27 Nov 2025).

3. Decoding, inference scaling, and long-context extension

Inference in LLDMs is not fixed by the training objective. Token order, unmasking policy, block structure, and denoising budget materially affect both quality and runtime. The decoding paper “Decoding Large Language Diffusion Models with Foreseeing Movement” recasts decoding order as an optimization problem and argues that local heuristics such as highest probability, highest margin, or lowest entropy ignore downstream effects. Its Foreseeing Decoding Method (FDM) augments local confidence with a global lookahead term,

$$q(x_t \mid x_0)=\mathrm{Cat}(x_t;\alpha_t x_0 + (1-\alpha_t)m),$$4

and FDM-A restricts deep exploration to critical steps. On LLaDA + GSM8K, FDM $$q(x_t \mid x_0)=\mathrm{Cat}(x_t;\alpha_t x_0 + (1-\alpha_t)m),$$5 reaches 82.03% at 8.28 TPS, while FDM-A reaches 81.96% at 42.65 TPS, an approximately 5.15$$q(x_t \mid x_0)=\mathrm{Cat}(x_t;\alpha_t x_0 + (1-\alpha_t)m),$$6 speedup with nearly unchanged accuracy (Mo et al., 3 Dec 2025).

At the systems level, diffusion and autoregressive models realize different hardware trade-offs. A performance characterization of DLMs versus ARMs argues that arithmetic intensity, roofline behavior, and sampling-step count are more informative than FLOPs alone. Naive DLMs have higher arithmetic intensity than ARM decode because they parallelize across positions, but they repeatedly reprocess the full sequence and therefore scale poorly with context length. Block-wise decoding improves this by restricting refinement to active blocks and enabling approximate KV caching; with $$q(x_t \mid x_0)=\mathrm{Cat}(x_t;\alpha_t x_0 + (1-\alpha_t)m),$$7 and $$q(x_t \mid x_0)=\mathrm{Cat}(x_t;\alpha_t x_0 + (1-\alpha_t)m),$$8, block-wise decoding reduced latency by roughly $$q(x_t \mid x_0)=\mathrm{Cat}(x_t;\alpha_t x_0 + (1-\alpha_t)m),$$9 to $m$0 relative to naive DLMs. The same study emphasizes that open-source DLMs often use $m$1, and that reducing $m$2 is the main lever for latency improvement, even though naive step reduction can collapse quality, as illustrated by GSM8K EM dropping from 0.755 at $m$3 to 0.069 at $m$4 and HumanEval pass@1 dropping from 0.323 to 0.000 (Kim et al., 5 Oct 2025).

Long-context scaling introduces additional diffusion-specific constraints because LLDMs use bidirectional attention rather than causal attention. UltraLLaDA extends LLaDA from 4K to 128K context via post-training, a diffusion-aware RoPE extension, and boundary-aware long-sequence packing. Its central argument is that autoregressive NTK-aware RoPE scaling underestimates the effective relative span of a diffusion model, since a model trained on length $m$5 can attend over approximately $m$6. Empirically, UltraLLaDA maintains perplexity around 11–12 from 4K to 128K, achieves 100% NIAH retrieval accuracy across all tested lengths up to 128K, and scores 73.63 on RULER-32K versus 5.69 for LongLLaDA (He et al., 12 Oct 2025).

4. Alignment, safety, watermarking, and uncertainty estimation

Preference alignment in LLDMs is more difficult than in autoregressive models because exact next-token log-likelihoods are unavailable and ELBO-based surrogates are noisy. AR-MAP addresses this by shifting preference optimization into the autoregressive teacher and transferring the resulting preference delta into the diffusion student through weight scaling. Across six benchmarks covering truthfulness, helpfulness, and mathematical reasoning, the paper reports a 69.08% average score across all tasks and models, with AR-MAP reaching 72.23 on SDAR-8B-Instruct and 65.94 on Dream-7B-Instruct, outperforming or matching direct DLLM alignment baselines on average. The authors also report that the reward-based scaling search finds the best or near-best factor in 11 of 12 experimental settings (Lin et al., 2 Feb 2026).

Safety work shows that LLDM vulnerabilities are architecture-specific rather than inherited unchanged from autoregressive jailbreak literature. PAD argues that apparent robustness to standard jailbreak prompts is largely an artifact of architectural mismatch: Direct, GCG, PAIR, and Slice rely on single-point left-to-right steering, whereas LLDMs make decisions through distributed parallel denoising. Its Multi-Point Attention Attack injects benign sequence connectors at multiple positions to seed harmful trajectories, and on four 8B LLDMs it reports PAD-Step attack success rates of 90% on LLaDA-Base, 86% on LLaDA-Ins, 91% on MMaDA-Base, and 97% on MMaDA-Mix. The same study reports that, relative to same-sized autoregressive LLMs, LLDMs can generate harmful content about 2$m$7 faster under successful jailbreak settings (Zhang et al., 25 Jul 2025).

Order-agnostic generation also breaks conventional provenance and reliability tooling. DMark shows that naive adaptation of autoregressive watermarking is inadequate because prefix context is often unavailable during out-of-order generation; its predictive-bidirectional watermarking restores detectability and achieves 92.0–99.5% detection rates at 1% false positive rate, compared with 49.6–71.2% for direct KGW adaptation (Wu et al., 3 Oct 2025). In parallel, LLDM-specific uncertainty quantification work argues that autoregressive likelihood-based UQ and repeated-sampling UQ are both misaligned with denoising generation. It proposes trajectory-based signals derived from intermediate generations, remasking dynamics, and denoising complexity, and proves that expected trajectory dissimilarity lower-bounds the masked diffusion training objective. Its D-CoCoA variants approach the strongest sampling-based baselines while incurring up to 100$m$8 lower computational overhead (Vazhentsev et al., 14 May 2026).

5. Benchmark performance and domain-specific applications

LLDMs have advanced beyond proof-of-concept status in core language benchmarking. iLLaDA presents an 8B masked diffusion LLM trained from scratch with fully bidirectional attention and reports broad improvements over LLaDA. iLLaDA-Base achieves MMLU 74.8, BBH 71.3, ARC-Challenge 60.8, GSM8K 81.9, HumanEval 50.0, MBPP 57.8, and an average of 63.9; iLLaDA-Instruct achieves MMLU 71.6, GSM8K 89.0, MATH 56.7, HumanEval 65.9, MBPP 58.0, and an average of 67.1. The paper emphasizes gains of +21.6 on BBH and +14.9 on ARC-Challenge for the base model, and +14.5 on MATH and +16.5 on HumanEval for the instruction-tuned model, while remaining competitive with Qwen2.5 7B on several benchmarks (Nie et al., 24 Jun 2026).

Specialized code generation has become a prominent testbed for LLDMs. DICE targets CUDA kernel synthesis and argues that diffusion-style generation is well matched to holistic structural planning and non-sequential refinement. Built on the CuKe dataset and a bi-phase curated reinforcement learning pipeline, DICE reports new state-of-the-art results on KernelBench. DICE-8B attains 40 execution-correct solutions on Level 1, 39 on Level 2, and 16 on Level 3; DICE-4B is notable for outperforming several 8B-scale baselines; and the full pipeline improves over plain SFT and naive RL baselines in both correctness and optimization stability (Bai et al., 12 Feb 2026).

Offline black-box optimization provides another domain where bidirectional refinement is operationally useful. The dLLM framework conditions a diffusion LLM on a natural-language task description and a few-shot offline dataset, then guides denoising with masked diffusion tree search. On Design-Bench few-shot settings with only 10 examples, it reports the best 100th-percentile normalized scores on all four tasks: 0.652 $m$9 0.030 on Ant, 0.942 $\alpha_t$0 0.012 on D’Kitty, 0.876 $\alpha_t$1 0.038 on TF8, and 0.642 $\alpha_t$2 0.013 on TF10, with rank mean and median both 1.0/16 (Yuan et al., 20 Jan 2026).

Multimodal extensions indicate that the LLDM paradigm is not confined to text-only generation. DIFFA introduces a frozen LLaDA-8B-Instruct backbone with semantic and acoustic adapters for spoken language understanding and reports competitive results despite using only 960 hours of ASR and 127 hours of synthetic instruction data. Its reported scores are 56.04 overall on MMSU, 49.71 average on MMAU, and 48.22 average on VoiceBench, outperforming several autoregressive open-source baselines (Zhou et al., 24 Jul 2025). In reasoning-specific fine-tuning, C$\alpha_t$3DLM reports a +12.00 percentage point gain on COT-OrderPerturb, from 38.60% to 50.60%, an average gain of 1.31% across six downstream reasoning tasks, and approximately 3.2$\alpha_t$4 training speedup in the synthetic benchmark setting (Han et al., 27 Nov 2025).

6. Limitations, misconceptions, and open research directions

A recurring misconception is that LLDMs are automatically faster than autoregressive LLMs. The comparative evidence is more conditional. A systematic experimental analysis of eight modern DLMs shows that pure full-sequence diffusion can be very expensive in end-to-end generation because it repeatedly denoises the whole sequence: reported generation costs include 23,726.95 TFLOPS for Dream, 25,357.94 TFLOPS for LLaDa, and 25,557.57 TFLOPS for LLaDa-1.5, whereas block-based systems such as Fast-dLLM and SDLM report 33.38 and 6.72 TFLOPS respectively. The same study finds that no single diffusion paradigm dominates all tasks: Dream is particularly strong on globally constrained tasks such as Sudoku, where it reaches 75%, while Fast-dLLM is strongest on GSM8K and HumanEval but weak on HellaSwag and Sudoku (Bertolani et al., 17 Jun 2026).

A second limitation is structural rather than merely engineering-related. The discreteness analysis argues that uniform masking does not respect how information is distributed across positions and that token-wise marginal training cannot capture multi-token dependencies during parallel decoding. Its empirical discussion identifies frequency collapse, in which distant masked positions drift toward common tokens or <eos>, and the “Marginal Trap,” in which independently plausible token marginals can still compose into invalid joint sequences. The paper therefore recommends information-aware corruption, sequence-level or structured objectives, soft intermediate states, and stronger modeling of joint compatibility across masked spans (Jin et al., 27 Dec 2025).

The broader survey literature places these problems within a wider research agenda. Current priorities include better step-efficient samplers, adaptive unmasking and remasking policies, stronger caching and prefilling infrastructure, distillation for fewer-step generation, diffusion-specific multimodal architectures, memorization mitigation, and safer deployment protocols. Alignment-specific work also points to more granular transfer mechanisms, since AR-MAP explicitly notes that a global scaling factor is an oversimplification and that layer-wise or parameter-wise adaptive scaling may be needed (Yu et al., 16 Jun 2025). Taken together, these results suggest that LLDMs are best understood neither as straightforward replacements for autoregressive LLMs nor as a mere change in sampler, but as a distinct foundation-model regime whose strengths in parallel denoising, bidirectional conditioning, and iterative refinement are inseparable from new challenges in coherence, efficiency, alignment, and safety.