Diffusion-LLM: Principles & Architectures

• Diffusion-LLM is a research area that combines diffusion-based iterative denoising with LLM semantic reasoning to enable text generation through parallel decoding and refinement.
• It encompasses both pure diffusion language models and hybrid systems where an LLM conditions a diffusion model for tasks like text-to-image generation and time-series forecasting.
• Key performance factors include inference-time control, token utilization trade-offs, and long-context handling that collectively determine efficiency and output quality.

Searching arXiv for papers on diffusion LLMs and closely related hybrid diffusion–LLM systems. Diffusion-LLM denotes a research area at the intersection of diffusion modeling and LLMs, but the term is used in two distinct senses. In the narrow sense, it refers to diffusion LLMs or large language diffusion models that generate text through an iterative denoising or sampling process rather than one-token-at-a-time next-token prediction (Liu et al., 17 Jun 2025). In a broader sense, it refers to hybrid systems that couple an LLM with a diffusion model so that each component handles a different part of a task, such as text conditioning in text-to-image generation, conditional density learning in time-series forecasting, or mixed discrete–continuous generation in crystal design (Hu et al., 2024). Across both senses, the central motivation is consistent: diffusion contributes iterative refinement, parallel updates, or probabilistic modeling, while the LLM contributes semantic structure, long-context reasoning, or discrete-symbol modeling (Jin et al., 27 Dec 2025).

1. Conceptual scope and terminology

The core distinction between diffusion-based and autoregressive language modeling is procedural. A diffusion LLM defines generation through a forward corruption process that progressively destroys information in a text sequence and a reverse denoising process that learns to recover the original text step by step, whereas an autoregressive model factorizes the sequence left-to-right as

$P(x_{1:n}) = \prod_{i=1}^{n} P(x_i \mid x_{<i}) .$

This procedural difference underlies the usual claims about diffusion LLMs: parallel decoding, flexible editing, and iterative refinement (Jin et al., 27 Dec 2025).

The literature surveyed here uses “Diffusion-LLM” in at least three ways. First, some papers use it to mean LLMs whose decoder itself is diffusion-based, as in LLaDA, Dream, SDLM, Fast-dLLM, and related systems (Bertolani et al., 17 Jun 2026). Second, some papers use it for LLM-conditioned diffusion systems, where an LLM acts as an encoder, conditioner, or semantic prior for a diffusion backbone, as in ELLA and Semantic Routing (Hu et al., 2024). Third, some works use diffusion in the sense of propagation or spread, especially in multi-agent social simulation, where LLMs act as diffusion participants or controllers rather than as denoisers (Liu et al., 11 Jun 2026). The coexistence of these usages is itself a defining feature of the field.

A practical implication is that “Diffusion-LLM” is not a single architecture class. It is an umbrella category spanning discrete token diffusion, continuous latent diffusion, hybrid block-diffusion decoders, and modular LLM–diffusion pipelines (Jin et al., 27 Dec 2025).

2. Diffusion LLMs as text generators

In the narrowest and most technically specific sense, a diffusion LLM is a language generator that corrupts a sequence into noise and iteratively denoises it back into text (Jin et al., 27 Dec 2025). The field is usually partitioned into two main families. Continuous diffusion in embedding space applies Gaussian diffusion to continuous representations and satisfies smooth corruption, tractable intermediate states, and iterative reverse generation, but does not preserve token discreteness directly. Discrete diffusion over tokens keeps the state in token space, often via masking or categorical transition kernels, thereby preserving discreteness and explicit corruption kernels, but only approximates smooth corruption and usually models structural dependency only implicitly (Jin et al., 27 Dec 2025).

A systematic experimental comparison across eight modern DLMs and eight benchmarks reports that no single diffusion paradigm dominates across tasks (Bertolani et al., 17 Jun 2026). Pure full-sequence diffusion models such as Dream and LLaDa are associated with strong global refinement behavior, while block-diffusion or hybrid systems such as Fast-dLLM and LLaDa-1.5 expose a different efficiency–quality profile. On the reported peak configurations, Dream is the strongest diffusion model on MMLU, MMLU Redux, HellaSwag, MBPP, and Sudoku, with 75.00% on Sudoku; Fast-dLLM is strongest among diffusion models on GSM8K: 83.39% and HumanEval: 69.51%; LLaDa-1.5 is the strongest diffusion model on WMT16 En–De with 54.85 chrF (Bertolani et al., 17 Jun 2026).

The same study emphasizes that diffusion models expose inference-time control variables that do not have close autoregressive equivalents: denoising steps, context length, block size, and parallel unmasking ratio (Bertolani et al., 17 Jun 2026). Performance is therefore partly a function of generation protocol rather than architecture alone. Jointly increasing generation length and denoising steps at a 1:1 ratio improves some reasoning and coding tasks initially, but results can saturate or decline after about 256–512 tokens, while WMT16 translation degrades almost monotonically at larger budgets (Bertolani et al., 17 Jun 2026). This makes diffusion language modeling unusually sensitive to deployment-time configuration.

3. Structural tensions in text diffusion

A major line of analysis argues that diffusion mechanics and language structure are not naturally aligned. One formulation separates five desiderata into three diffusion-side properties—smooth corruption, tractable intermediate states, and iterative reverse generation—and two language-side properties—discreteness and structural dependency (Jin et al., 27 Dec 2025). The central thesis is that current diffusion LLMs satisfy only subsets of these properties and therefore embody a structural trade-off.

Two failure modes receive particular emphasis. The first is that uniform corruption does not respect how information is distributed across positions (Jin et al., 27 Dec 2025). In text, some positions are highly informative and constrain many others, while some are predictable from local context. Under uniform masking, the same nominal corruption level can correspond to very different effective information loss across positions. This is formalized by the limiting intuition

$$I(x_i; x_{\mathcal O}) \to 0 \quad \Rightarrow \quad p(x_i \mid x_{\mathcal O}) \to p(x_i),$$

which explains why predictions drift toward common tokens and eventually $\langle eos \rangle$ when visible context becomes uninformative (Jin et al., 27 Dec 2025). This is described as frequency collapse.

The second failure mode is the marginal trap: token-wise marginal training does not guarantee multi-token compatibility during parallel decoding (Jin et al., 27 Dec 2025). The paper’s toy example shows that locally correct marginals can still yield globally inconsistent combinations such as “I likes tennis.” The problem becomes especially severe when intermediate token choices are committed early and later denoising steps must treat them as context. This diagnosis suggests that future diffusion LLMs require mechanisms for coupled multi-token decisions, not merely independent per-token cross-entropy objectives (Jin et al., 27 Dec 2025).

This line of work does not deny the utility of diffusion LLMs; rather, it argues that future progress depends on information-aware corruption, sequence-level or structured objectives, and reduced premature commitment in intermediate states (Jin et al., 27 Dec 2025). A plausible implication is that many current gains from diffusion arise despite a significant model–domain mismatch.

4. Long-context behavior and positional encoding

Long-context behavior has emerged as a specific point of divergence between diffusion LLMs and autoregressive LLMs. A systematic comparison between LLaDA-family diffusion LLMs and autoregressive models reports that diffusion LLMs maintain remarkably stable perplexity during direct context extrapolation, whereas autoregressive models such as LLaMA3 show the expected sharp perplexity increase and retrieval collapse once context exceeds the pretraining window (Liu et al., 17 Jun 2025).

This stability does not imply unrestricted long-range retrieval. In Needle-In-A-Haystack evaluations, diffusion LLMs exhibit a distinct local perception phenomenon: once the context exceeds the training length, they often recover the needle only from the most recent segment of the context, behaving somewhat like a sliding window (Liu et al., 17 Jun 2025). Retrieval depth also depends on diffusion sampling steps: more steps allow LLaDA to recover farther positions, but the effect remains bounded by the pretrained context range (Liu et al., 17 Jun 2025).

The proposed explanation is based on RoPE scaling theory. Auto-regressive models train with one-sided relative positions in $[0, T_{\text{train}}-1]$, because they attend only to past tokens, while diffusion LLMs use bidirectional attention and therefore see symmetric relative positions in $[1-T_{\text{train}}, T_{\text{train}}-1]$ during training (Liu et al., 17 Jun 2025). The paper argues that this richer and more symmetric exposure reduces out-of-distribution positional behavior under extrapolation. RoPE is defined there as

$$\bm{A}_{t,s}=\left(\bm{q}_t\bm{R}_t\right)\left(\bm{k}_s\bm{R}_s\right)^\top=\bm{q}_t\bm{R}_{t-s}\bm{k}_s^\top ,$$

with frequencies

$$\theta_n=\beta_0^{-2n/d}, \quad T_n=2\pi\cdot\beta_0^{2n/d}.$$

The same work introduces LongLLaDA, a training-free context-extension method that combines LLaDA with NTK-based RoPE extrapolation (Liu et al., 17 Jun 2025). For LLaDA-8B, using $\beta_0 = 500000$ and $T_{\text{train}} = 4\text{k}$, the paper reports extrapolation to 8k, 16k, 24k, and 32k, with the practical conclusion that effective extrapolation reaches about $6\times$ the original length, i.e. 24k, without further training (Liu et al., 17 Jun 2025). Empirically, $$I(x_i; x_{\mathcal O}) \to 0 \quad \Rightarrow \quad p(x_i \mid x_{\mathcal O}) \to p(x_i),$$0 and $$I(x_i; x_{\mathcal O}) \to 0 \quad \Rightarrow \quad p(x_i \mid x_{\mathcal O}) \to p(x_i),$$1 work very well, $$I(x_i; x_{\mathcal O}) \to 0 \quad \Rightarrow \quad p(x_i \mid x_{\mathcal O}) \to p(x_i),$$2 reaches a practical limit and shows a lost-in-the-middle pattern, and $$I(x_i; x_{\mathcal O}) \to 0 \quad \Rightarrow \quad p(x_i \mid x_{\mathcal O}) \to p(x_i),$$3 is no longer effective (Liu et al., 17 Jun 2025).

The benchmark picture is explicitly asymmetric. Diffusion LLMs are comparable to autoregressive models on retrieval-oriented tasks, underperform on aggregation tasks such as Variable Tracing and Frequent/Common Word Extraction, outperform on some QA tasks such as SQuAD and Hotpot in RULER, and are consistently stronger than the LLaMA3 series on synthetic tasks on LongBench (Liu et al., 17 Jun 2025). This establishes long-context diffusion LLMs as neither straightforwardly superior nor merely degraded variants of autoregressive systems.

5. Efficiency, serving, and inference-time control

Diffusion LLMs are often motivated by the possibility of better hardware utilization, because they decode multiple tokens per iteration rather than exactly one token per step. A systems study of serving reports that fixed block-size diffusion decoding improves throughput at low load but is highly load-sensitive: large blocks exploit idle GPU resources under low load, yet saturate early and incur substantial redundant computation under high load (Wei et al., 24 May 2026). The study contrasts this with autoregressive serving, where strict sequentiality leads to low GPU utilization; on an A100 with Qwen-8B, utilization can drop below 1% under small-batch workloads, and saturation may not occur until around batch size 512 (Wei et al., 24 May 2026).

The corresponding drawback of fixed-block diffusion is low token utilization. For BD32, the study reports about 3.8 committed tokens per 32 computed tokens, or roughly 12% token utilization (Wei et al., 24 May 2026). This motivates Optimus, a serving system for diffusion LLMs that treats decoding granularity as a runtime control variable and selects chunk sizes according to current load. Its objective is to maximize committed tokens per unit latency,

$$I(x_i; x_{\mathcal O}) \to 0 \quad \Rightarrow \quad p(x_i \mid x_{\mathcal O}) \to p(x_i),$$4

The system combines chunked decoding, prefix caching, suffix chunking, and saturation-aware scheduling (Wei et al., 24 May 2026).

Under the reported experiments, Optimus achieves up to 6.1x throughput improvement over AR decoding and 4.3x improvement over fixed-block diffusion LLM, while preserving model accuracy and improving end-to-end serving capacity under latency constraints (Wei et al., 24 May 2026). Average gains are reported as 2.07× over LMDeploy-AR and 1.31× over LMDeploy-BD32, with larger peak gains under specific loads (Wei et al., 24 May 2026). The broader conclusion is that diffusion LLM efficiency is not merely an architectural property; it depends heavily on runtime scheduling and decoding granularity.

This systems perspective complements the experimental-analysis literature. The latter shows that parallel unmasking and block size affect quality differently by task, while Optimus shows that runtime load affects the optimal decoding granularity even when model weights are fixed (Bertolani et al., 17 Jun 2026). Taken together, these results suggest that diffusion-LLM deployment is unusually dependent on inference policy.

6. Hybrid diffusion–LLM architectures beyond text generation

Outside pure language modeling, “Diffusion-LLM” often denotes a modular architecture in which an LLM provides semantics and a diffusion model handles iterative denoising, uncertainty modeling, or continuous generation.

Subarea	Representative mechanism	Example paper
Text-to-image	LLM as text encoder or multi-layer semantic source for diffusion transformers	(Hu et al., 2024, Li et al., 3 Feb 2026)
Time-series forecasting	Conditional diffusion regularizes an LLM latent forecasting space	(Ghosh et al., 22 Jun 2026)
Crystal generation	LLM generates composition; diffusion refines coordinates and lattice	(Khastagir et al., 27 Oct 2025)
Offline agent evaluation	Diffusion world model simulates environment responses to an LLM policy	(Liu et al., 4 Jun 2026)

In text-to-image generation, ELLA equips pretrained diffusion models with pretrained LLMs through a Timestep-Aware Semantic Connector (TSC), while keeping the U-Net, VAE, and LLM frozen (Hu et al., 2024). The connector converts arbitrary-length LLM hidden states into a fixed set of timestep-dependent semantic tokens so that early denoising can emphasize global structure and later denoising can emphasize fine attributes and style details (Hu et al., 2024). A related study, Semantic Routing, argues that conditioning should exploit the semantic hierarchy across LLM layers and the non-stationary nature of diffusion denoising (Li et al., 3 Feb 2026). It introduces a normalized convex fusion framework,

$$I(x_i; x_{\mathcal O}) \to 0 \quad \Rightarrow \quad p(x_i \mid x_{\mathcal O}) \to p(x_i),$$5

and finds that Depth-wise Semantic Routing is the best conditioning strategy, improving GenAI-Bench from 74.96 for the penultimate-layer baseline to 79.07, including +9.97 on the Counting task (Li et al., 3 Feb 2026). Purely time-wise fusion can degrade fidelity because nominal timesteps do not track the effective SNR under classifier-free guidance (Li et al., 3 Feb 2026).

In time-series forecasting, Diffusion-LLM augments an LLM forecaster with a conditional DDPM that learns the conditional distribution of future latent representations during training and is discarded at inference (Ghosh et al., 22 Jun 2026). The joint objective is

$$I(x_i; x_{\mathcal O}) \to 0 \quad \Rightarrow \quad p(x_i \mid x_{\mathcal O}) \to p(x_i),$$6

The reported gains are strongest in ultra-long-term and few-shot settings: on ETTh1 in the ultra-long-term setting, MSE improves from 0.758 to 0.612, a 19.26% reduction, and under 5% data the same dataset improves by 25.79% relative to TimeLLM (Ghosh et al., 22 Jun 2026). The authors interpret this as evidence that diffusion contributes distribution-aware regularization in settings where uncertainty is most severe.

In materials generation, CrysLLMGen divides the problem into discrete chemistry and continuous geometry (Khastagir et al., 27 Oct 2025). A fine-tuned LLaMA-2-7B first generates atom types, fractional coordinates, and lattice structure; the atom types are retained, while an equivariant diffusion model refines only coordinates and lattice starting from an intermediate timestep $$I(x_i; x_{\mathcal O}) \to 0 \quad \Rightarrow \quad p(x_i \mid x_{\mathcal O}) \to p(x_i),$$7 rather than from pure noise (Khastagir et al., 27 Oct 2025). On MP-20, the hybrid achieves 99.94% structural validity, 93.55% compositional validity, 99.84 coverage precision, and 98.52 coverage recall, and it reports 32% more stable materials than the best LLM-based model and 68% more stable materials than the best denoising model (Khastagir et al., 27 Oct 2025).

In offline evaluation of LLM agents, ADWM uses a latent diffusion world model to simulate how the environment responds to a candidate LLM policy without executing it online (Liu et al., 4 Jun 2026). The work argues that prior diffusion OPE methods assume jointly diffused continuous state-action trajectories, which is incompatible with discrete text actions that must be sampled after observing the current environment. ADWM instead derives a policy-guided autoregressive transition model and reports positive Spearman rank correlation in every benchmark/configuration, ranging from about 0.67 to 0.90, with a mean of about 0.82 (Liu et al., 4 Jun 2026). This suggests a different, simulator-oriented branch of diffusion–LLM research in which diffusion models do not generate language directly but rather model the latent environment dynamics surrounding language agents.

7. Social diffusion, security, and open research questions

A separate but related body of work uses “diffusion” in the sense of information diffusion or emotion diffusion rather than denoising. These studies are relevant because they show where LLMs succeed or fail when used to simulate propagation phenomena.

One multi-agent framework studies dynamic information diffusion under information asymmetry and proposes a Dynamic Attention Algorithm that weights incoming messages according to short-term memory, long-term memory, relationship salience, entropy change, and interaction frequency (Zhang et al., 16 Feb 2025). The work reports that a generic LLM agent retaining all past messages produced highly similar actions across rounds, with mean cosine similarity 0.80, minimum 0.63, maximum 0.94, and standard deviation 0.08, motivating explicit attention control (Zhang et al., 16 Feb 2025). Another framework, MIDSim, models multi-channel information diffusion by combining social exposure and algorithmic exposure in an LLM-powered multi-agent system (Liu et al., 11 Jun 2026). On real diffusion events from Sina Weibo, RedNote, and Twitter, MIDSim reports the lowest error on all platforms and metrics; for example, on Sina Weibo with Qwen2.5-14B it achieves MAPE: 0.047 and MRSE: 0.003 (Liu et al., 11 Jun 2026).

At the same time, a study of emotion diffusion in real and simulated social graphs finds that LLM-generated interaction graphs are structurally much simpler than real Reddit diffusion graphs (Qiang, 24 Dec 2025). Real graphs contain approximately 9,000 unique users and over 13,000 directed edges, with dense connectivity, repeated interactions, sentiment shifts, and emergent community structures, whereas the simulated graph consists of 100 reply chains of the form $$I(x_i; x_{\mathcal O}) \to 0 \quad \Rightarrow \quad p(x_i \mid x_{\mathcal O}) \to p(x_i),$$8, 400 nodes, 300 edges, clustering coefficient $$I(x_i; x_{\mathcal O}) \to 0 \quad \Rightarrow \quad p(x_i \mid x_{\mathcal O}) \to p(x_i),$$9, no cycles, and no re-engagement (Qiang, 24 Dec 2025). A striking quantitative result is that when the simulated chain starts from a neutral source node, 83.9% (26/31) of first-round replies are labeled positive and only 16.1% (5/31) remain neutral, leading to class imbalance severe enough that the GCN classifier has neutral-class F1 = 0 (Qiang, 24 Dec 2025). This is a caution against equating plausible local text generation with faithful diffusion-process simulation.

Security-oriented work shows a different use of diffusion text generation. DiffusionAttacker formulates jailbreak rewriting as a conditional seq2seq diffusion problem, using a DiffuSeq-style model to rewrite harmful prompts while preserving their semantics (Wang et al., 2024). The full control loss is

$\langle eos \rangle$0

and differentiable sampling is enabled through Gumbel-Softmax (Wang et al., 2024). On Advbench and Harmbench, the method reports the highest ASR overall among the compared methods, with, for example, on Llama3 $\langle eos \rangle$1 and $\langle eos \rangle$2, together with the lowest Self-BLEU among the reported attacks (Wang et al., 2024). This indicates that diffusion-based rewriting can expand the attack surface relative to suffix-only or strictly autoregressive jailbreak generation.

Across these branches, several research directions recur. For diffusion LLMs proper, open problems include information-aware corruption, structured multi-token objectives, fine-tuning-based extrapolation methods, and better handling of aggregation and middle-context retrieval (Jin et al., 27 Dec 2025). For hybrid diffusion–LLM systems, recurrent themes include trajectory-aware conditioning, stronger interaction between the LLM and diffusion modules, and tighter alignment between semantic hierarchies and denoising dynamics (Li et al., 3 Feb 2026). For social and agentic simulation, the literature repeatedly points to missing memory, recurrence, re-engagement, and community formation as the main obstacles to realistic diffusion modeling with LLMs (Qiang, 24 Dec 2025).

The field therefore remains heterogeneous but coherent in one respect: whether diffusion is used as a denoising process, a latent density model, or a propagation metaphor, Diffusion-LLM research is organized around the idea that iterative refinement and semantic reasoning are complementary but not yet fully reconciled.