Diffusion-LMs: Principles and Applications

• Diffusion-LMs are generative models that iteratively denoise corrupted token sequences to reconstruct original text.
• They use bidirectional, non-causal architectures to enable parallel generation and improved global context integration.
• Diffusion-LMs support controllable text generation and efficient fine-tuning, achieving state-of-the-art results on various NLP tasks.

Diffusion LLMs (Diffusion-LMs) are a class of generative models for language that formulate text modeling as a denoising process, generalizing the discrete or continuous denoising-diffusion probabilistic model (DDPM) paradigm from vision to language. In this framework, a sequence of tokens is progressively corrupted by noise according to a forward process, and then iteratively reconstructed by a trained model in the reverse "denoising" process. Distinct from autoregressive LLMs (AR LMs) that generate text strictly left-to-right with causal masking, Diffusion-LMs typically exploit bidirectional architectures and non-causal conditioning, offering new inductive biases and unique architectural and empirical advantages for text generation, embeddings, parallel generation, and controllable sampling.

1. Mathematical Foundations and Algorithmic Structure

At the core of Diffusion-LMs is a Markovian noising (forward) process and a learned denoising (reverse) process. Let $x_0 = (x_0^1, \ldots, x_0^L)$ be a clean token sequence of length $L$. The basic discrete forward diffusion process masks each token independently with noise level $t \in [0,1]$:

$$q(x_t^i\mid x_0^i, t) = \begin{cases} \mathbf{M}, & \text{with probability } t, \ x_0^i, & \text{with probability } 1-t, \end{cases}$$

where $\mathbf{M}$ is a special mask symbol. The fully noised state is $(\mathbf{M}, \ldots, \mathbf{M})$ at $t=1$ (Zhang et al., 21 May 2025). In continuous-time or embedding-space variants, the forward kernel becomes Gaussian, e.g.

$$q(z_t|z_{t-1}) = \mathcal{N}(z_t; \sqrt{\alpha_t}z_{t-1}, \beta_t I)$$

with appropriate schedule $\alpha_t, \beta_t$ (Zhu et al., 2023, Jin et al., 27 Dec 2025).

The reverse denoising process is realized by a neural network that either (a) for discrete-token LMs, jointly predicts masked positions via

$p_\theta(x_0 \mid x_t)$

trained with cross-entropy over masked tokens,

$$\mathcal{L}_\mathrm{diff}(\theta) = - \mathbb{E}_{t,x_0,x_t} \left[ \frac{1}{t} \sum_{i=1}^L \mathbf{1}[x_t^i = \mathbf{M}] \log p_\theta(x_0^i \mid x_t) \right],$$

or (b) for continuous-space LMs, predicts the added noise via MSE or scores as in classical DDPMs (Zhu et al., 2023, Jin et al., 27 Dec 2025).

Inference is iterative: starting from the fully masked sequence, the model repeatedly imputes (one or more) tokens in each step, guided by per-position confidence or heuristic schedules (Zhang et al., 21 May 2025, Fu et al., 26 Nov 2025). Efficient block or joint-sampling enhancements have been proposed to address the product-of-marginals artifact (Bansal et al., 25 Sep 2025).

2. Architectural Paradigms and Inductive Biases

Diffusion-LMs are typically implemented using Transformer architectures with bidirectional (non-causal) self-attention, aligning the modeling topology with tasks requiring global context. During pretraining, bidirectional masks (as in BERT/T5) are employed; during embedding/fine-tuning, masking is removed to extract context-rich representations (Zhang et al., 21 May 2025). This bidirectionality stands in contrast to the strictly causal attention of autoregressive LMs (e.g., GPT, LLMs), which only attend to past tokens during generation.

This setting is intrinsically any-order: the model is trained to predict any arbitrary subset of masked tokens given the rest, enabling parallelism and flexibility not available to autoregressive models (Fu et al., 26 Nov 2025, Ni et al., 5 Nov 2025).

Hybrid architectures—combining blockwise causal structure with parallel in-block denoising—enable controllable interpolation between diffusion and autoregressive regimes, supporting arbitrary-length generation while balancing global and local context (e.g., block diffusion (Arriola et al., 12 Mar 2025), DiffuApriel-H (Singh et al., 19 Nov 2025)).

Recent work also demonstrates that bidirectional state-space models (e.g., Mamba backbones) can serve as competitive denoisers in place of traditional Transformers, significantly improving throughput for long contexts (Singh et al., 19 Nov 2025).

3. Training Objectives and Fine-Tuning Methodologies

Pretraining uses the masked diffusion denoising loss described above, sometimes interpreted as a variational ELBO or score-matching objective. This generalizes the masked language modeling (MLM) objective: when the noise kernel is absorbing to [MASK], the loss is equivalent to BERT-style MLM as a special case (Ye et al., 2023). Continuous-space or hybrid objectives also exist (Jin et al., 27 Dec 2025).

For text embedding and retrieval tasks, downstream fine-tuning employs contrastive learning: $$\mathcal{L}_\mathrm{ctr}(q, p^+, \{p_j^-\}) = -\log \frac{ \exp(s(\phi(q), \phi(p^+))) }{ \exp(s(\phi(q), \phi(p^+))) + \sum_j \exp(s(\phi(q), \phi(p_j^-))) }$$ where text embeddings are obtained via mean pooling over final-layer token activations from the unmasked, bidirectionally attended Diffusion-LM (Zhang et al., 21 May 2025).

Instruction fine-tuning, analogous to "Flan-tuning" in the AR paradigm, reuses the masked diffusion loss over instruction-response pairs, showing strong emergence of zero-shot and few-shot in-context abilities (Ye et al., 2023).

Guidance techniques for controllable generation, including classifier-free or topic-conditioned denoising, as well as adaptive ODE solvers (rectified flow, Runge-Kutta), allow for adaptive compute and target-aware decoding (Cetin et al., 27 Jan 2025).

4. Decoding, Scaling, and Computational Considerations

Unlike AR models, Diffusion-LMs enable inherently parallel decoding, generating or infilling multiple tokens simultaneously. However, exact joint sampling is typically computationally intractable. Standard parallel decoding heuristics (exploiting high-confidence tokens) can create a mismatch between the product-of-marginals and true joint distributions, reducing text fidelity (Bansal et al., 25 Sep 2025, Fu et al., 26 Nov 2025). Recent innovations include:

• Approximate joint samplers: Lightweight, trainable layers (e.g., "ADJUST") trained on top of a frozen diffusion LM allow near-joint sampling over multiple tokens per block, preserving sample quality while accelerating inference (Bansal et al., 25 Sep 2025).
• Block and beam-based exploration: Advanced "explore-then-exploit" (ETE) strategies maximize information throughput per decoding round, balancing high-confidence exploitation with strategic sampling of high-entropy locations (Fu et al., 26 Nov 2025).
• Efficient KV caching: Techniques such as FreeCache circumvent the computational bottleneck of Transformer attention by reusing cached key/value projections for unmasked tokens, reducing per-step complexity (Hu et al., 27 May 2025).
• Alternative architectures: Bidirectional state-space models (e.g., Mamba) enable linear-time denoising, vastly improving throughput over quadratic-complexity Transformer denoisers for long sequences (Singh et al., 19 Nov 2025).

Despite these advancements, high computational cost at inference remains a constraint relative to autoregressive models, especially for long contexts or large numbers of diffusion steps (Hu et al., 27 May 2025, Singh et al., 19 Nov 2025).

5. Empirical Performance and Task-Specific Outcomes

Diffusion-LMs demonstrate strong, sometimes state-of-the-art, performance, especially in global-context and logic-heavy scenarios:

• Text embedding: Bidirectional Diffusion-LMs (e.g., Dream-7B-DiffEmbed) outperform autoregressive LLM-based embeddings by 20% on long-document retrieval and 8% on reasoning-intensive retrieval tasks, while matching on instruction-following and general-purpose benchmarks (Zhang et al., 21 May 2025).
• Parallel proposal generation: Diffusion-LMs, as efficient "thought proposers," enable the generation of multiple diverse reasoning steps in parallel for complex tasks such as game-solving and travel planning, nearly doubling throughput compared to autoregressive baselines without loss in pass@1 accuracy (Shao et al., 31 Oct 2025).
• Scaling and few-shot learning: With scaling, diffusion models match or exceed AR baselines on translation, summarization, and instruction-following; instruction fine-tuning elicits strong in-context learning and task transfer (Ye et al., 2023).
• Data efficiency in low-resource settings: Diffusion-LMs consistently outperform AR models given limited unique data by leveraging iterative denoising and built-in data augmentation, with a crossover in efficiency observed as models are trained for many epochs (Ni et al., 5 Nov 2025).
• Speech and multimodal extension: The DIFFA system, built on a diffusion backbone, demonstrates that diffusion LMs deliver competitive performance for audio-language understanding relative to open-source AR baselines, with strong reasoning accuracy and efficient parameterization (Zhou et al., 24 Jul 2025).

6. Challenges, Theoretical Trade-offs, and Advances

Key limitations and open issues include:

• Discrete vs. continuous process design: Existing approaches exhibit trade-offs between discrete-masked and continuous-embedding diffusion; discrete models naturally yield tokens but struggle with smoothness and multi-token dependencies, while continuous models benefit from smoother dynamics but require projection back to symbols (Jin et al., 27 Dec 2025).
• Marginal trap and multi-token dependency: Parallel decoding and per-token training can collapse joint dependencies, resulting in incoherent samples absent costly sequential decoding or approximate joint sampling (Jin et al., 27 Dec 2025, Bansal et al., 25 Sep 2025).
• Information-theoretic bottlenecks: Decoding schedules that merely exploit high-confidence tokens are bounded by bits-per-round limitations, with theoretical lower bounds scaling linearly with output information (Fu et al., 26 Nov 2025).
• Computational overhead: While scalable and parallel in theory, Diffusion-LMs typically require more computation per generated token versus AR models, though innovations in architecture and caching are closing the gap (Hu et al., 27 May 2025, Singh et al., 19 Nov 2025).

Recent directions address these challenges via hierarchical diffusion processes to enable richer intermediate structure (Zhou et al., 8 Oct 2025), contrastive-inspired denoising losses for efficient training (Zhu et al., 27 Oct 2025), soft-masked and information-aware noise schedules (Chen et al., 2023, Jin et al., 27 Dec 2025), and the development of task-adaptive and hybrid AR–diffusion models (Arriola et al., 12 Mar 2025).

7. Applications, Controls, and Future Prospects

Diffusion-LMs open unique dimensions of controllability and flexibility:

• Reference and controllable generation: Conditioning on specific attributes or prompts is straightforward, supporting plug-and-play guidance (e.g., for semantic style, length, syntactic structure) via classifier gradients or instruction prompts (Chen et al., 2023, Cetin et al., 27 Jan 2025, Ye et al., 2023).
• Watermarking: Diffusion-LMs can be robustly watermarked for traceability via context-expectation-based tilting, achieving >99% true positive rate with negligible quality impact—enabling order-agnostic and infill watermarking not feasible with AR approaches (Gloaguen et al., 29 Sep 2025).
• Collaboration and privacy: Diffusion-based models support efficient, privacy-preserving ensembling of small user-specialized models with large generalist models, facilitating customizability and data localization (Han et al., 2023).
• Diversity and semantic control: Novel sampling and perturbation strategies (e.g., Time-Annealed Perturbation Sampling, TAPS) leverage the temporal division of labor in the diffusion process to generate diverse, semantically branched outputs with minimal quality loss (Wu et al., 30 Jan 2026).

Ongoing research focuses on more structured corruption processes, scalable discrete diffusion, accelerated decoding, hybrid AR–diffusion architectures, multimodal extension, and improved theoretical understanding to further harmonize diffusion dynamics with language structure and downstream requirements (Jin et al., 27 Dec 2025, Zhou et al., 24 Jul 2025, Ye et al., 2023).

---

References:

• "Diffusion vs. Autoregressive LLMs: A Text Embedding Perspective" (Zhang et al., 21 May 2025)
• "From Bits to Rounds: Parallel Decoding with Exploration for Diffusion LLMs" (Fu et al., 26 Nov 2025)
• "Time-Annealed Perturbation Sampling: Diverse Generation for Diffusion LLMs" (Wu et al., 30 Jan 2026)
• "On the Role of Discreteness in Diffusion LLMs" (Jin et al., 27 Dec 2025)
• "Watermarking Diffusion LLMs" (Gloaguen et al., 29 Sep 2025)
• "LLMs to Diffusion Finetuning" (Cetin et al., 27 Jan 2025)
• "DIFFA: Large Language Diffusion Models Can Listen and Understand" (Zhou et al., 24 Jul 2025)
• "Accelerating Diffusion LLM Inference via Efficient KV Caching and Guided Diffusion" (Hu et al., 27 May 2025)
• "Diffusion Models in NLP: A Survey" (Zhu et al., 2023)
• "Diffuse Thinking: Exploring Diffusion LLMs as Efficient Thought Proposers for Reasoning" (Shao et al., 31 Oct 2025)
• "Block Diffusion: Interpolating Between Autoregressive and Diffusion LLMs" (Arriola et al., 12 Mar 2025)
• "A Survey of Diffusion Models in Natural Language Processing" (2305.14671)
• "Next Semantic Scale Prediction via Hierarchical Diffusion LLMs" (Zhou et al., 8 Oct 2025)
• "Enabling Approximate Joint Sampling in Diffusion LMs" (Bansal et al., 25 Sep 2025)
• "Breaking the Bottleneck with DiffuApriel: High-Throughput Diffusion LMs with Mamba Backbone" (Singh et al., 19 Nov 2025)
• "David helps Goliath: Inference-Time Collaboration Between Small Specialized and Large General Diffusion LMs" (Han et al., 2023)
• "Simple Denoising Diffusion LLMs" (Zhu et al., 27 Oct 2025)
• "A Cheaper and Better Diffusion LLM with Soft-Masked Noise" (Chen et al., 2023)
• "Diffusion LLMs are Super Data Learners" (Ni et al., 5 Nov 2025)
• "Diffusion LLMs Can Perform Many Tasks with Scaling and Instruction-Finetuning" (Ye et al., 2023)