Diffusion-LM: Denoising Language Model

• Diffusion-LM is a language modeling framework that uses iterative denoising on noised text to enable parallel infilling and bidirectional sequence representation.
• It leverages both continuous embedding diffusion and discrete token masking to adapt image-based diffusion processes for text generation, controllable sampling, and embedding.
• The approach offers enhanced controllable generation, improved text embedding, and efficient code synthesis while addressing challenges in global coherence and computational efficiency.

Diffusion-LM is a family of language modeling frameworks that employ denoising diffusion processes—originally developed for continuous data domains such as vision and audio—for discrete or continuous text generation, parallel infilling, controllable sampling, and bidirectional sequence representation. Unlike traditional autoregressive LLMs (AR LMs), which generate text sequentially from left to right, Diffusion-LMs apply iterative refinement, beginning from noised or masked text and progressively denoising toward fluent output. The approaches encompass both continuous (embedding-space) and discrete (token/mask) formulations, offer unique trade-offs and challenges, and have demonstrated advances in controllable generation, text embedding, test-time compute scaling, and efficient code synthesis.

1. Mathematical Foundations of Diffusion-LM

Diffusion-LMs are built on Markovian noising processes and learned denoising models that invert these processes to reconstruct structured text from corrupted inputs. Two main variants are prominent:

Continuous Embedding Diffusion:

Given a sentence $w = [w_1, ..., w_n]$, map each token to $\mathbb{R}^d$ embeddings, forming $x_0$. The forward (noising) process applies $\mathcal{N}$-distributed noise iteratively: $$q(x_t|x_{t-1}) = \mathcal{N}(x_t ; \sqrt{1-\beta_t} x_{t-1}, \beta_t I)$$ yielding $$x_t = \sqrt{\alpha_t} x_0 + \sqrt{1-\alpha_t} \epsilon$$, where $\alpha_t = \prod_{s=1}^t (1-\beta_s)$ and $\epsilon\sim\mathcal{N}(0, I)$ (Li et al., 2022, Xu, 2022, Cetin et al., 27 Jan 2025).

The learned denoising network parameterizes the reverse process: $$p_\theta(x_{t-1} | x_t) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t), \Sigma_\theta(t))$$ Training utilizes a combination of denoising loss (typically MSE) and—where needed—discrete rounding losses: $$L = \sum_t \mathbb{E}_{q(x_t|x_0)} || \mu_\theta(x_t, t) - \widehat{\mu}(x_t, x_0) ||^2 + \lambda\cdot\text{rounding loss}$$

Discrete Token/Masked Diffusion:

Let $\mathbb{R}^d$0. At each step, tokens are independently replaced with $\mathbb{R}^d$1: $\mathbb{R}^d$2 The reverse network, typically a bidirectional Transformer, predicts for each masked position: $\mathbb{R}^d$3 where $\mathbb{R}^d$4 is a learned logit vector for vocabulary tokens (Zhang et al., 21 May 2025, Chen et al., 27 Sep 2025, Chen et al., 2024).

The loss is a cross-entropy over masked positions, often weighted by $\mathbb{R}^d$5 to emphasize early decoding steps: $\mathbb{R}^d$6

Hybrid and Alternative Channels:

Certain models introduce information-aware schedules, progressive masking, or classify tokens into semantically meaningful groups before masking, aiming for a more text-aligned diffusion (Jin et al., 27 Dec 2025, Chen et al., 2024).

2. Key Properties and Theoretical Considerations

Diffusion-LMs have been analyzed through five desiderata (Jin et al., 27 Dec 2025):

• D1. Smooth Corruption: A continuous-time, infinitesimal noise process (well-satisfied by Gaussian embedding diffusion, violated by discrete-masking).
• D2. Tractable Intermediates: The marginal $\mathbb{R}^d$7 must be analytically computable; holds in both continuous and discrete settings.
• D3. Iterative Reverse Generation: Decoding refines a single state over multiple steps.
• L1. Discreteness: Output space must be fundamentally discrete; discrete-masked DLMs satisfy, continuous ones do not.
• L2. Structural Dependency: Generation must respect inter-token syntactic/semantic dependencies; this is not enforced by default in conventional per-token DLM objectives, potentially leading to incoherence from marginal trapping.

Current diffusion LMs trade off between D1/D2/D3 and L1/L2. Continuous DLMs enable smooth mathematical diffusion but require post-hoc discretization; discrete DLMs respect token structure but use coarse, less information-theoretically optimal noise and train only on per-token marginals.

Two persistent issues observed:

• Uniform Corruption: Masking uniformly across tokens does not respect varying token importance, causing inconsistency in information decay and, in extreme, frequency collapse of predictions.
• Marginal Training: Independent per-token losses cannot enforce global coherence, sometimes producing unsupported sequences when sampling each masked token independently.

Proposed remedies include information-aware noise schedules, hybrid discrete-continuous channels, and structured/energy-based objectives that directly address multi-token dependencies (Jin et al., 27 Dec 2025, Chen et al., 2024).

3. Diffusion-LM Algorithms: Inference, Training, and Guidance

Diffusion-LM sampling and training involve forward corruption, denoising, loss design, and potentially guidance:

• Inference (Continuous):

Discretize $\mathbb{R}^d$8 into $\mathbb{R}^d$9 steps, initialize $x_0$0, and iteratively update

$x_0$1

where $x_0$2 is an embedding predicted via token sampling or rounding (Cetin et al., 27 Jan 2025).

• Adaptive Compute/Integration:

Use adaptive ODE solvers (e.g., 2nd-order Runge–Kutta) to determine per-token or per-sequence compute budgets, allocating more steps to difficult cases and fewer to easy ones (Cetin et al., 27 Jan 2025). This enables monotonic accuracy–compute scaling.

• Guidance:

Classifier-free guidance interleaves unconditional and label-conditional logits:

$x_0$3

Varying $x_0$4 accesses accuracy–diversity trade-offs, empirically important for math/coding tasks (Cetin et al., 27 Jan 2025).

• Confidence-Guided and Efficient Decoding:

Masked diffusion LMs support "locking" of tokens whose posterior distribution has stabilized (high local confidence, low stepwise KL change). Locked positions are skipped in subsequent layers and attention is recomputed only for remaining "active" slots (Oba et al., 6 Feb 2026, Chen et al., 27 Sep 2025). This reduces per-step FLOPs from $x_0$5 to $x_0$6 ($x_0$7=active positions), yielding 30–50% computational savings at minimal quality loss.

• Discrete Masking Schedules:

Discrete DLMs can bias masking to favor content-bearing tokens for longer, reflecting position-dependent information content (e.g., in sentiment data augmentation, strong label tokens survive longer; (Chen et al., 2024)).

4. Applications: Controllability, Data Augmentation, Embedding, and Code

Diffusion-LM enables several applications beyond conventional LM tasks:

Controllable Generation:

Diffusion-LM enables plug-and-play, gradient-based control over arbitrary features (syntax, length, semantic constraints) via differentiable objectives on latent variables during sampling—unlike AR LMs which require left-to-right constraints or retraining (Li et al., 2022). Achieves substantially higher success in complex control tasks (e.g., syntax F1: 86.0% vs 17.9% for FUDGE; length: 99.9% vs 46.9%).

Textual Data Augmentation:

In sentiment classification (DiffusionCLS, (Chen et al., 2024)), DLMs generate pseudo-samples by reconstructing label-related tokens with label-aware, importance-weighted masking. Downstream classifier training incorporates both a cross-entropy and a contrastive loss to counter noise. Gains of up to +3.7% F1 in domain-specific settings or +10.9% in few-shot accuracy are reported compared to non-diffusion augmentation baselines.

Text Embedding:

Bidirectional attention and simultaneous conditional prediction at all token positions enable Diffusion-LMs to excel at text/document embedding (Zhang et al., 21 May 2025). On long-document retrieval (LongEmbed, 4096 tokens), DiffEmbed (Dream-7B) outperforms Llama3-8B by +20 nDCG, and on reasoning retrieval by +8 points, attributed to bidirectional global context integration. Traditional AR LMs with adaptations (LLM2Vec, Echo) cannot match performance in long-context or globally logical queries.

Code Synthesis:

Diffusion-based coding LMs (e.g., CoDA, Dream-7B, LLaDA-8B; (Chen et al., 27 Sep 2025)) pair large-scale masked diffusion pretraining with code-centric mid-training and instruction tuning. CoDA-1.7B-Instruct, using confidence-guided locking, matches or outperforms 7B-parameter diffusion models on Humaneval and MBPP pass@1 and reduces latency by ∼40%. Efficient decoding and bidirectional infilling yield competitive or superior results compared to similarly sized AR models.

Multimodal Captioning:

CLIP-Diffusion-LM conditions text denoising on CLIP-encoded vision features, achieving BLEU-4=0.2470 on Flickr30k+8k with five parallel refinement steps—demonstrating the feasibility of multimodal few-step diffusion decoding, albeit with slight underperformance compared to SOTA AR models (Xu, 2022).

5. Limitations, Open Challenges, and Design Trade-offs

Despite their successes, Diffusion-LMs encounter critical limitations both theoretical and empirical:

• Information-aware Corruption:

Uniform masking or noise schedules do not respect variable token importance, leading to "frequency collapse" (masking high-impact tokens early degrades informativeness; (Jin et al., 27 Dec 2025, Chen et al., 2024)). Progressive, label- or attention-weighted schedules partially mitigate this.

• Marginal Trap/Global Joint Modeling:

Per-token cross-entropy training fails to capture global dependencies, risking incoherent outputs under parallel decoding (e.g., "I likes tennis" from per-token marginals; (Jin et al., 27 Dec 2025)). Block-wise or energy-based objectives, or hybrid (e.g., soft dynamic) state representations, are under development.

• Compute and Latency:

Naive diffusion decoding is slower than AR LMs due to $x_0$8 steps. However, adaptive solvers and confidence-based skipping (SureLock, CoDA) yield 30–50% FLOP savings, matching AR latencies at interactive step counts (Oba et al., 6 Feb 2026, Chen et al., 27 Sep 2025).

• Scaling and Embedding Challenges:

Continuous methods can outperform on generation control and some embedding tasks, but rounding errors (continuous-to-discrete mapping) remain a key weakness. Token-identity loss at small noise levels and uncertainty at the output layer limit fluency and NLL, compared to AR models (Li et al., 2022, Xu, 2022).

• Integration With Language Structure:

Discrete DLMs are better aligned with text’s symbolic properties but lack the smoothness of continuous-space diffusion, leading to rougher and at times more brittle convergence (Jin et al., 27 Dec 2025).

6. Representative Results and Practical Implementations

Major experimental findings and implementation details:

Model/Task	Key Result	Reference
Diffusion-LM (controllability)	Semantic control 81.2% vs FUDGE 69.9%; POS 90.0% vs 27.0%; syntax F1 86.0% vs 17.9%	(Li et al., 2022)
L2D (Llama 3.2 1B)	GSM8K 38.9, MATH 17.2, HumanEval@10 47.8, MBPP@10 51.8, avg 35.5	(Cetin et al., 27 Jan 2025)
SureLock	30–50% FLOP reduction, <1.08× PPL drift, 0.1 point change on MT-Bench	(Oba et al., 6 Feb 2026)
DiffEmbed (Dream-7B)	LongEmbed nDCG 62.2 (+20 vs Llama3-8B); TheoQ +8 points	(Zhang et al., 21 May 2025)
CoDA-1.7B-Instruct	HE pass@1 54.3, MBPP+ 63.2, Eval+ 55.4, ∼40% faster than Dream-7B	(Chen et al., 27 Sep 2025)
CLIP-Diffusion-LM	Flickr8k BLEU-4=0.1876, Flickr30k+8k=0.2470, 5-step parallel decoding	(Xu, 2022)
DiffusionCLS	SMP2020 Macro-F1 +2.11%, India-COVID-X +3.66%, SST2 5-shot +10.9%	(Chen et al., 2024)

Adaptive solvers, bidirectional attention, and label-aware masking are consistently reported as practical enhancements. Diffusion path adapters (LoRA-style) enable orthogonality with AR finetuning, preserving single-step generation during diffusion-based adaptation (Cetin et al., 27 Jan 2025). The majority of modern implementations utilize full-sequence, parallel bidirectional Transformers with progressive masking or continuous noise schedules.

7. Outlook and Future Directions

Continued research aims to resolve structural trade-offs and further align diffusion mechanics with language requirements. Design directions include:

• Information-adaptive or semantically-aware noise channels, gradual and position-sensitive masking, and multi-stage corruption-schedules.
• Hybrid continuous–discrete models, combining smooth denoising with discrete token identity preservation (Jin et al., 27 Dec 2025).
• Explicit modeling of global structure: energy-based or sequence-level losses, block-updates, and integration of chain-of-thought or reasoning heuristics (Cetin et al., 27 Jan 2025).
• Large-scale pretraining (multi-trillion tokens) and scalable parallel decoding for practical, competitive inference—especially for long-form generation and retrieval tasks (Zhang et al., 21 May 2025).
• Extension to multilingual, multimodal, and interactive editing tasks, with improved rounding and soft-commitment mechanisms to bridge continuous and discrete states (Xu, 2022, Li et al., 2022).
• Unification of AR and diffusion frameworks, leveraging the complementary strengths for both one-step (“closed-loop”) and iterative (“open-loop”) generation (Cetin et al., 27 Jan 2025).

The evolution of Diffusion-LM signals a paradigm shift toward parallel, controllable, and globally coherent language modeling, with converging evidence for their strengths in embedding, data augmentation, and constraint-based generation. Limitations centered on joint decoding, information preservation, and computational efficiency continue to guide ongoing research.