Text Diffusion Language Models

• Text Diffusion Language Models are generative models that iteratively denoise corrupted text sequences rather than predict tokens sequentially.
• They employ both continuous methods with Gaussian noise on embeddings and discrete strategies using token masking or random replacement.
• These models enable parallel refinement across all sequence positions and extend to diverse modalities like molecule generation, sign language, and graph reasoning.

Searching arXiv for recent and foundational papers on text diffusion LLMs. A Text Diffusion LLM is a generative model that treats language modeling as an iterative denoising problem rather than a left-to-right next-token prediction problem. Across the literature, the term covers both continuous diffusion in embedding or latent spaces and discrete diffusion over token sequences. In continuous variants, a target sequence is mapped to continuous vectors and corrupted with Gaussian noise, then reconstructed through a learned reverse process; in discrete variants, tokens are progressively replaced by random vocabulary items or a special mask token and then denoised by a time-conditioned model. This family includes unconditional LLMs, sequence-to-sequence systems, controllable generation frameworks, and multimodal extensions in which text conditions other structured outputs such as sign-language pose sequences, molecules, or graph-aware prompts (Lin et al., 2022, Han et al., 2022, Kiruluta et al., 16 Mar 2025, Gong et al., 2024, Chen et al., 30 Jun 2026).

1. Definition and conceptual scope

Text diffusion LLMs define a sequence of latent states $x_0, x_1, \ldots, x_T$ or $z_0, z_1, \ldots, z_T$, where $x_0$ or $z_0$ corresponds to clean text and the terminal state is maximally corrupted. The forward process is fixed or parameterized, while the reverse process is learned. This contrasts with autoregressive LLMs, which factorize sequence probability as $p(x_{1:T}) = \prod_t p(x_t \mid x_{<t})$ and generate tokens one at a time (Bertolani et al., 17 Jun 2026, Kiruluta et al., 16 Mar 2025).

The literature distinguishes two major formulations. In continuous diffusion, the state is a continuous sequence $z_t \in \mathbb{R}^{n \times d}$, and the forward process uses Gaussian noise. In discrete diffusion, the state remains in token space $x_t \in \mathcal{V}^n$, and the forward process is a Markov chain over categorical states, often implemented through masking or random replacement (Jin et al., 27 Dec 2025). The survey literature also places these models in continuity with denoising autoencoders and masked LLMs: BERT-style denoising is effectively a single-step denoising process, whereas diffusion introduces a multi-step corruption schedule and an explicit generative interpretation (He et al., 2022, Zhu et al., 2023).

A recurrent claim across the field is that diffusion-based generation updates all positions in parallel at each step. This makes diffusion models non-autoregressive at the position level, although they remain iterative in diffusion time. Several papers describe this as enabling parallel refinement of an entire sequence rather than irreversible left-to-right commitment (Bertolani et al., 17 Jun 2026, Kiruluta et al., 16 Mar 2025, Gong et al., 2024).

2. Continuous and discrete formulations

Continuous text diffusion typically operates on embeddings or latent representations. In a standard form, the forward process is Gaussian: $$q(z_t \mid z_0) = \mathcal{N}\left(z_t; \alpha_t z_0,\ \sigma_t^2 I\right),$$ or equivalently through the DDPM-style reparameterization

$$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon.$$

This formulation underlies sequence-to-sequence diffusion systems such as GENIE, which diffuses target embeddings and reconstructs them with a conditional diffusion decoder (Lin et al., 2022), as well as text-guided molecule generation, where Gaussian diffusion is applied to the embedding matrix of a SMILES sequence (Gong et al., 2024). It also appears in latent diffusion work that seeks to reduce the likelihood gap with autoregressive models by learning a multivariate forward process in embedding space (Midavaine et al., 7 Jan 2026).

Discrete text diffusion instead keeps the state in token space. One common variant is absorbing diffusion, where tokens are replaced by a special mask token with a time-dependent probability $\beta(t)$ or $z_0, z_1, \ldots, z_T$0 (Bertolani et al., 17 Jun 2026, Chen et al., 30 Jun 2026). Another variant is uniform random replacement, where each token is replaced by a uniformly sampled vocabulary token with probability $z_0, z_1, \ldots, z_T$1: $z_0, z_1, \ldots, z_T$2 This formulation is central to the State–Fourier Diffusion LLM, which uses random replacement rather than masking (Kiruluta et al., 16 Mar 2025).

The literature also identifies a structural distinction between the two paradigms. Continuous models satisfy the diffusion-side properties of smooth corruption and tractable marginals, but they do not natively satisfy discreteness because the final mapping back to tokens is discontinuous. Discrete models satisfy token discreteness directly, but their corruption is stepwise rather than smooth (Jin et al., 27 Dec 2025). This suggests a persistent trade-off rather than a single canonical formulation.

3. Architectural realizations

Transformer denoisers remain common, but not universal. In masked diffusion LLMs such as TAG-DLM, the backbone is a bidirectional Transformer that predicts clean tokens at masked positions under a continuous-time masking schedule (Chen et al., 30 Jun 2026). DiffusionBERT likewise uses BERT as the reverse model for a discrete diffusion process with an absorbing state, explicitly exploiting the compatibility between masked language modeling and denoising diffusion objectives (He et al., 2022).

Sequence-to-sequence diffusion systems typically use encoder–decoder structures. GENIE consists of a bidirectional encoder over the source text and a diffusion-based decoder over continuous target representations (Lin et al., 2022). Text-guided molecule generation uses a frozen SciBERT encoder for the textual condition and a 12-layer Transformer denoiser operating on noised SMILES embeddings (Gong et al., 2024). In both cases, text guidance enters through cross-attention over encoded source representations.

Other work replaces attention-heavy backbones. The State–Fourier Diffusion LLM uses a U-Net-style denoiser built from state-space modules and a Complex Fourier MLP, with no transformers and no large convolution modules. Its stated complexity per denoising step is about $z_0, z_1, \ldots, z_T$3, rather than $z_0, z_1, \ldots, z_T$4 for attention (Kiruluta et al., 16 Mar 2025). This suggests an architecture-level response to the cost of repeated full-sequence denoising.

There are also semi-autoregressive and blockwise designs. SSD-LM generates blocks of tokens with diffusion while moving left-to-right at the block level. It performs diffusion directly in a simplex-based vocabulary representation and uses a bidirectional Transformer encoder to denoise each block (Han et al., 2022). More recent experimental analyses group such systems under block / hybrid diffusion models, alongside full-sequence diffusion and sliding-window diffusion, and report that block size and unmasking ratio materially affect the quality–compute trade-off (Bertolani et al., 17 Jun 2026).

4. Training objectives and conditioning strategies

Training objectives vary with the diffusion domain. In continuous models, the dominant objectives remain MSE-style denoising losses or equivalent ELBO-derived forms. GENIE follows the standard DDPM-style logic over target embeddings and combines denoising with a token-level reconstruction term $z_0, z_1, \ldots, z_T$5 (Lin et al., 2022). TGM-DLM uses $z_0, z_1, \ldots, z_T$6-prediction rather than noise prediction and combines a denoising loss with a rounding likelihood over tokens (Gong et al., 2024).

In discrete models, token-wise cross-entropy is standard. SFDLM trains with a denoising cross-entropy objective that predicts a less-noised sequence $z_0, z_1, \ldots, z_T$7 from $z_0, z_1, \ldots, z_T$8 (Kiruluta et al., 16 Mar 2025). MDLM-style models, including TAG-DLM, use a Rao–Blackwellized masked cross-entropy over masked positions: $z_0, z_1, \ldots, z_T$9 This ties diffusion training closely to masked language modeling while retaining a generative decoding procedure (Chen et al., 30 Jun 2026).

Some work modifies the standard objective to better fit language structure. DiffusionBERT introduces a Spindle noise schedule that controls corruption based on token information and reports improvements in both perplexity and BLEU over simpler schedules (He et al., 2022). The reparameterized discrete diffusion model of Zheng et al. reduces the training objective to a reweighted cross-entropy on corrupted tokens by explicitly introducing routing variables into the backward process (Zheng et al., 2023). Energy-Based Diffusion LLMs add a full-sequence energy term to correct the factorized denoising approximation used by standard discrete diffusion models (Xu et al., 2024).

Conditioning mechanisms are equally diverse. Cross-attention is standard for text-to-text and multimodal settings (Lin et al., 2022, Gong et al., 2024). Classifier-free guidance is explicitly used in DiffuSeq-style conditional generation, as summarized in the survey (Zhu et al., 2023). SSD-LM instead emphasizes modular control: because it diffuses directly in the vocabulary simplex, off-the-shelf classifiers can be used for gradient-based guidance without retraining the classifier (Han et al., 2022).

5. Inference, decoding, and compute trade-offs

Inference begins from a maximally corrupted state: pure Gaussian noise in continuous models, or all masks / nearly random tokens in discrete models. Reverse denoising then iteratively reconstructs the sequence. In continuous models such as GENIE and TGM-DLM, the reverse chain is implemented through DDPM- or DDIM-like updates from $x_0$0 (Lin et al., 2022, He et al., 20 Mar 2025). In discrete masking models such as TAG-DLM, inference starts from a fully or partially masked sequence and repeatedly predicts masked tokens using bidirectional context (Chen et al., 30 Jun 2026).

A major empirical theme is the trade-off between parallelism and number of denoising steps. Diffusion models update all positions in parallel at each step, but require multiple full-sequence passes. The large-scale experimental comparison of eight diffusion LLMs across eight benchmarks reports that behavior is strongly influenced by generation-time choices such as denoising steps, context length, block size, and parallel unmasking strategies (Bertolani et al., 17 Jun 2026). That study also reports that pure full-sequence diffusion can be orders of magnitude more expensive than autoregressive decoding, whereas block and hybrid systems narrow the gap substantially.

Several papers propose explicit acceleration strategies. TGM-DLM uses $x_0$1 training steps but only $x_0$2 DDIM-like refinement steps at inference (He et al., 20 Mar 2025). TGM-DLM for molecules uses uniform skipping with 200 reverse steps in phase 1 and 20 in phase 2 (Gong et al., 2024). SSD-LM varies $x_0$3 independently of $x_0$4 and uses a blockwise schedule to allow variable-length generation (Han et al., 2022). EDLM uses self-normalized importance sampling and reports a $x_0$5 sampling speedup over existing diffusion models without generation performance drop (Xu et al., 2024).

This suggests that “text diffusion LLM” refers not only to a training objective but also to a family of decoding regimes, ranging from full-sequence denoising to blockwise refinement and masked infilling.

6. Empirical performance, strengths, and open issues

The empirical record is heterogeneous. Some systems approach or exceed strong autoregressive baselines on selected tasks. SSD-LM reports MAUVE 81.76 and perplexity 17.99 on a 50-token OpenWebText continuation setting, with diversity advantages over comparable GPT-2 models (Han et al., 2022). GENIE reports competitive or superior results on XSum, CNN/DailyMail, Gigaword, and CommonGen relative to listed autoregressive and non-autoregressive baselines (Lin et al., 2022). DiffusionBERT improves over D3PM, Diffusion-LM, and earlier generative masked LLMs on LM1B in perplexity and BLEU (He et al., 2022).

More recent analyses emphasize that diffusion LLMs do not behave uniformly across tasks. The 2026 experimental study finds that no single architecture dominates: Dream performs strongly on structured constraint reasoning such as Sudoku, Fast-dLLM is especially strong on reasoning and coding, and translation remains comparatively fragile under many diffusion settings (Bertolani et al., 17 Jun 2026). The same study reports that aggressive parallel unmasking harms reasoning and code generation, while more denoising steps at fixed length often improve performance.

Another line of work focuses on limitations that arise from the structure of language rather than raw scaling. “On the Role of Discreteness in Diffusion LLMs” argues that current approaches satisfy only part of five essential properties and identifies two central issues: uniform corruption does not respect how information is distributed across positions, and token-wise marginal training cannot capture multi-token dependencies during parallel decoding (Jin et al., 27 Dec 2025). The paper’s “Marginal Trap” analysis shows that correct per-position marginals can still yield globally invalid or unattested sequences under independent parallel sampling. This suggests that the remaining gap between diffusion and autoregressive language modeling is partly structural.

Energy-based corrections and learned forward processes respond directly to these concerns. EDLM introduces a full-sequence residual energy model to correct factorized token predictions (Xu et al., 2024). Latent NFDM-style work argues that learning a multivariate forward process tailored to language narrows the likelihood gap with autoregressive models of the same size while preserving diffusion’s parallel-generation advantages (Midavaine et al., 7 Jan 2026). A plausible implication is that future text diffusion LLMs will increasingly combine structured forward processes, sequence-level scoring, and hybrid decoding schedules rather than relying on a single masking or Gaussian template.

7. Extensions beyond plain text generation

The concept generalizes readily beyond natural-language output tokens. TDM for sign language production is a text-conditioned diffusion-based sequence generator in which the output space is a 3D pose trajectory rather than text tokens (He et al., 20 Mar 2025). TGM-DLM generates SMILES strings from textual descriptions through diffusion in embedding space and adds a second correction phase for invalid strings (Gong et al., 2024). TAG-DLM treats graph neighborhoods linearized as text sequences and performs graph reasoning through a masked diffusion LLM with a topology attention mask (Chen et al., 30 Jun 2026). TextDiffuser-2 uses a LLM for layout planning and a diffusion model for text rendering, effectively turning layout into a structured textual conditioning language (Chen et al., 2023).

These systems are not all “LLMs” in the narrow autoregressive sense, but they exemplify the broader abstraction explicitly stated in several papers: a text-conditioned diffusion backbone can model structured sequences in multiple modalities, with the output domain determining the loss and decoder (He et al., 20 Mar 2025, Gong et al., 2024). This suggests that “Text Diffusion LLM” names a generative principle—iterative denoising under textual conditioning—rather than a single output type.

In this broader sense, the field encompasses discrete masked diffusion models, continuous embedding diffusers, simplex-based vocabulary diffusion, energy-corrected denoisers, and multimodal sequence generators. What unifies them is the replacement of next-token prediction with a learned reverse process over noisy sequences, whether those sequences are words, embeddings, poses, molecular strings, or graph-aware prompts (Zhu et al., 2023, Bertolani et al., 17 Jun 2026).