Discrete Diffusion Models Explained

• Discrete diffusion models are generative systems that corrupt discrete data via a forward process and reconstruct it using a reverse denoising mechanism.
• They leverage techniques such as continuous-time Markov chains, discrete scoring functions, and reparameterized kernels to optimize sampling and convergence.
• DDMs are applied in various domains including text, layout, speech, and 3D mesh generation, with advances in acceleration, privacy, and theoretical guarantees.

Discrete diffusion models (DDMs) are generative models for intrinsically discrete data—such as text tokens, visual tokens, categorical attributes, quantized geometry, or discrete latent codes—that construct a forward corruption process on a discrete state space and learn a reverse denoising process that maps noise back to data. In the contemporary literature, this family includes discrete-time Markov-chain formulations with masking or token replacement, continuous-time Markov chain (CTMC) formulations on spaces such as $[S]^d$, $\mathbb{Z}_m^d$, $[0,m]^d$, and $\mathbb{N}^d$, and task-specific instantiations for multimodal generation and structured outputs (Ma et al., 3 Oct 2025, Zhang et al., 2024, Gao et al., 15 Dec 2025). The central technical objects are the forward transition kernel, the reverse conditional or reverse rate matrix, and a learned denoiser or discrete score function that approximates probability ratios governing reverse-time dynamics.

1. Process formulation and state-space design

A defining feature of DDMs is that the data domain is discrete from the outset. In a masked-token formulation for token sequences, the forward process may be written as

$$x_t \sim q(x_t|x_0,t)=\mathrm{Cat}(x_t;\alpha_t x_0 + (1-\alpha_t)\mathbf{m}),$$

where $\alpha_t$ is a continuous, monotonically decreasing noise scheduler and $\mathbf{m}$ is an absorbing state, typically a mask token (Ma et al., 3 Oct 2025). In discrete-time formulations for text-to-text generation, corruption is represented by a transition matrix $\mathbf{Q}_t$ through

$$q(x_t|x_{t-1}) = v^\top(x_t)\mathbf{Q}_t v(x_{t-1}),$$

with $v(x_t)$ the one-hot encoding of the token state (Zhou et al., 2023).

A more general formulation models the forward process as a CTMC. In that setting, for a finite discrete state space $\mathbb{Z}_m^d$0,

$\mathbb{Z}_m^d$1

where $\mathbb{Z}_m^d$2 is a time-dependent transition rate matrix; a common parameterization is $\mathbb{Z}_m^d$3 (Gao et al., 15 Dec 2025). The forward process on $\mathbb{Z}_m^d$4 is often assumed to factorize across coordinates for tractability, while more specialized analyses study random walks on $\mathbb{Z}_m^d$5, masked diffusion on $\mathbb{Z}_m^d$6, and biased random walks on $\mathbb{Z}_m^d$7 (Zhang et al., 2024, Conforti et al., 29 Nov 2025).

This variety of kernels is not incidental. The literature treats masking, random replacement, random walk, and absorbing-state dynamics as distinct modeling choices with distinct theoretical consequences. A common simplification is to equate DDMs only with mask-based LLMs, but the published formulations include arbitrary discrete state spaces, CTMC generators, and domain-specific noising operators (Zhang et al., 2024, Conforti et al., 29 Nov 2025).

2. Reverse-time dynamics, discrete scores, and learning objectives

The reverse process is the generative component of a DDM. In CTMC formulations, the time-reversed generator is determined by marginal probability ratios. For $\mathbb{Z}_m^d$8,

$\mathbb{Z}_m^d$9

with the diagonal term chosen so rows sum to zero (Gao et al., 15 Dec 2025). In the notation of score-based discrete diffusion, the discrete score function is the collection of ratios

$[0,m]^d$0

and a neural score estimator $[0,m]^d$1 approximates this object at the discrete time points used by the sampler (Zhang et al., 2024).

Several objective families appear in the literature. One is score-entropy training: $[0,m]^d$2 where $[0,m]^d$3 is a generalized I-divergence (Zhang et al., 2024). Another is denoising cross-entropy in masked processes, where the model predicts original tokens from partially masked inputs. In Diffusion-NAT, the non-autoregressive denoising objective is

$[0,m]^d$4

with BART used as the denoiser and with the denoising step identified with masked token recovery (Zhou et al., 2023).

A third line reparameterizes the reverse transition itself. Reparameterized discrete diffusion models (RDMs) make the mixture structure of the reverse process explicit and introduce token-level routing variables. Under this construction, the training objective reduces to a weighted cross-entropy over noisy tokens only: $[0,m]^d$5 This formulation supports adaptive decoding strategies in which only high-confidence tokens are denoised at a given step (Zheng et al., 2023).

The information-theoretic analysis of discrete diffusion later sharpened the interpretation of these losses. The I-MDSE and I-MDCE relations show that minimum denoising score entropy and minimum denoising cross-entropy are linked to mutual information decay, and yield time-integral decompositions of log-likelihood. In that framework, DSE and DCE are described not merely as variational bounds but as tight and principled estimators of log-likelihood (Jeon et al., 28 Oct 2025).

3. Theoretical guarantees and analytical structure

The theory of DDMs has expanded from basic reverse-time constructions to convergence rates, exact simulation tools, sample complexity, and statistical-mechanics analyses. A major line of work models DDMs as CTMCs and analyzes the pathwise discrepancy between the true reverse process and an approximate reverse sampler.

Uniformization provides one exact implementation route. For CTMC-based discrete diffusion on a hypercube, uniformization constructs the trajectory from a Poisson process with rate $[0,m]^d$6 and a time-dependent transition kernel, and yields exact simulation of the reverse process up to score estimation error (Chen et al., 2024). In that analysis, the total number of transitions is

$[0,m]^d$7

and the resulting KL and TV guarantees align with state-of-the-art achievements for diffusion models in $[0,m]^d$8 (Chen et al., 2024).

A separate discrete-time analysis on $[0,m]^d$9 derives convergence bounds for both KL divergence and total variation distance, with nearly linear dependence on the dimension $\mathbb{N}^d$0. The theory explicitly distinguishes initialization error, discretization error, and score estimation error, and shows that early stopping is required when data has zero-probability entries, whereas full-support data permits tighter bounds without early stopping (Zhang et al., 2024). This is one of the points where DDM theory departs sharply from informal accounts that treat the reverse chain as unproblematic near $\mathbb{N}^d$1.

The stochastic-integral treatment of DDMs pushes the analogy with continuous diffusion further. By formulating discrete diffusion with Lévy-type stochastic integrals and a generalized Poisson random measure with time-independent and state-dependent intensity, that framework derives change-of-measure theorems analogous to Itô-integral Girsanov results and obtains the first KL-divergence error bound for the $\mathbb{N}^d$2-leaping scheme: $\mathbb{N}^d$3 (Ren et al., 2024).

Non-asymptotic convergence results were subsequently extended to masked and random-walk dynamics on both finite and countably infinite spaces. In that setting, the backward process is characterized by a discrete score whose monotonicity controls the error, and the resulting complexity scales linearly up to logarithmic factors rather than exponentially with the dimension. The same work states that this is the first non-asymptotic convergence guarantee that does not rely on boundedness of the estimated score, and that it covers uniform noising processes on $\mathbb{N}^d$4 and on $\mathbb{N}^d$5, as well as masking-based noising dynamics (Conforti et al., 29 Nov 2025).

A distinct learning-theoretic advance gives the first sample complexity bound of $\mathbb{N}^d$6 for discrete-state diffusion. The analysis decomposes score estimation error into approximation, statistical, optimization, and clipping components, and shows that, for sufficiently wide networks, the approximation error can vanish in the discrete setting (Srikanth et al., 12 Oct 2025). This suggests that discrete diffusion is theoretically tractable in a sense that differs from the continuous case, where approximation error remains a more persistent obstacle.

4. Sampling, acceleration, and policy optimization

Sampling efficiency is a central systems concern for DDMs because naive reverse-time simulation may require many function evaluations. Parallel samplers such as $\mathbb{N}^d$7-leaping reduce wall-clock cost but introduce an error specific to the discrete setting. “Jump Your Steps” formalizes this as Compounding Decoding Error (CDE), defined for one parallel step as the KL divergence between the true joint transition and the product of its marginals: $\mathbb{N}^d$8 The cumulative CDE upper-bounds the divergence between the true distribution and the output of the parallel sampler, and JYS derives a tractable KL-divergence upper bound (KLUB) to optimize the sampling schedule without extra computational cost (Park et al., 2024). The significance is methodological: fast discrete sampling is not treated as a mere engineering choice but as a controlled approximation problem.

Acceleration has also been pursued by distillation. A conditional-distribution-matching approach observes that the reverse conditional of clean data given a noisy state,

$\mathbb{N}^d$9

admits a Markov decomposition through intermediate times and can be recovered from marginal density ratios and the known forward CTMC kernel (Gao et al., 15 Dec 2025). Distillation then matches teacher and student reverse conditionals directly, rather than relying on proxy objectives or auxiliary networks. The proposal is presented as applicable to both one-step and few-step samplers.

Beyond passive sampling, DDMs have also been coupled to reinforcement learning. MaskGRPO addresses the problem that, in non-autoregressive discrete diffusion, importance sampling is intractable and rollouts are complex. The method introduces modality-specific importance estimators, AR-like masking for language, randomized high-rate masking for vision, and a “token emerge” sampler for visual sequences (Ma et al., 3 Oct 2025). On GSM8K, MATH500, and MBPP it yields absolute improvements of 5%–7% in pass@1 accuracy over pre-trained DDMs, while on GenEval the overall score for MMaDA rises from 0.56 to 0.80 under MaskGRPO (Ma et al., 3 Oct 2025). These results position DDMs not only as generative models but also as policies subject to reward optimization.

5. Architectures and domain-specific instantiations

The practical scope of DDMs is unusually broad because the discrete state can be a native token space or a learned quantized latent space. In text generation, RDMs introduce explicit token-level routing and adaptive decoding, while Diffusion-NAT revises the decoding process of BART so that BART denoising and DDM denoising become the same non-autoregressive masked token recovery task (Zheng et al., 2023, Zhou et al., 2023). In the latter system, iterative self-prompting further refines outputs, and experiments on 7 datasets show that the approach can outperform competitive non-autoregressive methods and even surpass autoregressive methods (Zhou et al., 2023).

In layout generation, a DDM operates on quantized attributes such as category, position, and size. The Layout-Corrector work identifies a “layout sticking” phenomenon: once inharmonious tokens are generated, standard DDMs struggle to revise them because transitions between regular tokens are extremely rare or nearly zero (Iwai et al., 2024). The proposed module scores token correctness and reinitializes low-scored tokens to the ungenerated state, after which the DDM regenerates harmonized tokens. The paper reports that this module consistently boosts layout-generation performance with various state-of-the-art DDMs and significantly mitigates the performance drop associated with fast sampling (Iwai et al., 2024).

Speech reconstruction provides a different use case. In text-aligned speech tokenization and reconstruction, a DDM replaces the auto-regressive speech decoder with a masked discrete diffusion decoder: $$x_t \sim q(x_t|x_0,t)=\mathrm{Cat}(x_t;\alpha_t x_0 + (1-\alpha_t)\mathbf{m}),$$0 The resulting system generates speech in just 10 denoising steps, supports single-step generation with only minor quality degradation, and reconstructs utterances 3.3× faster than the auto-regressive baseline. The same study reports that FSQ yields up to a 35% relative WER reduction and +0.14 UT-MOS improvement over RVQ for AR models, while also enhancing DDM performance (Ku et al., 24 Sep 2025).

For sign language production, G2P-DDM converts continuous pose generation into discrete latent generation by introducing Pose-VQVAE and then applying a discrete denoising diffusion model over variable-length discrete sequences. The architecture includes CodeUnet to leverage spatial-temporal information and a heuristic sequential clustering method to predict variable lengths. On RWTH-PHOENIX-WEATHER-2014T, the reported G2P-DDM result is WER 77.26% and BLEU-4 7.50, compared with WER 82.01% and BLEU-4 6.66 for the cited prior best system (Xie et al., 2022).

DDMs have also been extended to high-fidelity 3D meshes. TSSR uses discrete diffusion for parallel mesh token generation, decouples topology sculpting from shape refinement, adds multi-level RoPE within an improved hourglass architecture, and introduces a connection loss to enforce topological consistency. The reported system generates meshes with up to 10,000 faces at a spatial resolution of $$x_t \sim q(x_t|x_0,t)=\mathrm{Cat}(x_t;\alpha_t x_0 + (1-\alpha_t)\mathbf{m}),$$1 (Song et al., 24 Oct 2025). A plausible implication is that DDMs are especially attractive in domains where parallel token prediction and iterative correction matter more than strict causal ordering.

6. Failure modes, interpretive debates, and boundary conditions

Several recurrent issues qualify the otherwise broad applicability of DDMs. First, fast parallel sampling is not neutral: $$x_t \sim q(x_t|x_0,t)=\mathrm{Cat}(x_t;\alpha_t x_0 + (1-\alpha_t)\mathbf{m}),$$2-leaping and related approximations induce compounding decoding error, so step allocation matters (Park et al., 2024). Second, standard absorbing or mask-heavy kernels can cause “sticking” behavior, in which generated tokens become difficult to revise once they leave the masked state (Iwai et al., 2024). Third, convergence analyses frequently require early stopping when the data distribution lacks full support, because the discrete score can become unbounded as $$x_t \sim q(x_t|x_0,t)=\mathrm{Cat}(x_t;\alpha_t x_0 + (1-\alpha_t)\mathbf{m}),$$3 (Zhang et al., 2024).

A more conceptual debate concerns how discrete and continuous diffusion should be related. One recent framework argues that prior continuous approaches to discrete data miss the guidance supplied by discrete decision boundaries and therefore proposes a two-step forward process that first estimates the boundary as a prior distribution and then rescales the trajectory to build a boundary conditional diffusion model (Gu et al., 2024). This is not a DDM in the strict CTMC sense, but it highlights a persistent issue in the field: whether discrete generative modeling is best served by intrinsically discrete dynamics or by continuous dynamics with stronger discrete conditioning.

Another boundary condition is privacy. The first theoretical study of inherent privacy for DDMs analyzes per-instance differential privacy and shows that training with $$x_t \sim q(x_t|x_0,t)=\mathrm{Cat}(x_t;\alpha_t x_0 + (1-\alpha_t)\mathbf{m}),$$4-sized data points leads to a surge in privacy leakage from

$$x_t \sim q(x_t|x_0,t)=\mathrm{Cat}(x_t;\alpha_t x_0 + (1-\alpha_t)\mathbf{m}),$$5

to

$$x_t \sim q(x_t|x_0,t)=\mathrm{Cat}(x_t;\alpha_t x_0 + (1-\alpha_t)\mathbf{m}),$$6

during the transition from pure noise to synthetic clean data, while a faster decay in diffusion coefficients amplifies the privacy guarantee (Wei et al., 2023). This directly contradicts the common assumption that synthetic generation by itself provides strong privacy protection.

Finally, statistical-mechanics analysis has introduced a dynamical-regime perspective on DDMs. For DDMs trained on two-class Ising variable data, the backward dynamics exhibit a random regime, a speciation regime, and a collapse regime. The speciation transition is identified through a second-order phase transition analysis, and the speciation time is given analytically by

$$x_t \sim q(x_t|x_0,t)=\mathrm{Cat}(x_t;\alpha_t x_0 + (1-\alpha_t)\mathbf{m}),$$7

where $$x_t \sim q(x_t|x_0,t)=\mathrm{Cat}(x_t;\alpha_t x_0 + (1-\alpha_t)\mathbf{m}),$$8 is the largest eigenvalue of the covariance matrix of the data distribution (Takahashi et al., 13 Apr 2026). The collapse transition is described by a condensation transition of Random Energy Model type. This suggests that DDM sampling dynamics may be interpretable not only algorithmically but also through the language of phase transitions.

Taken together, these limitations and refinements show that DDMs are neither a single canonical architecture nor a solved theory problem. They form a rapidly developing class of discrete generative models in which kernel design, score estimation, sampler approximation, correction mechanisms, and privacy or generalization criteria all materially affect the final behavior of the model.