Discrete Variable Diffusion

• Discrete variable diffusion is a generative process that randomizes and reconstructs finite or countable data using categorical transition kernels and CTMC formulations.
• The approach unifies discrete-time and continuous-time diffusion by employing analytical tools such as closed-form marginals and score ratio estimations.
• Key model families like USD3, CoM-DAD, and VADD demonstrate practical applications in image, music, and text generation with improved efficiency and accuracy.

Discrete variable diffusion denotes a family of generative constructions for categorical, combinatorial, or other non-Euclidean discrete data in which a forward process progressively randomizes a discrete state and a learned reverse process reconstructs samples from a simple terminal distribution. In contrast to Gaussian diffusion in $\mathbb{R}^d$, the state space is finite or countable, the forward law is described by categorical transition kernels or continuous-time Markov chains (CTMCs), and the reverse dynamics are parameterized through discrete score ratios, conditional marginals, or continuous relaxations such as diffusion on the probability simplex (Sun et al., 2022, Zhao et al., 2024, Floto et al., 2023).

1. Discrete state spaces and forward corruption

A standard formalism starts from a finite alphabet $X=\{1,\dots,K\}$ or a product space such as $X^d$, $[S]^d$, or the hypercube $\{0,1\}^d$. In discrete time, one common forward kernel is

$$q(x_t \mid x_{t-1})=\operatorname{Cat}(x_t; Q_t^\top x_{t-1}),$$

with one-hot states and a transition matrix $Q_t$. For nominal categorical data, USD3 uses

$Q_t=\alpha_t I + (1-\alpha_t)\mathbf{1}\pi^\top,$

where $\pi$ is the stationary distribution and $\alpha_t$ is a scheduling parameter. Multi-step marginals preserve the same form through the product of transition matrices, which yields analytically tractable posteriors in the categorical setting (Zhao et al., 2024).

In the continuous-time limit, the forward process becomes a CTMC with rate matrix $X=\{1,\dots,K\}$0. General-state-space treatments write the marginal evolution as the master equation

$X=\{1,\dots,K\}$1

while for product spaces with i.i.d. per-coordinate noise the global generator is a Kronecker sum of coordinatewise generators. This is the discrete analogue of the Fokker–Planck description used for continuous diffusion (Pauline et al., 4 Dec 2025).

The theoretical literature studies several state spaces. The exact-uniformization analysis focuses on arbitrary distributions over the hypercube $X=\{1,\dots,K\}$2 (Chen et al., 2024). Discrete-time convergence theory considers the general categorical product space $X=\{1,\dots,K\}$3 (Zhang et al., 2024). Non-asymptotic analyses also treat $X=\{1,\dots,K\}$4 and the countably infinite lattice $X=\{1,\dots,K\}$5, under masked, random-walk, and biased-random-walk forward dynamics (Conforti et al., 29 Nov 2025).

2. Reverse dynamics and discrete score functions

The central difficulty is that the classical score $X=\{1,\dots,K\}$6 is not defined on discrete spaces. Score-based continuous-time discrete diffusion therefore replaces the continuous score by ratios of conditional marginals. For a coordinate $X=\{1,\dots,K\}$7, SDDM uses

$X=\{1,\dots,K\}$8

and learns the associated singleton conditionals with a cross-entropy objective over categorical marginals (Sun et al., 2022).

For CTMC formulations, the exact time-reversed process is again a CTMC. A standard reverse generator is

$X=\{1,\dots,K\}$9

which makes the reverse rates proportional to a forward jump intensity multiplied by a discrete score ratio. The same structure appears in general-state-space expositions, where the reverse rate matrix is written as

$X^d$0

This is the discrete counterpart of score-corrected reverse drifts in continuous diffusion (Chen et al., 2024, Pauline et al., 4 Dec 2025).

The reverse parameterization need not be unique. Some constructions predict reverse rates directly; others predict $X^d$1 and recover reverse transitions analytically from forward marginals; still others learn local score ratios. In SDDM, the continuous-time jump-process formulation admits analytical backward simulation rather than purely Euler-style approximation, and the learning rule is framed as categorical ratio matching rather than Euclidean score matching (Sun et al., 2022).

3. Model families and representational choices

Discrete variable diffusion now encompasses several distinct but partially overlapping model families.

Family	Core mechanism	Representative paper
Categorical transition-kernel diffusion	Markov kernels or CTMC rates on one-hot states	USD3 (Zhao et al., 2024)
Absorbing or masked diffusion	Tokens transition to a special $X^d$2 state	CoM-DAD (Xu et al., 7 Jan 2026)
Simplex diffusion	OU dynamics in latent space mapped by softmax to the simplex	"Diffusion on the Probability Simplex" (Floto et al., 2023)
One-hot Gaussian diffusion	Gaussian corruption of one-hot vectors with argmax-based denoising	ADD (Li et al., 1 Oct 2025)
Latent-augmented masked diffusion	Auxiliary latent variable $X^d$3 induces non-factorized denoising marginals	VADD (Xie et al., 23 May 2025)

USD3 emphasizes analytical simplification and unification. It derives closed-form forward and backward probabilities, simplifies the variational lower bound, and unifies discrete-time and continuous-time discrete diffusion under a common categorical formalism. The same framework allows arbitrary categorical noise distributions, including different noise distributions for different elements of a multi-element object, while keeping backward denoising exact and accelerated (Zhao et al., 2024).

A different bridge is diffusion on the probability simplex. There the forward process is an Ornstein–Uhlenbeck process in $X^d$4, mapped to the simplex by the additive logistic, or softmax, transformation. The resulting marginals are logistic-normal, transitions and scores are available in closed form, and the construction extends naturally to diffusion on the unit cube for bounded image generation (Floto et al., 2023).

ADD places the process directly in float-encoded one-hot space. Its forward corruption is Gaussian on one-hot labels, the reverse model predicts categorical probabilities with a timestep-conditioned cross-entropy loss, and inference alternates argmax discretization with re-noising. The same paper explicitly distinguishes this scheme from what it calls “pseudo” discrete diffusion based on masking policies or continuous latent diffusion (Li et al., 1 Oct 2025).

VADD augments masked diffusion with a latent variable $X^d$5. Conditional on $X^d$6, the denoising transition factorizes, but after marginalizing $X^d$7 the reverse process becomes non-factorized and can model correlations among dimensions more effectively, especially when the number of denoising steps is small (Xie et al., 23 May 2025).

CoM-DAD couples a continuous semantic manifold with a discrete absorbing diffusion process for token generation. Its discrete forward process masks tokens with a variable-rate schedule, while the reverse process is conditioned on a continuous semantic plan, thereby separating high-level semantic planning from low-level token synthesis (Xu et al., 7 Jan 2026).

4. Objectives, dependence structure, and decoding

The variational perspective remains foundational. General-state-space treatments derive a discrete-time ELBO whose diffusion term is a sum of conditional KL divergences, and whose continuous-time limit yields denoising score entropy objectives for reverse-rate learning (Pauline et al., 4 Dec 2025). USD3 sharpens this line by deriving analytic expressions for $X^d$8 and $X^d$9, then introducing a Taylor-expansion-based approximation that reduces the loss to a squared mismatch term plus a cross-entropy auxiliary term. A further extension permits training between arbitrary partial steps $[S]^d$0, not only adjacent steps (Zhao et al., 2024).

A major modeling issue is dependence across coordinates. Standard masked diffusion models often use a reverse distribution of the form

$[S]^d$1

which assumes conditional independence across dimensions during denoising. VADD argues that this becomes limiting when many dimensions are denoised in one step, because high-order inter-dimensional dependencies are then weakly represented. Its remedy is a latent-variable mixture model together with an auxiliary recognition network $[S]^d$2 and a variational objective based on a double evidence lower bound (Xie et al., 23 May 2025).

In language modeling, the same dependence problem appears as a decoding mismatch rather than a factorization mismatch. Discrete diffusion LLMs denoise masked token sequences with bidirectional attention, but practical generation often uses blockwise semi-autoregressive decoding. VSB identifies the mismatch between training with full-sequence context and inference with bounded blocks, defines “self-containedness” through the divergence between No-Future and Future-Aware predictive distributions, and selects variable-size blocks by this divergence rather than by fixed-size heuristics (Wang et al., 27 Apr 2026).

5. Sampling, exactness, and convergence theory

One of the strongest distinctions between discrete and continuous diffusion lies in simulation. For CTMC-based models on finite state spaces, uniformization can simulate the reverse process exactly by embedding it in a Poisson process with random jump times. On the hypercube, this removes discretization error entirely at the process-simulation level and yields TV and KL guarantees under score-learning assumptions. In the regime described in the hypercube analysis, if the score estimator error is $[S]^d$3, then after $[S]^d$4 simulation time and $[S]^d$5 steps, the KL error is $[S]^d$6 (Chen et al., 2024).

A broader stochastic-calculus framework for discrete diffusion recasts CTMC dynamics as Lévy-type stochastic integrals driven by a Poisson random measure with time-independent and state-dependent intensity. This framework derives a discrete analogue of Girsanov’s theorem, makes pathwise KL divergence explicit, and provides the first error bound for the $[S]^d$7-leaping scheme in KL divergence. The analysis decomposes the total error into truncation, score-approximation, and discretization components (Ren et al., 2024).

Discrete-time score-based analysis over $[S]^d$8 studies samplers that use score estimators only at predefined time points. The resulting KL and TV bounds are nearly linear in dimension $[S]^d$9, and the theory highlights early stopping as a necessary device when the data distribution lacks full support, because discrete scores can become singular as $\{0,1\}^d$0 (Zhang et al., 2024).

Non-asymptotic convergence theory further extends these guarantees to random-walk and masked forward dynamics on $\{0,1\}^d$1, and to biased random walks on $\{0,1\}^d$2. A distinctive claim of this line is that the analysis does not rely on bounded estimated scores; instead, the monotonicity of the discrete score function drives the argument, and the overall complexity scales linearly up to logarithmic factors with dimension (Conforti et al., 29 Nov 2025).

6. Dynamical regimes, uncertainty, and empirical domains

Statistical-mechanical analysis suggests that reverse-time dynamics in discrete diffusion exhibit macroscopic phase transitions analogous to those identified in continuous diffusion. For two-class Ising variable data, the reverse process shows a speciation transition, where trajectories begin to reflect class-level structure, and a collapse transition, where they commit to individual training samples. The former is analyzed as a second-order phase transition via high-temperature expansion, while the latter is characterized as a Random Energy Model condensation transition; both predictions are reported as consistent with numerical simulations and experiments on trained discrete diffusion models (Takahashi et al., 13 Apr 2026).

Empirically, discrete diffusion has been applied across several data types. SDDM evaluates synthetic and real-world music and image benchmarks using continuous-time jump processes (Sun et al., 2022). VADD reports improvements over masked diffusion baselines on 2D toy data, pixel-level image generation, and text generation, with especially large gains when the number of denoising steps is small (Xie et al., 23 May 2025). CoM-DAD targets unified text-image generation through a hierarchical continuous–discrete architecture (Xu et al., 7 Jan 2026).

Three-dimensional generation provides a particularly native use case. DVD models sparse voxel occupancy as a binary categorical variable rather than as a thresholded continuous latent. This avoids continuous-to-discrete thresholding, yields interpretable predictive probabilities for each voxel, and enables uncertainty estimation through predictive entropy. The same paper introduces block-structured perturbation fine-tuning so that the model can inpaint and edit voxels within a single sampling round, with negligible auxiliary computation and no additional model evaluations (Xiang et al., 8 May 2026).

The simplex formulation adds another application-oriented bridge. Because it performs diffusion directly on categorical probability vectors rather than on hard categories, it provides a continuous path on the simplex and extends naturally to the unit cube, where the paper notes applications to bounded image generation (Floto et al., 2023).

7. Conceptual tensions and open questions

Discrete variable diffusion is not a single settled architecture. General-state-space treatments place categorical kernels, masking or absorbing kernels, CTMC limits, and their ELBOs under one unified theory (Pauline et al., 4 Dec 2025). By contrast, ADD explicitly argues that methods based on masking policies or continuous latent diffusion are not fully “authentic” discrete diffusion, reserving that term for processes that preserve core diffusion characteristics directly in one-hot space (Li et al., 1 Oct 2025). The coexistence of these positions indicates that the field’s terminology remains unsettled.

Several technical issues recur across otherwise different frameworks. First, reverse scores can become unstable or singular near terminal denoising times, which is why both discrete-time CTMC theory and non-asymptotic convergence work introduce early stopping under limited-support assumptions (Zhang et al., 2024, Conforti et al., 29 Nov 2025). Second, few-step generation makes dependence modeling difficult: masked denoisers with factorized reverse transitions can under-represent inter-dimensional correlations, motivating latent-variable remedies such as VADD (Xie et al., 23 May 2025). Third, in discrete diffusion LLMs, the training objective assumes full bidirectional context while practical blockwise decoding commits tokens without future context, which motivates self-containedness-based block selection (Wang et al., 27 Apr 2026).

A plausible implication is that the most durable future syntheses will combine exact CTMC-based reasoning where possible, richer dependence models where factorization fails, and adaptive decoding or stopping rules where reverse uncertainty remains high.