Masked Discrete Diffusions

• Masked discrete diffusions are models on finite alphabets that use an absorbing mask token to progressively corrupt data.
• They employ continuous-time Markov chains and tailored masking schedules to enable structured variational training across diverse domains.
• Their reverse process recovers original tokens through masked-language-model style losses, ensuring efficient sampling and strong convergence guarantees.

Searching arXiv for papers on masked discrete diffusion to ground the article. Masked discrete diffusions are diffusion models on finite alphabets in which the forward corruption process progressively replaces tokens with a distinguished absorbing symbol, usually written as $m$ or $[MASK]$, and the reverse model learns to reconstruct data from partially or fully masked states. In continuous time they are naturally formulated as continuous-time Markov chains (CTMCs) on discrete product spaces; in discrete time they appear as schedules of masking probabilities or unmasking sets. Recent work has turned this apparently simple absorbing process into a substantial theory spanning variational training, information geometry, schedule design, CTMC sampling guarantees, and domain-specific instantiations in language, proteins, images, and single-cell omics (Pauline et al., 4 Dec 2025, Shi et al., 2024).

1. Forward process and state-space formulation

The canonical masked discrete diffusion augments a vocabulary with a single absorbing mask state. For a single token, the cumulative forward kernel has the form

$$Q_t=\alpha_t I + (1-\alpha_t)e_{[MASK]}\mathbf 1^\top,$$

so that

$$q_t(x\mid x_0)=\alpha_t\,\delta_{x,x_0}+(1-\alpha_t)\,\delta_{x,[MASK]}.$$

Here $\alpha_t$ is a survival probability, typically written as

$$\alpha_t=\exp\!\left(-\int_0^t \beta(s)\,ds\right),$$

with $\beta(t)\ge 0$. For sequences, coordinates evolve independently in the forward process, so

$$q(x_t\mid x_0)=\prod_{n=1}^N q(x_t^{(n)}\mid x_0^{(n)}),$$

and the forward marginal path is

$q_t(x_t)=\mathbb E_{x_0\sim q_0}[q(x_t\mid x_0)].$

This is the standard absorbing or masking kernel in discrete diffusion theory (Pauline et al., 4 Dec 2025, Shi et al., 2024).

The CTMC limit makes the absorbing structure explicit. For masking diffusion,

$$R_t=\frac{\alpha_t'}{\alpha_t}\Big(I-e_{[MASK]}\mathbf 1^\top\Big),$$

so the only nonzero off-diagonal rates are jumps from a non-mask token into $[MASK]$0; once a coordinate becomes masked, it remains masked in the forward chain (Pauline et al., 4 Dec 2025). In the sequence setting, the joint generator is the Kronecker sum of tokenwise generators, so only Hamming-distance-1 transitions appear in continuous time.

For masked sequence models, the probability path is effectively governed by the scalar parameter $[MASK]$1. In the information-geometric treatment of masked diffusion, the forward marginal can be written as a binomial-like factor in the number of masked coordinates,

$[MASK]$2

times a consistency term enforcing agreement between visible coordinates and the original data sample. This makes the masking path a one-dimensional curve in distribution space, parameterized by $[MASK]$3 rather than by the full combinatorics of the ambient sequence space (Zhang, 6 Aug 2025).

2. Reverse process, score parameterizations, and training objectives

The exact reverse conditional of masked diffusion is unusually simple. If the current token is not masked, the previous token is deterministically the same; if the current token is masked, the previous token is a mixture of remaining masked and reverting to the clean token $[MASK]$4. This gives masked diffusion its characteristic “copy visible tokens, predict masked tokens” structure (Pauline et al., 4 Dec 2025).

A central consequence is that the continuous-time variational objective collapses to a masked-language-model-style loss. In the general-state-space derivation, the denoising score entropy objective specializes to

$[MASK]$5

which is a time-reweighted masked-LM cross-entropy (Pauline et al., 4 Dec 2025). In the simplified continuous-time formulation, masked diffusion training is therefore a weighted integral of cross-entropy losses over currently masked positions rather than a high-variance categorical score-matching objective (Shi et al., 2024).

This reduction also explains the standard mean parameterization. Instead of directly parameterizing reverse transition rates, one predicts a clean-token distribution $[MASK]$6 for masked positions and plugs it into the analytic reverse kernel. In the absorbing setting, the discrete score at a masked state is proportional to the posterior mean of the clean token, so score-based and mean-based parameterizations coincide after enforcing the masking constraint (Shi et al., 2024).

More general objectives preserve the same structure. State-dependent masking schedules replace the scalar $[MASK]$7 by a vector-valued survival profile, allowing different token types to be masked at different rates while keeping a variational objective of the same absorbing form (Shi et al., 2024). For mixed discrete–continuous data, the same template can be extended by predicting both token identity and an associated continuous attribute, as in joint gene-identity/gene-expression denoising (Wang et al., 3 Feb 2026).

3. Schedules, information geometry, and why cosine laws appear

A discretization schedule can be viewed as a reparameterization of the same continuous probability path. If $[MASK]$8, then different choices of $[MASK]$9 choose different intermediate marginals $$Q_t=\alpha_t I + (1-\alpha_t)e_{[MASK]}\mathbf 1^\top,$$0 while preserving the underlying masked diffusion. In the Fisher–Rao analysis of masked diffusion, the path metric is

$$Q_t=\alpha_t I + (1-\alpha_t)e_{[MASK]}\mathbf 1^\top,$$1

and the geodesic, constant-speed parameterization yields

$$Q_t=\alpha_t I + (1-\alpha_t)e_{[MASK]}\mathbf 1^\top,$$2

When the endpoint is fully masked, $$Q_t=\alpha_t I + (1-\alpha_t)e_{[MASK]}\mathbf 1^\top,$$3, this becomes

$$Q_t=\alpha_t I + (1-\alpha_t)e_{[MASK]}\mathbf 1^\top,$$4

which is exactly the cosine schedule. The result identifies the widely used cosine law as the unique Fisher–Rao geodesic schedule for masked discrete diffusion under the true probability path (Zhang, 6 Aug 2025).

A separate line of work explains masking’s strong empirical performance from the jump-process viewpoint. Schedule-conditioned discrete diffusion argues that masking succeeds because it bakes in the known distribution of jump times and only learns where jumps go; in that perspective, the key benefit is not merely the absorbing state itself, but the fact that the model need not relearn the schedule of jump events (Amin et al., 10 Jun 2025). This reframes masking as one endpoint of a broader family of schedule-conditioned jump-process models.

For factorized multi-token updates, schedule optimization takes a different form. Under random-order planners, the factorization error is expressed through an information profile $$Q_t=\alpha_t I + (1-\alpha_t)e_{[MASK]}\mathbf 1^\top,$$5, and asymptotically optimal non-constant schedules solve a variational problem. In the scaling limit, if $$Q_t=\alpha_t I + (1-\alpha_t)e_{[MASK]}\mathbf 1^\top,$$6 is the normalized derivative of the information profile and

$$Q_t=\alpha_t I + (1-\alpha_t)e_{[MASK]}\mathbf 1^\top,$$7

then the optimal schedule is

$$Q_t=\alpha_t I + (1-\alpha_t)e_{[MASK]}\mathbf 1^\top,$$8

This gives a principled schedule beyond uniform block sizes: when dependencies are concentrated in particular stages of generation, optimal masked-diffusion schedules should allocate smaller blocks there and larger blocks elsewhere (Lavenant et al., 29 Oct 2025).

4. Reverse-time simulation and convergence guarantees

Once a reverse score or clean-token predictor has been learned, sampling requires approximating the reverse CTMC. For masking diffusion, the exact score for a Hamming-distance-1 unmasking move has an especially simple form: $$Q_t=\alpha_t I + (1-\alpha_t)e_{[MASK]}\mathbf 1^\top,$$9 where $$q_t(x\mid x_0)=\alpha_t\,\delta_{x,x_0}+(1-\alpha_t)\,\delta_{x,[MASK]}.$$0 is masked at one coordinate and $$q_t(x\mid x_0)=\alpha_t\,\delta_{x,x_0}+(1-\alpha_t)\,\delta_{x,[MASK]}.$$1 is obtained by unmasking that coordinate. This separability in time and data distribution underlies several sharp sampling analyses (Dmitriev et al., 16 Feb 2026).

One such analysis studies modified truncated $$q_t(x\mid x_0)=\alpha_t\,\delta_{x,x_0}+(1-\alpha_t)\,\delta_{x,[MASK]}.$$2-leaping for masking diffusion. Its key information-theoretic quantity is the effective total correlation,

$$q_t(x\mid x_0)=\alpha_t\,\delta_{x,x_0}+(1-\alpha_t)\,\delta_{x,[MASK]}.$$3

with $$q_t(x\mid x_0)=\alpha_t\,\delta_{x,x_0}+(1-\alpha_t)\,\delta_{x,[MASK]}.$$4 defined as a sum of conditional mutual informations along the masking path. Under an exponential-then-constant schedule, the iteration complexity satisfies

$$q_t(x\mid x_0)=\alpha_t\,\delta_{x,x_0}+(1-\alpha_t)\,\delta_{x,[MASK]}.$$5

and $$q_t(x\mid x_0)=\alpha_t\,\delta_{x,x_0}+(1-\alpha_t)\,\delta_{x,[MASK]}.$$6, but can be much smaller for structured data such as hidden Markov models, manifold-like image distributions, or random graphs. The result is an adaptive guarantee: the same masking sampler provably speeds up on low-dimensional structure without being told that structure in advance (Dmitriev et al., 16 Feb 2026).

For absorbing-rate CTMCs, a direct total-variation analysis of the Euler sampler gives a complementary picture. The upper bound is $$q_t(x\mid x_0)=\alpha_t\,\delta_{x,x_0}+(1-\alpha_t)\,\delta_{x,[MASK]}.$$7 steps to achieve TV error $$q_t(x\mid x_0)=\alpha_t\,\delta_{x,x_0}+(1-\alpha_t)\,\delta_{x,[MASK]}.$$8, and there is a matching lower bound showing that this dependence on $$q_t(x\mid x_0)=\alpha_t\,\delta_{x,x_0}+(1-\alpha_t)\,\delta_{x,[MASK]}.$$9 and $\alpha_t$0 is tight up to logarithmic factors. The same work proves that the First-Hitting Sampler (FHS), a sampler specialized to masked diffusion, incurs no sampling error beyond the score-estimation error; its KL guarantee is tight with a matching lower bound (Liang et al., 26 Feb 2026).

A separate non-asymptotic theory establishes KL and TV guarantees for masked diffusion without assuming bounded estimated scores. In that framework, the forward process is an absorbing masking CTMC on $\alpha_t$1, the backward process is described by a discrete score ratio, and suitably chosen early stopping plus step-size schedules yield linear dependence on dimension up to logarithmic factors (Conforti et al., 29 Nov 2025). At a more general level, Lévy-type stochastic-integral formulations for discrete diffusion identify truncation, approximation, and discretization errors via a Poisson-random-measure analog of Girsanov’s theorem, and yield the first KL error bound for $\alpha_t$2-leaping in this setting (Ren et al., 2024).

5. Variants and application domains

The absorbing-mask template has been extended in several orthogonal directions: richer latent state spaces, non-factorized or speculative reverse samplers, joint discrete–continuous decoders, and image-token systems with explicit editing. These models preserve the central masking corruption while modifying how information is represented or how reverse steps are executed.

Direction	Mechanism	Reported outcome
MD4 / GenMD4	Continuous-time weighted cross-entropy objective; generalized state-dependent masking schedules	$\alpha_t$3 on CIFAR-10 and $\alpha_t$4 on ImageNet 64x64 bits per dimension (Shi et al., 2024)
Prime	Partial masking via sub-token intermediate states between masked and unmasked	OpenWebText perplexity $\alpha_t$5; FID $\alpha_t$6 on CIFAR-10 and $\alpha_t$7 on ImageNet-32 (Chao et al., 24 May 2025)
Self-speculative MDMs	Causal validation head inside a masked-diffusion transformer for non-factorized speculative decoding	$\alpha_t$8 reduction in required network forward passes (Campbell et al., 4 Oct 2025)
scDiVa	Masked discrete diffusion over gene identities plus continuous expression values	Pre-trained on 59 million cells; strong transfer on batch integration, annotation, and perturbation response (Wang et al., 3 Feb 2026)
Nemotron-Labs-Diffusion-Image	Token editing for self-correction and Grouped Cross-Entropy for large image vocabularies	$\alpha_t$9 on GenEval, $$\alpha_t=\exp\!\left(-\int_0^t \beta(s)\,ds\right),$$0 on DPG, and $$\alpha_t=\exp\!\left(-\int_0^t \beta(s)\,ds\right),$$1 of HPSv3 (Li et al., 29 Jun 2026)

These variants clarify what is intrinsic to masked discrete diffusion and what is not. The absorbing corruption process itself is stable across text, image tokens, proteins, and omics, but model behavior changes substantially when the latent state space is refined. Prime inserts intermediate partially observed states between masked and unmasked tokens, reducing idle reverse steps and making denoising more fine-grained (Chao et al., 24 May 2025). Self-speculative masked diffusion keeps the absorbing process but replaces factorized parallel token prediction with a hybrid non-causal/causal mechanism, so that multiple masked positions can be proposed and validated with a non-factorized predictive law (Campbell et al., 4 Oct 2025).

Domain-specific instantiations also expose what masking aligns with. In scDiVa, the forward process is explicitly matched to dropout-like missingness in single-cell RNA-seq, with a bidirectional denoiser that jointly reconstructs discrete gene identities and continuous values (Wang et al., 3 Feb 2026). In high-resolution image generation, Nemotron-Labs-Diffusion-Image identifies two discrete-image-specific bottlenecks: lack of self-correction once a token is unmasked, and sparse training signal for very large token vocabularies; token editing and Grouped Cross-Entropy are introduced precisely to repair those failure modes while retaining a masked-discrete-diffusion backbone (Li et al., 29 Jun 2026).

6. Comparative assessments, misconceptions, and open directions

A persistent misconception is that masking’s strong empirical record proves that absorbing corruption is universally the best discrete noising mechanism. One counter-argument is theoretical: schedule-conditioned discrete diffusion attributes masking’s strength to baking in the known distribution of jump times rather than forcing the model to learn them (Amin et al., 10 Jun 2025). Another is empirical: on random $$\alpha_t=\exp\!\left(-\int_0^t \beta(s)\,ds\right),$$2-SAT and $$\alpha_t=\exp\!\left(-\int_0^t \beta(s)\,ds\right),$$3-XORSAT benchmarks, continuous diffusions outperform masked discrete diffusions, even though the data themselves are discrete; the same study also reports that learned local denoisers can match the theoretical “ideal” accuracy and that smart variable ordering can significantly improve accuracy, although not following popular heuristics (Bhatt et al., 21 Mar 2026).

A second tension concerns training versus inference. Masked diffusion models are trained over an exponentially large family of infilling problems, and the analysis of token ordering shows that, relative to autoregressive models, they indeed train on computationally intractable subproblems in certain synthetic settings. Yet the same flexibility creates a major inference advantage: by choosing decoding order adaptively, pretrained masked diffusion models can avoid especially hard subproblems. On Sudoku, adaptive inference raised solving accuracy from $$\alpha_t=\exp\!\left(-\int_0^t \beta(s)\,ds\right),$$4 to $$\alpha_t=\exp\!\left(-\int_0^t \beta(s)\,ds\right),$$5, outperforming autoregressive models with $$\alpha_t=\exp\!\left(-\int_0^t \beta(s)\,ds\right),$$6 as many parameters that had been explicitly trained to follow the right decoding order (Kim et al., 10 Feb 2025). This makes order selection a central algorithmic variable rather than a peripheral implementation detail.

Several open directions are explicit in the current literature. The Fisher–Rao cosine-schedule result is derived for the true masked probability path and does not account for reverse-model approximation error or alternative corruption kernels; extending it beyond mask-only corruption requires recomputing the path metric (Zhang, 6 Aug 2025). The adaptive $$\alpha_t=\exp\!\left(-\int_0^t \beta(s)\,ds\right),$$7-leaping theory leaves open analogous adaptive schemes for uniform diffusion, methods for learning scores that minimize effective-total-correlation-driven error, and extensions to other noising mechanisms (Dmitriev et al., 16 Feb 2026). More broadly, the strongest domain-scale successes now come from relaxing the classic monotone-unmasking assumption—through token editing, partial masking, or speculative non-factorized decoders—while keeping the absorbing-mask prior as the organizing principle. This suggests that the most durable content of masked discrete diffusion is not a single sampler or objective, but a family of absorbing-state generative processes whose tractability comes from the mask token and whose performance depends on how aggressively one exploits the structure exposed by masking.