Masked Diffusion Language Models (MDMs)

• MDMs are discrete diffusion models for text that progressively mask and reconstruct sequences, allowing any-order and parallel token infilling.
• They employ an absorbing mask token and factorized reverse transitions to efficiently denoise multiple positions simultaneously.
• Recent developments demonstrate that with optimized training and simplified objectives, MDMs match autoregressive perplexity while improving inference efficiency.

Masked diffusion-based LLMs (MDMs) are discrete diffusion models for text in which a clean sequence is progressively corrupted by an absorbing mask token and then reconstructed by iterative denoising. Unlike standard autoregressive (AR) LLMs, which commit to a fixed left-to-right factorization, MDMs permit any-order generation and can infill multiple masked positions in parallel. Recent work has shown that simple masked discrete diffusion is more effective than earlier comparisons suggested: with an effective training recipe and a simplified Rao–Blackwellized objective, masked diffusion can be trained as a mixture of classical masked language modeling losses, reach a new state of the art among diffusion models, and approach AR perplexity while admitting efficient samplers, including semi-autoregressive generation for arbitrary text lengths (Sahoo et al., 2024).

1. Core probabilistic structure

An MDM operates on a token sequence $x_0=(x_0^1,\dots,x_0^L)$ over a vocabulary augmented with a special mask token. In the standard absorbing forward process, each position is independently kept or replaced by the mask according to a time-dependent schedule. A common formulation is

$$q_{t\mid 0}(x_t\mid x_0)=\prod_{i=1}^L q_{t\mid 0}(x_t^i\mid x_0^i),$$

with $q_{t\mid 0}(x_t^i\mid x_0^i)=\alpha_t$ when $x_t^i=x_0^i$ and $1-\alpha_t$ when $x_t^i$ is the mask token, often with $\alpha_t=1-t$ (Nie et al., 2024). The learned reverse model predicts the clean token distribution at masked positions and induces reverse transitions that also factorize across positions. In continuous time, training minimizes a negative ELBO or an equivalent reweighted cross-entropy over masked positions, typically of the form

$$\mathcal L_{\mathrm{MDM}} = \mathbb E_{t,x_t} \left[ \frac{1}{t} \sum_{i:x_t^i=m} -\log p_\theta(x_0^i\mid x_t) \right]$$

for the linear schedule (Nie et al., 2024).

This factorized reverse transition is central to the efficiency of MDMs, because many masked positions can be denoised simultaneously. It is also the main approximation. Recent flow-based reformulations state the limitation explicitly: the reverse transition factorizes across token positions, and this approximation degrades in the few-step regime where parallel generation should offer its largest efficiency gains (Azangulov et al., 26 Jun 2026). In discrete MDMs this manifests as a tension between parallelism and cross-token dependence: the model predicts one-dimensional marginals efficiently, but joint consistency across many simultaneously revealed tokens is not guaranteed.

A common misconception is that MDMs are merely masked LLMs used iteratively. The formal distinction is that MDMs are trained through a probabilistic diffusion objective and define a generative reverse process from a fully masked sequence to a clean sample. That diffusion interpretation gives them a likelihood-based training criterion, a sampling schedule, and an explicit connection to reverse-time generation rather than one-shot masked-token prediction (Xue et al., 24 Jun 2025).

2. Objectives, generation order, and architectural realizations

A major line of research has reinterpreted MDMs as order-sensitive LLMs rather than as a wholly separate family from AR models. In a decoder-only formulation, the any-order autoregressive objective averages the chain-rule loss over permutations $\sigma\in S_n$,

$$\mathcal L_{\mathrm{AO\text{-}AR}}(\theta) = \mathbb E_{\sigma\sim \mathrm{Uniform}(S_n)} \left[ -\sum_{i=1}^n \log p_\theta(x_{\sigma_i}\mid x_{\sigma_{<i}}) \right],$$

and NADE-style derivations show that the standard MDM objective is equivalent to this any-order AR objective (Xue et al., 24 Jun 2025). The consequence is conceptual rather than merely notational: the difference between diffusion and autoregression can be recast as a difference in order distribution, conditioning structure, and architecture.

This reinterpretation has been strengthened by schedule-learning results. With multivariate noise schedules $$q_{t\mid 0}(x_t\mid x_0)=\prod_{i=1}^L q_{t\mid 0}(x_t^i\mid x_0^i),$$0, the continuous-time MDM objective decomposes exactly into a weighted mixture of autoregressive losses over decoding orders, so that MDMs become “learned-order” autoregressive models whose order-distribution is induced by the schedule itself (Garg et al., 24 Nov 2025). Related work generalizes this into an order-expressive framework, OeMDM, in which position- and context-dependent schedules subsume standard MDMs, AR models, block diffusion, and GenMD4. Building on OeMDM, LoMDM jointly learns the diffusion backbone and the generation ordering from scratch and reports the best diffusion perplexity among the compared discrete diffusion baselines on LM1B and OpenWebText (Hong et al., 2 Feb 2026).

Architecturally, most early text MDMs were encoder-only, since bidirectional attention matches the conditional structure of masked-token prediction. Yet this makes direct comparison with decoder-only AR models ambiguous. A decoder-only MDM formulation shows that encoder-only MDMs model a simpler conditional probability space, while decoder-only MDMs can obtain roughly $$q_{t\mid 0}(x_t\mid x_0)=\prod_{i=1}^L q_{t\mid 0}(x_t^i\mid x_0^i),$$1 faster generation and comparable generation perplexity with temperature annealing, at the cost of modeling a much larger order-dependent conditional family (Xue et al., 24 Jun 2025). Taken together, these results suggest that “diffusion versus autoregression” is not a single axis: objective, order distribution, and attention architecture can be varied semi-independently.

3. Parallel generation, computational classes, and reasoning

The operational appeal of MDMs is parallel unmasking. A formal MDM can be described by a forward chain that masks tokens with probability $$q_{t\mid 0}(x_t\mid x_0)=\prod_{i=1}^L q_{t\mid 0}(x_t^i\mid x_0^i),$$2 at step $$q_{t\mid 0}(x_t\mid x_0)=\prod_{i=1}^L q_{t\mid 0}(x_t^i\mid x_0^i),$$3, together with a learned reverse chain that factorizes position-wise. In practical terms, a transformer implements a planner that selects which masked positions to unmask and a predictor that infills them independently and simultaneously (Svete et al., 15 Oct 2025). If a task decomposes into independent subproblems, this permits much fewer transformer evaluations than left-to-right decoding.

Theoretical analysis makes this precise. In the finite-precision, log-width setting, MDMs and polynomially padded looped transformers (PLTs) are equivalent as language recognizers: for any denoising steps $$q_{t\mid 0}(x_t\mid x_0)=\prod_{i=1}^L q_{t\mid 0}(x_t^i\mid x_0^i),$$4 and padding $$q_{t\mid 0}(x_t\mid x_0)=\prod_{i=1}^L q_{t\mid 0}(x_t^i\mid x_0^i),$$5, $$q_{t\mid 0}(x_t\mid x_0)=\prod_{i=1}^L q_{t\mid 0}(x_t^i\mid x_0^i),$$6, up to an $$q_{t\mid 0}(x_t\mid x_0)=\prod_{i=1}^L q_{t\mid 0}(x_t^i\mid x_0^i),$$7 padding enlargement when simulating PLTs by MDMs (Svete et al., 15 Oct 2025). The same framework shows that MDMs and chain-of-thought (CoT) transformers simulate one another with polynomial overhead, but also identifies classes where MDMs are strictly more efficient. With $$q_{t\mid 0}(x_t\mid x_0)=\prod_{i=1}^L q_{t\mid 0}(x_t^i\mid x_0^i),$$8 and $$q_{t\mid 0}(x_t\mid x_0)=\prod_{i=1}^L q_{t\mid 0}(x_t^i\mid x_0^i),$$9, MDMs recognize all regular languages; with constant $q_{t\mid 0}(x_t^i\mid x_0^i)=\alpha_t$0 and polynomial padding they capture $q_{t\mid 0}(x_t^i\mid x_0^i)=\alpha_t$1; with $q_{t\mid 0}(x_t^i\mid x_0^i)=\alpha_t$2 and polynomial padding they capture $q_{t\mid 0}(x_t^i\mid x_0^i)=\alpha_t$3 (Svete et al., 15 Oct 2025). The stated implication is a “sequentiality bottleneck”: parallel unmasking helps on highly parallelizable problems, but it does not by itself overcome inherently sequential complexity barriers.

Empirical work on LoopMDM provides complementary evidence from model design. LoopMDM shares the weights of an early-middle transformer block and applies it multiple times per forward pass. On matched training FLOPs, a 170M-parameter LoopMDM reaches baseline NLL with up to $q_{t\mid 0}(x_t^i\mid x_0^i)=\alpha_t$4 fewer FLOPs on LM1B, $q_{t\mid 0}(x_t^i\mid x_0^i)=\alpha_t$5 on OpenWebText, and $q_{t\mid 0}(x_t^i\mid x_0^i)=\alpha_t$6 on FineWeb-Edu; on GSM8K, under matched total FLOPs, it improves over the 14-layer baseline by $q_{t\mid 0}(x_t^i\mid x_0^i)=\alpha_t$7 Top-2 and $q_{t\mid 0}(x_t^i\mid x_0^i)=\alpha_t$8 Top-3 percentage points at $q_{t\mid 0}(x_t^i\mid x_0^i)=\alpha_t$9 (Lee et al., 25 May 2026). The same study reports a Sudoku experiment in which a standard MDM under rigid left-to-right unmasking attains about $x_t^i=x_0^i$0 accuracy, whereas a LoopMDM with one mid-block layer and $x_t^i=x_0^i$1 exactly solves $x_t^i=x_0^i$2 of puzzles, with errors dropping from $x_t^i=x_0^i$3 as $x_t^i=x_0^i$4 increases (Lee et al., 25 May 2026). Attention analysis further shows monotonically increasing mask-to-mask attention as loops grow. This suggests that iterative computation within a denoising step can partially compensate for the independence approximation of parallel token updates.

4. Optimization dynamics and inductive bias

Despite their parallelism, MDMs are not uniformly easy to optimize. A detailed training analysis attributes much of the slowdown relative to AR models to the locality bias of language: predictive information for a token is concentrated in nearby positions, so uniform sampling over masking levels spends excessive computation on contexts that are either too impoverished or too redundant (Hong et al., 13 May 2026). The proposed remedy is bell-shaped time sampling, typically a truncated Gaussian over $x_t^i=x_0^i$5 with $x_t^i=x_0^i$6 and $x_t^i=x_0^i$7. This does not change the infinite-capacity minimizer set of the standard NELBO, but it changes which masking levels are emphasized during optimization. On LM1B, the resulting recipe reaches the same validation NLL up to about $x_t^i=x_0^i$8 faster; the reported speedups are $x_t^i=x_0^i$9 on sentence-packed LM1B, $1-\alpha_t$0 on unpacked LM1B, and $1-\alpha_t$1 on OpenWebText (Hong et al., 13 May 2026).

Controlled task studies show that trainability also depends sharply on whether the target structure is local, global, or directionally misaligned with random masking. On in-context linear regression, standard random-masking MDMs and block diffusion remain at high MSE, while AR and the locality-aware Jigsaw variant drive MSE to zero even at $1-\alpha_t$2. On star-graph path-finding, AR fails at about $1-\alpha_t$3, while standard MDM, SDAR, and Scatter reach about $1-\alpha_t$4 sequence accuracy. On Sudoku, AR remains at $1-\alpha_t$5 solve rate, while standard MDM and Jigsaw achieve about $1-\alpha_t$6 accuracy (Wang et al., 27 Apr 2026). The accompanying interpretation is that random masking trains over many possible generation orders, but only a small subset may align with the task’s dependency graph. Jigsaw and Scatter inject left-to-right locality within blocks while preserving iterative refinement across blocks, and the paper concludes that standard random-masking MDMs, even with blockwise variants, may be a suboptimal instantiation of diffusion LLMs for ordered generation (Wang et al., 27 Apr 2026).

These findings sharpen a broader point. MDMs are not defined by parallelism alone; their practical behavior is strongly mediated by the corruption process, order distribution, and the locality structure of the task. This suggests that progress may depend as much on aligning masking schedules with dependency structure as on scaling the denoiser.

5. Sampling, caching, and inference-time scaling

Inference efficiency has become a major systems theme because full bidirectional attention prevents the straightforward KV caching used by AR decoders. In vanilla MDMs, each of $1-\alpha_t$7 denoising steps incurs an $1-\alpha_t$8 attention computation, for overall $1-\alpha_t$9 inference cost (Goyal et al., 6 Apr 2026). Several recent proposals attack this bottleneck from different directions.

Method	Mechanism	Reported effect
EB-Sampler	Entropy-bounded adaptive multi-token unmasking	roughly $x_t^i$0–$x_t^i$1 fewer function evaluations without loss on coding and math benchmarks (Ben-Hamu et al., 30 May 2025)
Eso-LM	Hybrid AR–MDM model with KV caching for diffusion	up to $x_t^i$2 faster than standard MDMs and $x_t^i$3 faster than prior semi-autoregressive approaches (Sahoo et al., 2 Jun 2025)
DualDiffusion	Speculative decoding with drafter and verifier MDMs	$x_t^i$4 speedup versus LLaDA on MMLU and $x_t^i$5 on GSM8K, with a large GSM8K accuracy gap to the verifier (Goyal et al., 6 Apr 2026)

EB-Sampler is motivated by the observation that a partially masked sequence often determines several unknown tokens almost deterministically. It sorts masked positions by an error proxy such as entropy or confidence, then unmasks the largest prefix whose total joint-dependence surrogate remains under a user-chosen tolerance $x_t^i$6. The paper provides a KL upper bound showing a trade-off of the form model error plus a joint-dependence term controlled by $x_t^i$7 (Ben-Hamu et al., 30 May 2025).

Eso-LMs fuse a masked-diffusion “partial” decoder with an autoregressive completion decoder. The resulting family interpolates smoothly between pure MDM and pure AR via an initial masking parameter $x_t^i$8, and, crucially, introduces KV caching for MDM-style generation through causalized attention over already denoised tokens (Sahoo et al., 2 Jun 2025). This is explicitly presented as the first KV-cached MDM while preserving parallel generation.

Inference-time scaling has also been explored on conditional generation. For text style transfer, an MDM with classifier-free guidance and a soft-value diffusion decoding verifier uses off-the-shelf sentence embeddings to score candidates during denoising. On WikiLarge, the reported MDM with CFG+SVDD attains BLEU $x_t^i$9, SARI $\alpha_t=1-t$0, and LENS $\alpha_t=1-t$1, compared with BLEU $\alpha_t=1-t$2, SARI $\alpha_t=1-t$3, and LENS $\alpha_t=1-t$4 for MDM with CFG only (Padole et al., 14 Aug 2025). This indicates that verifier-guided candidate selection can be layered on top of standard MDM sampling without retraining the denoiser.

6. Scaling behavior, extensions, and unresolved issues

Large-scale evidence now indicates that MDMs are not merely small-model curiosities. The first scaling-law study for text MDMs reports a scaling exponent comparable to AR models, with $\alpha_t=1-t$5 versus $\alpha_t=1-t$6, and a constant compute gap of about $\alpha_t=1-t$7 FLOPs to match ARM loss (Nie et al., 2024). The same work reports optimal model size scaling $\alpha_t=1-t$8, and a 1.1B MDM that outperforms a same-data 1.1B TinyLlama on four of eight zero-shot benchmarks. It also reports that MDMs break the reverse curse: on the synthetic reverse-direction benchmark, the 1.1B MDM reaches $\alpha_t=1-t$9 reverse-direction accuracy on Desc$$\mathcal L_{\mathrm{MDM}} = \mathbb E_{t,x_t} \left[ \frac{1}{t} \sum_{i:x_t^i=m} -\log p_\theta(x_0^i\mid x_t) \right]$$0Name and $$\mathcal L_{\mathrm{MDM}} = \mathbb E_{t,x_t} \left[ \frac{1}{t} \sum_{i:x_t^i=m} -\log p_\theta(x_0^i\mid x_t) \right]$$1 BLEU on Name$$\mathcal L_{\mathrm{MDM}} = \mathbb E_{t,x_t} \left[ \frac{1}{t} \sum_{i:x_t^i=m} -\log p_\theta(x_0^i\mid x_t) \right]$$2Desc Reverse, while much larger AR baselines collapse to zero on the reverse direction (Nie et al., 2024). On temporal-shift evaluation, the reported FineWeb perplexities are $$\mathcal L_{\mathrm{MDM}} = \mathbb E_{t,x_t} \left[ \frac{1}{t} \sum_{i:x_t^i=m} -\log p_\theta(x_0^i\mid x_t) \right]$$3 and $$\mathcal L_{\mathrm{MDM}} = \mathbb E_{t,x_t} \left[ \frac{1}{t} \sum_{i:x_t^i=m} -\log p_\theta(x_0^i\mid x_t) \right]$$4 for the MDM versus $$\mathcal L_{\mathrm{MDM}} = \mathbb E_{t,x_t} \left[ \frac{1}{t} \sum_{i:x_t^i=m} -\log p_\theta(x_0^i\mid x_t) \right]$$5 and $$\mathcal L_{\mathrm{MDM}} = \mathbb E_{t,x_t} \left[ \frac{1}{t} \sum_{i:x_t^i=m} -\log p_\theta(x_0^i\mid x_t) \right]$$6 for the ARM, indicating greater robustness to distributional shift (Nie et al., 2024).

Several extensions address the central weakness of factorized discrete denoising. Prime introduces partial masking by decomposing each token into sub-tokens with intermediate states between fully masked and fully unmasked. On OpenWebText it reports perplexity $$\mathcal L_{\mathrm{MDM}} = \mathbb E_{t,x_t} \left[ \frac{1}{t} \sum_{i:x_t^i=m} -\log p_\theta(x_0^i\mid x_t) \right]$$7, outperforming the previous MDM result $$\mathcal L_{\mathrm{MDM}} = \mathbb E_{t,x_t} \left[ \frac{1}{t} \sum_{i:x_t^i=m} -\log p_\theta(x_0^i\mid x_t) \right]$$8, an autoregressive model at $$\mathcal L_{\mathrm{MDM}} = \mathbb E_{t,x_t} \left[ \frac{1}{t} \sum_{i:x_t^i=m} -\log p_\theta(x_0^i\mid x_t) \right]$$9, and a hybrid at $\sigma\in S_n$0, without using an autoregressive formulation (Chao et al., 24 May 2025). CRoCoDiL instead moves guidance into a continuous sentence-level latent space. With a jointly trained encoder–demasker and two hybrid synthesis algorithms, it reports superior generation quality and more than $\sigma\in S_n$1 faster sampling speeds in unconditional generation; at 1024 tokens, ConWithinDisc with NFE $\sigma\in S_n$2 reports MAUVE $\sigma\in S_n$3 and perplexity $\sigma\in S_n$4, versus MAUVE $\sigma\in S_n$5 and perplexity $\sigma\in S_n$6 for vanilla LLaDA with NFE $\sigma\in S_n$7 (Uziel et al., 2 Mar 2026). MLFMs combine masking with continuous flows through a Brownian-bridge interpolant and a sampler that alternates continuous denoising with confident-token promotion; the reported MT-Bench score is $\sigma\in S_n$8 versus $\sigma\in S_n$9 for SMDM and $$\mathcal L_{\mathrm{AO\text{-}AR}}(\theta) = \mathbb E_{\sigma\sim \mathrm{Uniform}(S_n)} \left[ -\sum_{i=1}^n \log p_\theta(x_{\sigma_i}\mid x_{\sigma_{<i}}) \right],$$0 for an AR baseline, though GSM8K remains lower at $$\mathcal L_{\mathrm{AO\text{-}AR}}(\theta) = \mathbb E_{\sigma\sim \mathrm{Uniform}(S_n)} \left[ -\sum_{i=1}^n \log p_\theta(x_{\sigma_i}\mid x_{\sigma_{<i}}) \right],$$1 versus about $$\mathcal L_{\mathrm{AO\text{-}AR}}(\theta) = \mathbb E_{\sigma\sim \mathrm{Uniform}(S_n)} \left[ -\sum_{i=1}^n \log p_\theta(x_{\sigma_i}\mid x_{\sigma_{<i}}) \right],$$2 for SMDM and LLaMA-2 (Azangulov et al., 26 Jun 2026).

MDMs have also exposed paradigm-specific failure modes in downstream tooling. In knowledge editing, the location of factual memory transfers from ARMs to MDMs—early-to-mid-layer MLP at the last subject token—but the outcome does not. Single-token edits succeed in both settings, whereas multi-token edits degrade sharply in MDMs because generation passes through partially unmasked intermediate states for which the edit was never optimized. Optimizing the edit over those intermediate mask patterns raises LLaDA’s efficacy at target length $$\mathcal L_{\mathrm{AO\text{-}AR}}(\theta) = \mathbb E_{\sigma\sim \mathrm{Uniform}(S_n)} \left[ -\sum_{i=1}^n \log p_\theta(x_{\sigma_i}\mid x_{\sigma_{<i}}) \right],$$3 from $$\mathcal L_{\mathrm{AO\text{-}AR}}(\theta) = \mathbb E_{\sigma\sim \mathrm{Uniform}(S_n)} \left[ -\sum_{i=1}^n \log p_\theta(x_{\sigma_i}\mid x_{\sigma_{<i}}) \right],$$4 to $$\mathcal L_{\mathrm{AO\text{-}AR}}(\theta) = \mathbb E_{\sigma\sim \mathrm{Uniform}(S_n)} \left[ -\sum_{i=1}^n \log p_\theta(x_{\sigma_i}\mid x_{\sigma_{<i}}) \right],$$5, and Dream’s from $$\mathcal L_{\mathrm{AO\text{-}AR}}(\theta) = \mathbb E_{\sigma\sim \mathrm{Uniform}(S_n)} \left[ -\sum_{i=1}^n \log p_\theta(x_{\sigma_i}\mid x_{\sigma_{<i}}) \right],$$6 to $$\mathcal L_{\mathrm{AO\text{-}AR}}(\theta) = \mathbb E_{\sigma\sim \mathrm{Uniform}(S_n)} \left[ -\sum_{i=1}^n \log p_\theta(x_{\sigma_i}\mid x_{\sigma_{<i}}) \right],$$7 (Park et al., 2 Jun 2026).

Open problems remain substantial. Theoretical results rely on finite-precision, log-width assumptions, and identify output-space blowups when simulating unmasked attention in causal models; they also emphasize that inherently sequential problems such as Circuit-Value, LP-solving, and CFG membership do not become easy merely because an MDM can unmask in parallel (Svete et al., 15 Oct 2025). Empirically, the reverse factorization still breaks down in few-step sampling, random masking remains a questionable default for ordered generation, and some successful architectural choices—such as the manually selected looped mid-block in LoopMDM—remain heuristic (Azangulov et al., 26 Jun 2026, Wang et al., 27 Apr 2026, Lee et al., 25 May 2026). The current literature therefore presents MDMs less as a finished replacement for autoregression than as a rapidly developing design space in which order, masking geometry, latent parameterization, and systems techniques jointly determine performance.