Masked Diffusion Language Modeling

• Masked Diffusion Language Modeling is a method that iteratively reconstructs text by denoising masked tokens, enabling parallel and bidirectional prediction.
• It replaces tokens with a special mask via a forward process and learns to recover the original sequence through an iterative reverse denoising process.
• The approach supports diverse decoding policies and has been applied to multilingual, multimodal, and biological sequence generation with promising results.

Masked diffusion language modeling (MDLM) is a family of discrete diffusion methods for text in which a clean token sequence is corrupted by replacing tokens with a special absorbing mask symbol and is then reconstructed by an iterative reverse denoising process. In contrast to autoregressive LLMs, which generate strictly left-to-right under a factorization such as $p(x)=\prod_{t=1}^{T} p(x_t \mid x_{<t})$, MDLMs predict masked positions in parallel and can revise multiple positions over several denoising steps. Recent work has developed MDLMs from encoder-initialized absorbing-state diffusion into a broader design space encompassing simplified continuous-time objectives, learned and training-free decoding policies, order-aware generation, and domain-specific systems for multilingual, multimodal, and biological sequence generation (He et al., 2022, Sahoo et al., 2024, Hong et al., 2 Feb 2026).

1. Formal definition and basic generative mechanism

In the standard absorbing-state formulation, the forward process gradually replaces tokens with a special mask token. A representative single-token transition is

$$q(x_t^i \mid x_{t-1}^i) = \begin{cases} 1 - \beta_t & \text{if } x_t^i = x_{t-1}^i, \ \beta_t & \text{if } x_t^i = \langle mask\rangle, \ 0 & \text{otherwise.} \end{cases}$$

As $t$ increases, more tokens are masked until the terminal state is essentially all masks. The learned reverse process predicts less corrupted states from more corrupted ones, typically via a denoiser $p_\theta(x_{t-1}\mid x_t)$ or an equivalent $x_0$-prediction parameterization (Kocabay et al., 20 Mar 2026).

This absorbing-mask construction is also the canonical setup in early discrete diffusion LLMs. DiffusionBERT, for example, formulates text diffusion directly in discrete token space, uses $[MASK]$ as the absorbing state, and trains a BERT backbone to learn the reverse process. The model begins from a fully masked sequence and iteratively predicts $$\mathbf{x}_{T-1},\mathbf{x}_{T-2},\dots,\mathbf{x}_0$$, while allowing uncertain tokens to be reconsidered in later denoising steps rather than fixing them permanently after a single prediction (He et al., 2022).

The defining contrast with autoregressive modeling is therefore not only parallelism. MDLMs replace a fixed causal factorization with an iterative denoising trajectory over partially observed sequences. This permits bidirectional conditioning on visible context and supports any-order or semi-autoregressive decoding schemes, depending on the reverse-time schedule and unmasking policy (Luo et al., 2 Feb 2026).

2. Objectives, parameterizations, and training recipes

A standard continuous-time MDLM objective is a masked-token negative ELBO of the form

$$\mathcal{L}_{\mathrm{MDM}} = \int_{0}^{1} \frac{-\alpha_t'}{1-\alpha_t} \mathbb{E}\Bigl[ -\sum_{i:x_t^i=m}\log p_\theta(x^i\mid x_t,t) \Bigr]\,dt,$$

where $\alpha_t$ controls the masking schedule. In practice, this makes MDLM training a weighted average of masked-token reconstruction losses over noise levels, rather than a single fixed-rate masked language modeling objective (Hong et al., 13 May 2026).

Several lines of work simplify or restructure that objective. "Simple and Effective Masked Diffusion LLMs" reports an effective training recipe and a simplified, Rao-Blackwellized objective whose form is a mixture of classical masked language modeling losses; in that formulation, encoder-only LLMs can be trained with efficient samplers, including ones that generate arbitrary lengths of text semi-autoregressively, and a range of masked diffusion models trained with modern engineering practices achieves a new state-of-the-art among diffusion models and approaches autoregressive perplexity (Sahoo et al., 2024). DiffusionBERT instead emphasizes encoder initialization, a spindle noise schedule that masks informative tokens earlier, and time-agnostic decoding, with the latter reported as the best of several time-conditioning designs (He et al., 2022). A later denoising-based USDM variant, SDDLM, argues that a naïve full reconstruction loss “often causes models to collapse during training” because unchanged positions induce a conflicting copy-the-input signal; its core simplification is to optimize only the noise-replaced tokens, and contrastive-inspired negative-gradient variants are reported to improve generation quality substantially, especially in the few-step regime (Zhu et al., 27 Oct 2025).

Training behavior is also sensitive to masking policy and time sampling. Frequency-informed masking for BabyLM 2025 increases masking probability on rarer tokens while preserving the target global masking rate, and its ablations report that cosine schedules outperform linear ones while bimodal Gaussian schedules become competitive only when derivative weighting is softened or omitted (Kosmopoulou et al., 5 Sep 2025). A separate analysis of standard MDM training attributes slow optimization to locality bias in language, arguing that low-context samples saturate early and high-context samples are redundant; its proposed remedy, bell-shaped time sampling centered on the middle-context region, reaches the same validation NLL up to $\sim4\times$ faster than standard training on LM1B (Hong et al., 13 May 2026). Soft-Masked Diffusion LLMs introduce a different training adaptation: when a token remains masked, the next-step input becomes a convex mixture of the mask embedding and the top-$$q(x_t^i \mid x_{t-1}^i) = \begin{cases} 1 - \beta_t & \text{if } x_t^i = x_{t-1}^i, \ \beta_t & \text{if } x_t^i = \langle mask\rangle, \ 0 & \text{otherwise.} \end{cases}$$0 predicted token embeddings from the previous step, preserving partial predictive information across denoising iterations and improving perplexity and MAUVE after continued pretraining (Hersche et al., 20 Oct 2025).

3. Decoding, generation order, and inference-time control

Inference in MDLMs is often organized around confidence-based remasking. A typical decoder predicts token distributions for the currently masked positions, keeps only the most confident subset, and remasks the rest. In one formulation, token confidence is $$q(x_t^i \mid x_{t-1}^i) = \begin{cases} 1 - \beta_t & \text{if } x_t^i = x_{t-1}^i, \ \beta_t & \text{if } x_t^i = \langle mask\rangle, \ 0 & \text{otherwise.} \end{cases}$$1, and standard decoding selects positions via top-$$q(x_t^i \mid x_{t-1}^i) = \begin{cases} 1 - \beta_t & \text{if } x_t^i = x_{t-1}^i, \ \beta_t & \text{if } x_t^i = \langle mask\rangle, \ 0 & \text{otherwise.} \end{cases}$$2 or threshold rules. This is efficient, but it is also described as greedy, token-local, and noise-sensitive: once locally confident choices are preserved, later steps may have limited ability to recover from them (Luo et al., 2 Feb 2026).

A major research direction therefore replaces or augments token-level confidence with trajectory-level or dependency-aware control. Self-rewarding sequential Monte Carlo launches multiple parallel diffusion trajectories (“particles”), reweights them by the product of accepted-token confidences, and resamples promising trajectories; on OpenWebText-based generation, it improves MDLM generative perplexity from $$q(x_t^i \mid x_{t-1}^i) = \begin{cases} 1 - \beta_t & \text{if } x_t^i = x_{t-1}^i, \ \beta_t & \text{if } x_t^i = \langle mask\rangle, \ 0 & \text{otherwise.} \end{cases}$$3 to $$q(x_t^i \mid x_{t-1}^i) = \begin{cases} 1 - \beta_t & \text{if } x_t^i = x_{t-1}^i, \ \beta_t & \text{if } x_t^i = \langle mask\rangle, \ 0 & \text{otherwise.} \end{cases}$$4 at $$q(x_t^i \mid x_{t-1}^i) = \begin{cases} 1 - \beta_t & \text{if } x_t^i = x_{t-1}^i, \ \beta_t & \text{if } x_t^i = \langle mask\rangle, \ 0 & \text{otherwise.} \end{cases}$$5 and from $$q(x_t^i \mid x_{t-1}^i) = \begin{cases} 1 - \beta_t & \text{if } x_t^i = x_{t-1}^i, \ \beta_t & \text{if } x_t^i = \langle mask\rangle, \ 0 & \text{otherwise.} \end{cases}$$6 to $$q(x_t^i \mid x_{t-1}^i) = \begin{cases} 1 - \beta_t & \text{if } x_t^i = x_{t-1}^i, \ \beta_t & \text{if } x_t^i = \langle mask\rangle, \ 0 & \text{otherwise.} \end{cases}$$7 at $$q(x_t^i \mid x_{t-1}^i) = \begin{cases} 1 - \beta_t & \text{if } x_t^i = x_{t-1}^i, \ \beta_t & \text{if } x_t^i = \langle mask\rangle, \ 0 & \text{otherwise.} \end{cases}$$8, while maintaining high entropy (Luo et al., 2 Feb 2026). The Dependency-Oriented Sampler ranks masked positions by the attention mass they place on already unmasked tokens, using

$$q(x_t^i \mid x_{t-1}^i) = \begin{cases} 1 - \beta_t & \text{if } x_t^i = x_{t-1}^i, \ \beta_t & \text{if } x_t^i = \langle mask\rangle, \ 0 & \text{otherwise.} \end{cases}$$9

and reports large gains on code and math benchmarks, including GSM8K and MATH500, while remaining compatible with EB-sampler acceleration (Zhou et al., 16 Mar 2026).

A second direction treats unmasking order as an explicit policy variable. Unmasking Policy Optimization formulates denoising as a KL-regularized MDP around a reference policy such as max-confidence or top-$t$0, proves reward-improvement and reference-KL tightening results under standard assumptions, and reports consistent gains over heuristic schedules, including $t$1 on Sudoku versus $t$2 for max-confidence (Hong et al., 7 Oct 2025). Order-expressive masked diffusion models generalize this idea by introducing a position-dependent scheduler whose velocity determines denoising order, thereby placing ordinary MDMs, autoregressive models, and block diffusion inside one framework. Its learnable-order instantiation, LoMDM, jointly learns the scheduler and denoising backbone from scratch and reports lower perplexity than earlier diffusion baselines on LM1B and OpenWebText (Hong et al., 2 Feb 2026). A related reinforcement-learning line, MDPO, frames denoising trajectories as sequential decision making under the same progressive remasking schedule used at inference, matches the previous SOTA with $t$3 fewer gradient updates, and combines effectively with Running Confidence Remasking, which revisits tokens according to running rather than one-step confidence (He et al., 18 Aug 2025).

4. Empirical behavior, failure modes, and misconceptions

A common misconception is that bidirectional denoising automatically yields context-uniform reasoning. Recent analysis shows otherwise. "Masks Can Be Distracting" finds that MDLMs retain a strong locality bias: performance is highest when relevant information is close to the mask, and accuracy declines as relevant information moves farther away. The same work shows that appending many masks for generation can degrade comprehension because masks act as distractors rather than neutral placeholders. A mask-agnostic fine-tuning loss combining cross-entropy with a total-variation regularizer reduces relative performance degradation from $t$4 to $t$5 for LLaDA-Base and from $t$6 to $t$7 for LLaDA-Instruct (Piskorz et al., 26 Nov 2025).

Few-step generation is another recurrent weakness. SDDLM explicitly positions MDLMs as competitive with strong autoregressive baselines at scale, but emphasizes that they degrade sharply in the few-step regime because masked-token priors and discrete transitions do not provide the intrinsic noise-to-data mapping used by many continuous-diffusion distillation methods. Uniform-state diffusion is അവതരിപ്പated as one way to make the initial state more analogous to Gaussian noise, yet the paper argues that prior USDM losses were complex and difficult to scale (Zhu et al., 27 Oct 2025).

Trainability depends strongly on how masking interacts with the dependency structure of language. One analysis attributes slow standard MDM training to the locality bias of language and proposes bell-shaped time sampling as a remedy (Hong et al., 13 May 2026). Another studies blockwise MDMs on controlled tasks and concludes that standard random-masking MDMs can fail to learn in-context linear regression reliably, exhibit high-variance dynamics on graph path-finding, yet outperform autoregressive LMs on Sudoku. Its proposed locality-aware blockwise variants, Jigsaw and Scatter, inject left-to-right inductive bias within blocks while preserving iterative refinement across blocks, indicating that random masking may be a suboptimal instantiation of diffusion language modeling for ordered generation (Wang et al., 27 Apr 2026).

5. Specialized adaptations and cross-domain applications

MDLMs have been adapted well beyond English open-domain text. These adaptations preserve the core absorbing-mask denoising paradigm while changing the corpus, backbone, conditioning structure, or output domain.

Domain	Representative system	Reported outcome
Turkish generation	Diffutron (Kocabay et al., 20 Mar 2026)	Perplexity improves from $t$8 to $t$9; the 307M second-stage model reaches an average CETVEL subset score of $p_\theta(x_{t-1}\mid x_t)$0
Sign language generation	MaDiS (Zuo et al., 27 Jan 2026)	Superior DTW-JPE, SiBLEU, and SiCLIP on CSL-Daily, Phoenix-2014T, and How2Sign; inference latency reduced by nearly $p_\theta(x_{t-1}\mid x_t)$1
Membrane protein design	MeMDLM (Goel et al., 2024)	Transmembrane residue frequency $p_\theta(x_{t-1}\mid x_t)$2 versus $p_\theta(x_{t-1}\mid x_t)$3 for ProtGPT2; TM motif scaffolding pseudo-perplexity $p_\theta(x_{t-1}\mid x_t)$4 versus $p_\theta(x_{t-1}\mid x_t)$5 for EvoDiff

Diffutron adapts a multilingual encoder, $p_\theta(x_{t-1}\mid x_t)$6, to Turkish via LoRA-based continual pretraining and a two-stage instruction-tuning pipeline. The model is motivated by Turkish morphology, where broader sentence-level context may be especially useful, and the authors position MDLMs as attractive for compact models because they combine bidirectional context with parallel refinement (Kocabay et al., 20 Mar 2026). MaDiS, in turn, replaces causal attention with non-causal attention in a decoder-style backbone, adds tri-level cross-modal pretraining in token, latent, and 3D physical spaces, and introduces temporal checkpoints that reduce the space of possible unmasking orders by over $p_\theta(x_{t-1}\mid x_t)$7 times for $p_\theta(x_{t-1}\mid x_t)$8 and $p_\theta(x_{t-1}\mid x_t)$9 (Zuo et al., 27 Jan 2026). MeMDLM transfers MDLMs to protein design with an ESM-2-150M denoiser, membrane-protein fine-tuning, and masked inpainting around motifs; it uses the same absorbing-state discrete diffusion logic but with amino-acid tokens rather than words (Goel et al., 2024).

Other applications emphasize low-resource or controlled generation settings. The BabyLM 2025 strict-track system uses masked diffusion as a data-efficient alternative to hybrid AR/MLM objectives, with a 126.6M-parameter LTG-BERT-based model trained for 10 epochs on a 100M-word corpus (Kosmopoulou et al., 5 Sep 2025). For text style transfer, verifier-based inference-time scaling samples multiple reverse candidates at each denoising step and scores them with off-the-shelf sentence embeddings; on WikiLarge and Bible style-transfer benchmarks, MDMs with classifier-free guidance and soft-value diffusion decoding outperform autoregressive baselines reported in the study (Padole et al., 14 Aug 2025).

6. Unlearning, security, and broader implications

The distinct denoising mechanism of MDLMs has prompted MDLM-specific work on model editing and security. Masked Diffusion Unlearning interprets supervised fine-tuning as a shift from a prompt-masked unconditional prediction $x_0$0 toward a prompt-conditional prediction $x_0$1, and proposes to reverse that shift by minimizing a forward KL divergence from the conditional distribution to a temperature-scaled unconditional anchor at each masked position. On TOFU and RWKU, the paper reports that this MDLM-native objective achieves high unlearning performance relative to adapted LLM unlearning baselines while preserving more utility (Lee et al., 18 May 2026).

Security analysis has also identified attack surfaces unique to masked diffusion. ShadowMask modifies the standard all-mask terminal prior into a trigger-mask mixture prior $x_0$2, thereby creating a dedicated denoising route from trigger-corrupted states to attacker-specified targets. The study reports near-100% attack success on DiT-based MDLM and LLaDA-8B-Instruct across WikiText-103, OpenWebText, and Alpaca, substantially outperforming standard data poisoning while largely preserving clean utility and remaining effective under both full-model and parameter-efficient fine-tuning (Cao et al., 19 May 2026).

Taken together, these results suggest that masked diffusion language modeling is best understood not as a single alternative to autoregressive generation, but as a design space in which masking schedule, denoising order, remasking policy, and the semantics of the mask token itself are first-class variables. Recent work increasingly treats those variables as learnable or controllable rather than fixed. A plausible implication is that future progress will depend less on the generic claim that diffusion is bidirectional and more on whether a given MDLM instantiation aligns its corruption process, training distribution, and inference trajectory with the structure of the target task.