Masked Diffusion Models (MDMs) Overview

• Masked Diffusion Models (MDMs) are generative models that iteratively denoise data using random masking to recover partially observed inputs.
• They employ flexible architectures—encoder-only, decoder-only, and hybrid—to enable any-order token unmasking and achieve substantial speedups in inference.
• MDMs power applications in image synthesis, language modeling, and molecular design, delivering state-of-the-art performance across diverse domains.

Masked Diffusion Models (MDMs) are a family of generative models operating on discrete or continuous data, characterized by their iterative denoising process that leverages masking—a binary or partially observed corruption—to produce samples in a non-sequential, potentially any-order fashion. MDMs generalize classical diffusion models by substituting additive noise with masking, thereby enabling flexible training, efficient sampling, and improved adaptability across a broad range of tasks including language, vision, molecular generation, and multimodal synthesis.

1. Core Principles and Formulation

MDMs implement a generative process by corrupting input data via randomly masking a subset of tokens or regions (rather than imposing additive Gaussian noise, as in classic continuous diffusion). Given an initial data point $x_0$, a forward Markov process stochastically masks elements over time; the reverse process aims to recover the original input from the corrupted, partially observed version. The central objectives are defined in terms of evidence lower bounds (ELBOs) or corresponding variational bounds, which can be written for a discrete sequence as:

$$- \log p_\theta(x_0) \le \int_{0}^{1} \frac{1}{t} \mathbb{E}_{q(x_t | x_0)} \left[ \sum_{i:x_t^i = m} -\log p_\theta(x_0^i | x_t) \right] dt$$

where $m$ denotes the mask token, $x_t$ is the corrupted version at time $t$, and $p_\theta$ is the learned denoising model (often a Transformer-based architecture). Notably, MDMs can operate in either encoder-only or decoder-only configurations and can be interpreted as an “any-order autoregressive” model, where unmasking can occur in arbitrary orders (Xue et al., 24 Jun 2025). The formulation is rigorously connected to masked LLMs and order-agnostic generation paradigms.

2. Architectural Variants and Training Schemes

MDMs employ various architectural adaptations to maximize flexibility and efficiency on different data modalities:

• Encoder-Only MDMs: These utilize encoder-based backbones (e.g., ViT for vision, BERT for language), often focusing on denoising via bidirectional context. The conditional probability modeled is order-invariant on the context set, favoring simpler density estimation but incurring higher computational costs—often $O(n^2)$ for sequence length $n$—especially for iterative sampling (Xue et al., 24 Jun 2025).
• Decoder-Only MDMs: Recent work has adapted MDMs to decoder-only architectures (as in GPT), supporting fast, cacheable (KV cache) parallel or any-order sampling. The conditioning here is order-dependent, and principled objective design (e.g., including left-to-right supervision alongside any-order training) yields faster convergence and strong perplexity (Xue et al., 24 Jun 2025).
• Partially Masked and Hybrid Models: Extensions such as Prime (Chao et al., 24 May 2025) introduce partial masking, decomposing tokens into sub-tokens to allow for intermediate, partially unmasked states. Hybrid models (e.g., Eso-LMs (Sahoo et al., 2 Jun 2025)) couple an initial MDM-style parallel denoising phase with a sequential (autoregressive) pass, leveraging both bidirectional and causal attention and enabling KV-cache in MDMs for the first time.
• Masking Strategies: In vision (Wei et al., 2023), elements such as spatial/image patches or motion embedding tokens may be masked; in text, tokens are masked in arbitrary orders; in molecular settings, masking granularity extends to atom- and bond-level trajectories with learnable rates (Seo et al., 22 May 2025). Adaptive scheduling, context-aware masking, and bidirectional processing are commonly employed.

3. Inference and Sampling Strategies

MDMs permit considerable flexibility in the order and method of token unmasking during inference, with several advances aimed at boosting generative quality and efficiency:

• Any-Order Sampling: Tokens can be unmasked in parallel or sequentially, with the order either fixed, random, or adaptively chosen based on model confidence or entropy. Adaptive planning strategies, such as “Path Planning (P2)” (Peng et al., 5 Feb 2025), decouple planning and denoising stages, and allow iterative refinement and remasking—addressing the historical inability of standard MDMs to revisit unmasked tokens.
• Entropy-Bounded and Adaptive Unmasking: EB-Sampler (Ben-Hamu et al., 30 May 2025) dynamically unmasks subsets of tokens with low model entropy in each sampling step, providing 2–3× acceleration without sacrificing quality on code and math reasoning benchmarks.
• Time-Agnostic and Efficient Sampling: Analyses reveal that the time variable in MDMs is often superfluous; with the “first-hitting sampler (FHS),” token unmasking is analytically scheduled based on the unmasking event distribution, yielding significant speedups (Zheng et al., 4 Sep 2024). In language, decoder-only architectures with hybrid scheduling and KV-cache support achieve 25–65× inference speed improvements over encoder-only MDMs for long sequences (Sahoo et al., 2 Jun 2025, Xue et al., 24 Jun 2025).
• Theoretical Speed and Accuracy Limits: The number of required sampling steps for globally correct sequences (SER) must scale at least linearly with sequence length, neutralizing efficiency gains over AR models in exact-sequence settings, although token-error (perplexity) can be competitively achieved with constant (sequence-length independent) steps (Feng et al., 13 Feb 2025).

4. Applications and Empirical Performance

MDMs have demonstrated competitive or superior results across various domains:

• Image and Video Generation: Masked training accelerates model convergence, yielding FIDs of 2.28 on ImageNet-256x256 with 31% of the training resources of baseline DiT models (Zheng et al., 2023). The pre-training plus fine-tuning schemes allow rapid adaptation and state-of-the-art performance on tasks requiring inpainting, style transfer, and cross-modal synthesis (Lei et al., 2023, Wei et al., 2023).
• LLMing and Reasoning: MDMs are competitive with ARMs in zero-shot benchmarks, and excel on tasks requiring bidirectional reasoning, reverse inference, or infilling—all enabled by their non-causal factorization (Nie et al., 24 Oct 2024). They outperform much larger ARMs on temporally shifted data, break the reverse curse, and are robust in math reasoning tasks (Nie et al., 24 Oct 2024, Peng et al., 5 Feb 2025).
• Molecular and Structural Data: Element-wise noise scheduling (MELD) in molecular graphs prevents state-clashing and dramatically increases validity (from 15% to 93% on ZINC250K) and property alignment (Seo et al., 22 May 2025). In protein and RNA sequence design, MDMs achieve state-of-the-art foldability and structural accuracy (Peng et al., 5 Feb 2025).
• Representation Learning and Label-Efficient Learning: Masked diffusion approaches surpass both classical denoising and MAE baselines in semantic segmentation, particularly in few-shot and medical imaging contexts (Pan et al., 2023).
• One-Step and Accelerated Generation: Di$\mathtt{[M]}$O provides near-teacher-level image and text-conditioned generation with a single forward pass by token-level distribution matching and strategic initialization (Zhu et al., 19 Mar 2025). Eso-LMs realize inference speed gains of up to 65× over traditional MDMs (Sahoo et al., 2 Jun 2025).

5. Theoretical Insights and Training-Generation Trade-offs

Research has clarified important theoretical properties and limitations of MDMs:

• Schedule Conditioning and Markovian Analysis: Masking diffusion leverages the fact that, for discrete Markov processes, the jump (masking) event schedule is known; conditioning the backward process on the forward jump schedule (SCUD) separates learning “when” from “where,” improving fit and allowing injection of domain-specific inductive biases (e.g., Gaussian or BLOSUM noise) for improved performance on images, text, and proteins (Amin et al., 10 Jun 2025).
• Inductive Bias via Order Distribution: Uniform averaging over all unmasking (token) orders leads to slower convergence and poorer practicality than privileging the inherent left-to-right structure of language. Incorporating even 10% of left-to-right data in training yields improved perplexity and convergence for AO-AR/MDM models (Xue et al., 24 Jun 2025).
• Variational and Latent Extensions: Introducing continuous latent variables into the reverse process (e.g., via VADD) enables efficient capture of inter-dimensional correlations, improving sample quality, especially when only a few denoising steps are permitted (Xie et al., 23 May 2025).
• Limitations of Parallel Generation: Although MDMs offer sequence-length-independent parallelism for token-level fluency (as measured by perplexity), for sequence-level accuracy (SER) in tasks demanding total correctness (e.g., chain-of-thought reasoning), the number of steps and thus cost reapproaches or exceeds that of AR models (Feng et al., 13 Feb 2025).

6. Alignment, Steering, and Preference Optimization

MDMs can be aligned and steered with advanced learning techniques:

• Preference Optimization: Variance-Reduced Preference Optimization (VRPO) (Zhu et al., 25 May 2025) addresses high ELBO estimation variance in RL-alignment of MDMs, achieving substantial and stable alignment improvements in large language instruction models such as LLaDA 1.5 across math, code, and alignment metrics.
• Discrete Denoising Posterior Prediction: DDPP offers a general, simulation-free framework for property-controlled generation and RLHF-based alignment of MDMs, with empirical successes in images, text, and protein design (Rector-Brooks et al., 10 Oct 2024).
• Generalizability and Transfer: Pre-training with masking strategies offers robust representations that adapt efficiently across domains (e.g., transferring a VGGFace2-trained MDM to 3,000-sample target datasets yields up to 46% quality improvement) (Lei et al., 2023).

7. Future Directions and Open Issues

The field is actively developing along several axes:

• Objective and Order Refinement: Moving beyond uniform order-agnostic objectives to hybrid or adaptive order distributions, ideally informed by the intrinsic structure of language or data, is shown to be advantageous (Xue et al., 24 Jun 2025, Kim et al., 10 Feb 2025).
• Domain-Specific Diffusion Schedules: Flexible, learnable and element-specific forward processes (as in MELD (Seo et al., 22 May 2025)) and schedule-conditioned models (as in SCUD (Amin et al., 10 Jun 2025)) suggest greater potential for modeling complex, structured discrete domains.
• Efficient Sampling and Token Revisit: Combining fast parallel unmasking (e.g., EB-Sampler (Ben-Hamu et al., 30 May 2025)) with iterative refinement (remasking and planning) broadens the deployment possibilities of MDMs in latency-sensitive and sequence-accuracy-critical applications.
• Scalability and Hybrid Paradigms: Eso-LMs demonstrate that fusion of AR and MDM principles, along with careful architectural and loss design, can achieve both high quality and practical inference speeds in long-context language tasks (Sahoo et al., 2 Jun 2025).
• RLHF and Policy Optimization: Principled variance reduction for ELBO-based RL optimization (as in VRPO (Zhu et al., 25 May 2025)) and posterior-guided methods (DDPP (Rector-Brooks et al., 10 Oct 2024)) enhance the value of MDMs in preference alignment and controlled generation tasks.

In sum, Masked Diffusion Models constitute a versatile and theoretically principled framework for data generation and representation in both discrete and continuous settings. By combining flexible training, bidirectional generative reasoning, and a growing suite of efficient inference techniques, MDMs have achieved state-of-the-art outcomes across vision, language, molecular design, and beyond, and continue to inspire ongoing research on generative modeling’s efficiency and expressiveness frontier.