Scaling Beyond Masked Diffusion Language Models

Abstract: Diffusion LLMs are a promising alternative to autoregressive models due to their potential for faster generation. Among discrete diffusion approaches, Masked diffusion currently dominates, largely driven by strong perplexity on language modeling benchmarks. In this work, we present the first scaling law study of uniform-state and interpolating discrete diffusion methods. We also show that Masked diffusion models can be made approximately 12% more FLOPs-efficient when trained with a simple cross-entropy objective. We find that perplexity is informative within a diffusion family but can be misleading across families, where models with worse likelihood scaling may be preferable due to faster and more practical sampling, as reflected by the speed-quality Pareto frontier. These results challenge the view that Masked diffusion is categorically the future of diffusion language modeling and that perplexity alone suffices for cross-algorithm comparison. Scaling all methods to 1.7B parameters, we show that uniform-state diffusion remains competitive on likelihood-based benchmarks and outperforms autoregressive and Masked diffusion models on GSM8K, despite worse validation perplexity. We provide the code, model checkpoints, and video tutorials on the project page: http://s-sahoo.github.io/scaling-dllms

• The paper demonstrates that masked diffusion models achieve state-of-the-art perplexity but require 14–16× more compute compared to autoregressive baselines.
• It evaluates uniform-state and interpolating diffusion methods, showing they enable higher throughput and practical tradeoffs despite increased compute costs.
• The study highlights that perplexity alone is insufficient for assessing LLM performance, urging holistic evaluation including sampling efficiency and downstream results.

Scaling Beyond Masked Diffusion LLMs: An In-Depth Analysis

Introduction

"Scaling Beyond Masked Diffusion LLMs" (2602.15014) delivers a comprehensive examination of discrete diffusion LLMs (d-LLMs), contrasting them with traditional autoregressive (AR) models and expanding the discussion beyond the dominant Masked Diffusion paradigm. By systematically evaluating Masked, Uniform-state, and interpolating diffusion methods, the paper rigorously addresses scaling laws, compute efficiency, speed-quality tradeoffs, and downstream performance, particularly in large-scale (up to 1.7B parameter) regimes. The findings contest established assumptions about perplexity and diffusion-model dominance, providing new insights into practical LLM design.

Discrete Diffusion Model Families

Masked Diffusion

Masked Diffusion LLMs (MDLMs) implement a forward process that progressively converts tokens to a special [MASK] state using a bidirectional Transformer. This design achieves state-of-the-art perplexity among diffusion models but precludes efficient inference via KV caching.

Uniform-state Diffusion

Uniform-state diffusion models (USDMs), such as Duo, use a forward process that blends clean data toward a uniform distribution rather than masking. This enables self-correction and efficient few-step sampling in parallel generation but yields poorer perplexity compared to MDLMs.

Interpolating (Eso-LM)

Esoteric LLMs (Eso-LM) interpolate between AR and MDLM, supporting causal attention and KV caching. This sacrifices modeling capacity (worse perplexity in full diffusion mode) for substantially enhanced decoding throughput via support for practical caching strategies.

Scaling Laws and IsoFLOP Analysis

A compute-matched scaling law study was performed for all three diffusion families and the AR baseline, leveraging IsoFLOP methodology.

Figure 2: Validation loss (likelihood) versus FLOPs, comparing scaling behavior of AR, MDLM, Duo (USDM), and Eso-LM.

Main Observations

• Likelihood scaling for MDLM closely tracks AR, but MDLM requires approximately $14$--$16\times$ more compute to match AR's perplexity. Using a low-variance training objective improves MDLM's compute efficiency by $\approx 12\%$ and reduces compute-optimal model sizes.
• Duo (USDM) needs $\approx 23\times$ more compute than AR for comparable perplexity, while Eso-LM requires $\approx 32\times$ more.
• Despite these scaling gaps, all diffusion models yield compute-optimal model sizes roughly $2\times$ smaller than their AR counterparts.

Figure 3: IsoFLOP analysis under fixed compute budgets, comparing AR, MDLM, Duo, and Eso-LM.

Speed-Quality Tradeoff and Sampling Efficiency

Likelihood-based metrics like perplexity fail to capture practical inference-time considerations. The paper analyzes sample throughput and quality using Generative Perplexity (Gen PPL) to construct a speed-quality Pareto frontier.

Figure 1: Speed-Quality Pareto Frontier, reporting highest throughput by compute-optimal models across training FLOP budgets.

• AR yields the highest sample quality but at consistently lowest throughput, due to its sequential decoding.
• Duo dominates at moderate and high throughput regimes ($[200,400]$ and $[600,\infty]$ tokens/sec) by supporting few-step sampling thanks to self-correction.
• Eso-LM is optimal in intermediate throughput ($[400,600]$), leveraging KV caching.
• Sample diversity (measured by entropy) remains largely similar across all algorithms, with slight reductions for Duo.

  Figure 4: Throughput (tokens/sec) vs time discretization $T$ for various diffusion models.

Empirical Results at Scale

The models were scaled to 1.7B parameters and trained on a large standardized corpus (Nemotron-Pre-Training-Dataset, 2.1T tokens) with consistent preprocessing. Evaluation covered both standard likelihood-based benchmarks and mathematical reasoning tasks.

Likelihood-Based Downstream Tasks

• AR remains strongest on commonsense/reading comprehension benchmarks (ARC-e, BoolQ, PIQA, SIQA, OBQA, RACE).
• Among diffusion models, MDLM performs best on ARC-e, BoolQ, SIQA; Duo attains superior results on OBQA, PIQA, and RACE.

Mathematical Reasoning

On GSM8K after supervised fine-tuning:

• Duo (Uniform-state diffusion) outperforms all other methods, including AR and MDLM, on accuracy despite inferior perplexity scaling.
• This finding substantiates the paper's thesis that perplexity is insufficient as a sole criterion for practical model assessment.

Key Claims and Contradictions

• Bold claim: Perplexity is only informative within a diffusion family; cross-family comparisons based on perplexity are frequently misleading.
• Contradiction to prevailing view: Masked diffusion is not categorically superior; Uniform-state and interpolating methods deliver better practical tradeoffs depending on application constraints (e.g. sampling speed, math reasoning).

Implications and Future Prospects

The study compels researchers to reconsider the dominance of Masked diffusion and the over-reliance on perplexity for evaluation. For practical LLM deployments, choosing between AR, Masked, Uniform-state, or interpolating diffusion should depend on downstream objectives:

• Throughput-sensitive scenarios benefit from Uniform-state and interpolating models.
• Tasks requiring maximal sample quality and next-token prediction accuracy still favor AR and Masked models.

Theoretically, the work suggests distinct generative paradigms may yield differentiated emergent capabilities and inference-time efficiencies as scaling continues. Further research at larger scales is needed to clarify if current trends persist, and to fully understand the tradeoffs induced by non-AR decoding strategies.

Conclusion

This comprehensive scaling and comparative study demonstrates that Masked diffusion is not the definitive path for diffusion language modeling. The choice of diffusion mechanism introduces significant constant-factor compute effects and sampling behaviors, uncaptured by perplexity alone. Uniform-state and interpolating diffusion models present concrete benefits for sample efficiency and practical inference. Evaluation protocols for d-LLMs should thus holistically account for both likelihood-based and sampling-centric metrics, guiding future architecture and deployment decisions.