Generalized Discrete Diffusion from Snapshots

Abstract: We introduce Generalized Discrete Diffusion from Snapshots (GDDS), a unified framework for discrete diffusion modeling that supports arbitrary noising processes over large discrete state spaces. Our formulation encompasses all existing discrete diffusion approaches, while allowing significantly greater flexibility in the choice of corruption dynamics. The forward noising process relies on uniformization and enables fast arbitrary corruption. For the reverse process, we derive a simple evidence lower bound (ELBO) based on snapshot latents, instead of the entire noising path, that allows efficient training of standard generative modeling architectures with clear probabilistic interpretation. Our experiments on large-vocabulary discrete generation tasks suggest that the proposed framework outperforms existing discrete diffusion methods in terms of training efficiency and generation quality, and beats autoregressive models for the first time at this scale. We provide the code along with a blog post on the project page : \href{https://oussamazekri.fr/gdds}{https://oussamazekri.fr/gdds}.

• The paper introduces a novel CTMC-based framework that enables flexible, semantically informed noising schemes for large discrete state spaces.
• The paper employs a snapshot-centric ELBO loss, achieving superior language modeling performance with lower perplexity compared to AR and baseline models.
• The paper demonstrates that the GDDS approach enhances training stability and scalability while setting new benchmarks in discrete diffusion for generative tasks.

Generalized Discrete Diffusion from Snapshots: A Unified Framework for Discrete Diffusion Modeling

Introduction

"Generalized Discrete Diffusion from Snapshots" (GDDS) (2603.21342) introduces a principled and computationally efficient approach for discrete diffusion generative modeling. The GDDS framework supports arbitrary noising schemes over large discrete state spaces, encompassing existing discrete diffusion paradigms and substantially extending their flexibility. This work addresses two core limitations in prior dLLMs: rigid and semantically blind corruption mechanics, and loss formulation mismatches that complicate training at scale. GDDS advances the field through a mathematically general interpolating diffusion process, Poisson-based uniformization for efficient corruption dynamics, and a coherent snapshot-level ELBO that aligns training with standard deep generative architectures.

Figure 1: GDDS overview; a clean sequence $\mathbf{x}_0$ is corrupted by the CTMC at randomly sampled time $t$, yielding the snapshot $(\mathbf{x}_t, t)$, from which a denoiser network directly predicts the clean-token posterior.

Generalized Interpolating Discrete Diffusion

GDDS defines the forward corruption as an arbitrary continuous-time Markov chain (CTMC), governed by a time-dependent rate matrix $Q_t$. This covers all previously explored discrete diffusion models (uniform, masked, etc.) as special cases, and enables the incorporation of new structure-aware noising processes. The Kolmogorov forward equation is solved via a mixing matrix $\Pi_t$ and mixing rate $\alpha_t$, yielding:

$K_t = \alpha_t I_m + (1-\alpha_t)\Pi_t$

This interpolating matrix admits closed-form solutions for standard processes, and, per Proposition 1, can represent any CTMC rate matrix via $\Pi_t$. Therefore, GDDS provides a mathematical framework for flexible and semantically meaningful corruption beyond uniform/mask paradigms.

Efficient Noising via Uniformization

Simulating the noised marginals $q_t(\cdot | x_0)$ is typically intractable for large vocabularies. GDDS implements exact noising by uniformization: a Poisson jump process traverses the CTMC with a time-dependent jump kernel $F_t$. Sampling requires only column access to $F_t$, sidestepping the need to compute matrix exponentials. For a sequence, token-wise corruption is parallelized, enabling scalability for $m \sim 50$k (e.g., GPT-2). This process generalizes to any structured kernel, including semantic-informed kernels (SIK), where transitions depend on token embedding similarity.

Reverse Process, Loss Alignment, and ELBO

A central contribution is the snapshot-centric loss formulation. Prior mean-parametrization approaches conflate jump times and destinations in the reverse chain, causing information-misalignment between the generative model and the optimization objective. GDDS disentangles these via a jump-states parametrization: the neural network $j_\theta$ learns the jump destination kernel, while exit-rate schedules are fixed. The resulting path-wise ELBO is a clean, weighted cross-entropy that directly matches the neural prediction to the ideal kernel.

However, GDDS recognizes that powerful architectures empirically optimize snapshot-level denoising, not full path-wise objectives. This motivates a snapshot latent variational formulation: the ELBO is computed on pairs $(x_t, t)$, requiring only sampled snapshots. This loss is computationally tractable, aligns with standard transformer models, and reflects a principled tradeoff between information loss (from discarding trajectory $\omega$) and optimization tractability/calibration.

Figure 2: Comparison of snapshot vs. path-wise training, illustrating GDDS’s focus on snapshot sampling versus full trajectory conditioning.

Experimental Evaluation

GDDS is benchmarked on character-level (Text8) and large-scale BPE-tokenized (OpenWebText) tasks. All models use comparable transformer backbones and matched compute. GDDS variants (Absorb, Uniform, Gauss) are compared to strong AR and prior discrete diffusion baselines.

Figure 3: Zero-shot transfer performance: GDDS Gauss achieves the lowest perplexity across datasets, indicating superior generalization.

Figure 4: OWT training curves: GDDS models exhibit tighter perplexity bounds and faster convergence trajectories than AR and baseline discrete diffusion models.

Results demonstrate several critical findings:

• Snapshot ELBO outperforms path-wise objectives: Despite theoretical information loss, calibration improvements result in lower expected NLL, sometimes surpassing AR models in validation perplexity.
• GDDS Absorb achieves BPC $<$ 1.16 on Text8, outperforming retrained AR (1.35) and prior discrete diffusion baselines.
• On OWT, GDDS Gauss attains ELBO-perplexity $\leq 7.65$ compared to AR ($20.49$), MDM ($31.03$), and UDLM ($36.82$).
• Zero-shot transfer: GDDS Gauss dramatically reduces downstream validation perplexity (e.g., PTB, LM1B, Wikitext), indicating improved generalization from semantic kernel corruption mechanics.

Semantic-Informed Kernels and Language Structure

The GDDS framework facilitates semantic-informed corruption dynamics via scalable kernels (e.g., Gaussian SIK). By leveraging token embedding distances, GDDS Gauss structures the forward CTMC to propose semantically proximal corruptions. This both aligns the noising process with natural linguistic structure and empirically yields significant gains in denoising and transfer performance.

Language Generation: Quality and Diversity

GDDS models achieve superior tradeoffs in generative perplexity and sequence entropy (quality-diversity frontier), outperforming baselines in both metrics, especially under limited decoding budgets.

Figure 5: Gen-PPL vs. entropy: GDDS Absorb and GDDS Uniform realize strictly higher entropy and lower Gen-PPL relative to MDM and UDLM baselines.

Lexical diversity is also improved, with GDDS Uniform matching or exceeding masked methods in Distinct-$n$ statistics.

Training Stability and Architectural Implications

Training loss stability is analyzed, with snapshot-based GDDS losses exhibiting higher short-term fluctuations relative to path-wise Campbell objectives, but ultimately yielding stronger empirical performance.

Figure 6: Training loss curves on Text8: Campbell two-stream architecture exhibits smooth optimization relative to snapshot-based losses.

Theoretical and Practical Implications

GDDS enables discrete diffusion models to scale flexibly to large vocabularies and complex data modalities. The snapshot ELBO provides a well-aligned loss for both theoretical optimization and practical architecture compatibility. The empirical demonstration that discrete diffusion models can surpass AR models at scale on language tasks undermines the long-standing view that AR approaches are inherently dominant, providing a foundation for broader adoption of diffusion paradigms. The ability to structure corruption dynamics based on semantic relationships is a clear theoretical and practical advantage.

Conclusion

GDDS constitutes a mathematically unified, implementation-efficient, and empirically validated framework for discrete diffusion modeling. By enabling arbitrary CTMC-based corruption with semantically meaningful kernels and aligning loss functions with snapshot-level prediction, GDDS both broadens the design space for dLLMs and establishes state-of-the-art performance on language modeling and generation benchmarks. The approach provides a promising avenue for future research, particularly in designing new semantic corruption kernels and adapting snapshot-based diffusion losses for other discrete modalities. Theoretical foundations and empirical results suggest GDDS will be influential in motivating next-generation diffusion generative architectures for discrete data.