Generalized Discrete Diffusion from Snapshots

• GDDS is a unified CTMC framework for discrete diffusion that leverages snapshot-based evidence to model arbitrary noising processes in large-vocabulary tasks.
• It employs an innovative interpolating representation combining identity and mixing matrices to generalize masked, uniform, and rank-one diffusion models.
• GDDS uses uniformization for efficient sampling and a snapshot ELBO training objective to achieve competitive performance on language modeling benchmarks.

Generalized Discrete Diffusion from Snapshots (GDDS) is a unified framework for discrete diffusion modeling in which the forward corruption process is a continuous-time Markov chain (CTMC) on a discrete state space, the noising dynamics can be arbitrary rather than restricted to masking or uniform replacement, and training is carried out with a snapshot-based evidence lower bound (ELBO) that uses single noised observations $(x_t,t)$ rather than entire noising paths (Zekri et al., 22 Mar 2026). In the formulation introduced in 2026, GDDS is designed for large-vocabulary discrete generation and explicitly subsumes masked diffusion, uniform diffusion, rank-one interpolating schemes, and other CTMC-based discrete diffusion constructions (Zekri et al., 22 Mar 2026). In a broader conceptual sense, the phrase also aligns with earlier snapshot-driven reconstruction of stochastic dynamics: a continuous-time SDE inferred from snapshot distributions can be discretized into a discrete-time Markov diffusion, which suggests a continuous-to-discrete interpretation of “diffusion from snapshots” beyond the token-based CTMC setting (Zhu et al., 2024).

1. Definition and scope

In GDDS, the single-token state space is $V=\{v_1,\dots,v_m\}$, and the forward noising process is a possibly time-inhomogeneous CTMC with rate matrix $Q_t\in\mathbb{R}^{m\times m}$ (Zekri et al., 22 Mar 2026). The forward transition operator $K_t$ solves

$\frac{d K_t}{d t} = Q_t K_t,\quad K_0 = I_m,$

with

$K_t = T\exp\left(\int_0^t Q_s\,ds\right),$

where $T$ is the time-ordering operator (Zekri et al., 22 Mar 2026). Given a clean token $x_0$, the conditional marginal at time $t$ is

$q_t(x_t\mid x_0) = K_t(x_t,x_0).$

A central structural result is the interpolating representation

$V=\{v_1,\dots,v_m\}$0

where $V=\{v_1,\dots,v_m\}$1 is a smooth decreasing mixing rate with $V=\{v_1,\dots,v_m\}$2 and $V=\{v_1,\dots,v_m\}$3, and $V=\{v_1,\dots,v_m\}$4 is a smooth column-stochastic mixing matrix (Zekri et al., 22 Mar 2026). This representation contains several established discrete diffusion families as special cases.

Special case	Choice inside GDDS	Consequence
Masked diffusion	$V=\{v_1,\dots,v_m\}$5	Absorbing-mask corruption
Uniform diffusion	$V=\{v_1,\dots,v_m\}$6	Uniform token corruption
GIDD	$V=\{v_1,\dots,v_m\}$7	Rank-one interpolating scheme

The 2026 GDDS paper states that this construction “encompasses all existing discrete diffusion approaches” while permitting “arbitrary Markovian noising processes” (Zekri et al., 22 Mar 2026). A closely related predecessor, Generalized Interpolating Discrete Diffusion (GIDD), already generalized masked diffusion by replacing the mask-only mixing distribution with a time-varying $V=\{v_1,\dots,v_m\}$8, yielding marginals of the form

$V=\{v_1,\dots,v_m\}$9

but GDDS removes the rank-one restriction and makes the forward process fully CTMC-general (Rütte et al., 6 Mar 2025).

The emphasis on “snapshots” distinguishes GDDS from path-centric formulations. Instead of requiring access to or explicit optimization over entire forward trajectories, GDDS uses single noised observations $Q_t\in\mathbb{R}^{m\times m}$0 as latent variables in its variational objective (Zekri et al., 22 Mar 2026). This places it in the same broad methodological family as snapshot-based continuous-time dynamics recovery, where only marginal distributions at times $Q_t\in\mathbb{R}^{m\times m}$1 are observed and governing stochastic laws are reconstructed from them (Zhu et al., 2024).

2. Forward process and arbitrary corruption dynamics

The forward generator is factorized as

$Q_t\in\mathbb{R}^{m\times m}$2

where $Q_t\in\mathbb{R}^{m\times m}$3 is the exit rate from state $Q_t\in\mathbb{R}^{m\times m}$4, and $Q_t\in\mathbb{R}^{m\times m}$5 is a column-stochastic matrix describing jump destinations conditional on leaving (Zekri et al., 22 Mar 2026). In the shared-exit-rate case emphasized for computation,

$Q_t\in\mathbb{R}^{m\times m}$6

with scalar rate $Q_t\in\mathbb{R}^{m\times m}$7 and column-stochastic jump kernel $Q_t\in\mathbb{R}^{m\times m}$8 (Zekri et al., 22 Mar 2026).

Two propositions define the scope of the framework. First, if $Q_t\in\mathbb{R}^{m\times m}$9 is differentiable and invertible, then

$K_t$0

Second, given any rate matrix $K_t$1 and any mixing rate $K_t$2 with $K_t$3, there exists a unique column-stochastic $K_t$4 such that

$K_t$5

(Zekri et al., 22 Mar 2026). This establishes that the interpolating form is not a restricted model class but an alternative parameterization of general Markovian noising.

The underlying CTMC perspective links GDDS to earlier theoretical work on discrete diffusion. Uniformization-based exact implementation of reverse CTMCs on discrete spaces had already been analyzed for hypercube-valued diffusion models, where forward and reverse processes are continuous-time Markov chains and pathwise error bounds can be derived without time-discretization error (Chen et al., 2024). Likewise, exact reverse-time generators for arbitrary discrete-state Markov processes were previously derived in the “Blackout Diffusion” framework, which emphasized that reverse-time discrete diffusion can be formulated exactly rather than only through variational approximations (Santos et al., 2023). GDDS inherits this CTMC lineage but repurposes it for large-vocabulary generative modeling with snapshot-based learning (Zekri et al., 22 Mar 2026).

A further conceptual extension appears in continuous-state snapshot identification. In Sparse Identification of Differential Equations from Snapshots (SpIDES), the learned Itô SDE

$K_t$6

induces, after Euler–Maruyama discretization,

$K_t$7

which yields a discrete-time Markov diffusion derived entirely from snapshot distributions (Zhu et al., 2024). This suggests an abstract relation: GDDS in the token-CTMC sense and SpIDES in the SDE sense both build discrete diffusion dynamics from snapshot information, but they do so on different state spaces and with different inference machinery.

3. Uniformization and efficient snapshot noising

The principal computational obstacle for arbitrary discrete diffusion is that exact evaluation of $K_t$8 by matrix exponentials or time-ordered exponentials is intractable when $K_t$9 is large (Zekri et al., 22 Mar 2026). GDDS addresses this with uniformization. For the shared-rate generator

$\frac{d K_t}{d t} = Q_t K_t,\quad K_0 = I_m,$0

define

$\frac{d K_t}{d t} = Q_t K_t,\quad K_0 = I_m,$1

Let $\frac{d K_t}{d t} = Q_t K_t,\quad K_0 = I_m,$2 be a non-homogeneous Poisson process with intensity $\frac{d K_t}{d t} = Q_t K_t,\quad K_0 = I_m,$3, with ordered jump times $\frac{d K_t}{d t} = Q_t K_t,\quad K_0 = I_m,$4. Then the mixing matrix satisfies

$\frac{d K_t}{d t} = Q_t K_t,\quad K_0 = I_m,$5

(Zekri et al., 22 Mar 2026). As a consequence, sampling $\frac{d K_t}{d t} = Q_t K_t,\quad K_0 = I_m,$6 requires neither explicit $\frac{d K_t}{d t} = Q_t K_t,\quad K_0 = I_m,$7 nor explicit $\frac{d K_t}{d t} = Q_t K_t,\quad K_0 = I_m,$8.

The resulting token-level noising algorithm is exact: sample $\frac{d K_t}{d t} = Q_t K_t,\quad K_0 = I_m,$9, sample and sort the jump times, initialize $K_t = T\exp\left(\int_0^t Q_s\,ds\right),$0, and iterate

$K_t = T\exp\left(\int_0^t Q_s\,ds\right),$1

returning $K_t = T\exp\left(\int_0^t Q_s\,ds\right),$2 (Zekri et al., 22 Mar 2026). Sequence-level noising runs this procedure independently across positions (Zekri et al., 22 Mar 2026).

This construction is the mechanism behind the paper’s claim that the forward noising process “relies on uniformization and enables fast arbitrary corruption” (Zekri et al., 22 Mar 2026). In practice, it reduces forward corruption to repeated column access in $K_t = T\exp\left(\int_0^t Q_s\,ds\right),$3, so the cost scales with the expected number of jumps and the cost of sampling from a column of $K_t = T\exp\left(\int_0^t Q_s\,ds\right),$4, not with dense $K_t = T\exp\left(\int_0^t Q_s\,ds\right),$5 matrix algebra (Zekri et al., 22 Mar 2026). The same uniformization principle had already been used in CTMC-based discrete diffusion theory to provide exact trajectory simulation through Poisson event times and discrete jump kernels (Chen et al., 2024), but GDDS adapts it to high-cardinality vocabularies and snapshot-driven training (Zekri et al., 22 Mar 2026).

A distinct discrete snapshot paradigm appears in the Glauber Generative Model (GGM), where the forward process is a discrete-time single-site Markov chain and the reverse kernel is identified from forward snapshots through binary classification of “signal vs noise” events (Varma et al., 2024). GGM also learns from snapshots rather than reverse trajectories, but it assumes a known corruption family with sitewise updates and uses a classification reduction, whereas GDDS uses a CTMC, uniformization, and a variational snapshot latent (Varma et al., 2024).

4. Snapshot latents, reverse parameterization, and the ELBO

GDDS separates two modeling problems that earlier discrete diffusion formulations often conflated: learning a reverse-time CTMC over paths, and learning a predictor of the clean token from a single noised observation (Zekri et al., 22 Mar 2026). The paper argues that the usual “mean parametrization,” in which a network $K_t = T\exp\left(\int_0^t Q_s\,ds\right),$6 predicts $K_t = T\exp\left(\int_0^t Q_s\,ds\right),$7 and is then plugged into a reverse kernel, entangles “where” and “when” jumps happen in the reverse process (Zekri et al., 22 Mar 2026).

For the reverse CTMC, GDDS introduces a jump-states parametrization

$K_t = T\exp\left(\int_0^t Q_s\,ds\right),$8

where the exit rates $K_t = T\exp\left(\int_0^t Q_s\,ds\right),$9 are fixed independently of $T$0, and the learned column-stochastic matrix $T$1 is produced by a network $T$2 (Zekri et al., 22 Mar 2026). Under this parameterization, the path-wise ELBO simplifies to a weighted cross-entropy between the true reverse jump kernel and the learned one. Using Campbell’s formula, it can be rewritten as

$T$3

where $T$4 is the full forward path (Zekri et al., 22 Mar 2026).

The framework’s distinctive move is to abandon full paths as training latents in favor of single snapshots

$T$5

with variational distribution

$T$6

and, in the paper, $T$7 (Zekri et al., 22 Mar 2026). The snapshot generative model is defined through the mean network

$T$8

This yields the snapshot ELBO

$T$9

which is the training objective used in practice (Zekri et al., 22 Mar 2026).

The paper further decomposes the NLL difference between snapshot and path latents as

$x_0$0

where the information path gap reflects information discarded by using snapshots, while the calibration gap measures conditional calibration quality (Zekri et al., 22 Mar 2026). The result that minimizing the snapshot loss is equivalent to minimizing $x_0$1, whereas minimizing the path-wise loss is not generally equivalent to minimizing $x_0$2, is used to justify the empirical preference for snapshot latents (Zekri et al., 22 Mar 2026). This suggests that snapshots are not merely a computational shortcut but a statistically better latent variable for mean-network training.

The same reliance on single time-indexed marginals rather than trajectories also appears in continuous-time dynamics discovery from snapshots. SpIDES reconstructs a probability flow field and a score function from snapshot marginals and then identifies sparse drift and diffusion terms, rather than regressing on observed paths (Zhu et al., 2024). The methodological overlap is not identity, but both frameworks replace inaccessible trajectories with time-conditioned snapshot statistics.

5. Relation to masked, uniform, and interpolating discrete diffusion

GDDS is explicitly presented as a unifying framework. Masked diffusion, uniform diffusion, GIDD, and other discrete diffusion constructions are all obtained by particular choices of $x_0$3, $x_0$4, or both (Zekri et al., 22 Mar 2026).

Masked or absorbing diffusion corresponds to $x_0$5, and its continuous-time variational objective had already been shown to reduce to a weighted integral of cross-entropy losses over masked tokens (Shi et al., 2024). In GIDD, the same masked process appears when the mixing distribution is fixed to the mask token, and the GIDD ELBO provably reduces to the masked diffusion ELBO (Rütte et al., 6 Mar 2025). GDDS retains this special case but situates it inside a larger CTMC family (Zekri et al., 22 Mar 2026).

Uniform diffusion corresponds to $x_0$6, again recovered as a special case of GDDS (Zekri et al., 22 Mar 2026). GIDD also supports uniform or hybrid noise by choosing the mixing distribution $x_0$7 appropriately, but because GIDD restricts to rank-one $x_0$8, it does not cover the full class of state-dependent or semantically structured kernels allowed by GDDS (Rütte et al., 6 Mar 2025).

The relation between the two frameworks can be summarized succinctly. GIDD generalizes masked diffusion by replacing the mask-only mixing distribution with an arbitrary $x_0$9 and derives a CTMC ELBO for this interpolating family (Rütte et al., 6 Mar 2025). GDDS proves that any rate matrix $t$0 can be written in interpolating form with an appropriate $t$1, and therefore lifts the generalization from rank-one interpolations to arbitrary CTMC noising processes (Zekri et al., 22 Mar 2026).

This broader scope matters because the choice of corruption process affects not only likelihood optimization but also qualitative model behavior. GIDD used hybrid mask-plus-uniform noise to reintroduce the ability to revise already generated tokens, which masked diffusion lacks because once a position is filled it is fixed in the reverse process (Rütte et al., 6 Mar 2025). GDDS extends the design space further: semantic or graph-structured kernels can be used to define nontrivial jump geometries over the vocabulary, rather than only mask-based or uniform replacements (Zekri et al., 22 Mar 2026). A plausible implication is that GIDD should be viewed as a specific low-rank corridor inside the larger GDDS design space.

6. Semantic-informed kernels, inference, and empirical behavior

The most distinctive application-specific extension in the GDDS paper is the use of Semantic-Informed Kernels (SIKs) (Zekri et al., 22 Mar 2026). In the Gaussian SIK variant,

$t$2

with $t$3 increasing in time, so early corruption is local in embedding space and later corruption becomes more uniform (Zekri et al., 22 Mar 2026). The paper reports both dense implementations using KeOps and approximate implementations using $t$4-nearest neighbors with $t$5 neighbors per token (Zekri et al., 22 Mar 2026).

At inference time, GDDS uses a discretized ancestral sampler. Starting from $t$6, it steps backward across a schedule $t$7 using a plug-in Bayes reverse kernel built from $t$8 and the forward CTMC (Zekri et al., 22 Mar 2026). This is computationally straightforward for absorb and uniform processes, and heavier for SIK because forward operators must be approximated during reverse sampling (Zekri et al., 22 Mar 2026).

The empirical evaluation is on Text8 and OpenWebText (OWT) (Zekri et al., 22 Mar 2026). On Text8, GDDS Absorb achieves a reported upper bound of $t$9 bits per character, outperforming a retrained autoregressive baseline at approximately $q_t(x_t\mid x_0) = K_t(x_t,x_0).$0, a retrained MDM baseline at $q_t(x_t\mid x_0) = K_t(x_t,x_0).$1, and a retrained UDLM baseline at $q_t(x_t\mid x_0) = K_t(x_t,x_0).$2 (Zekri et al., 22 Mar 2026). On OWT validation perplexity, the reported upper bounds are $q_t(x_t\mid x_0) = K_t(x_t,x_0).$3 for GDDS Uniform, $q_t(x_t\mid x_0) = K_t(x_t,x_0).$4 for GDDS Absorb, and $q_t(x_t\mid x_0) = K_t(x_t,x_0).$5 for GDDS Gauss, compared with $q_t(x_t\mid x_0) = K_t(x_t,x_0).$6 for retrained MDM, $q_t(x_t\mid x_0) = K_t(x_t,x_0).$7 for retrained UDLM, and $q_t(x_t\mid x_0) = K_t(x_t,x_0).$8 for the retrained autoregressive baseline under the paper’s matched-compute protocol (Zekri et al., 22 Mar 2026).

The paper also reports zero-shot perplexity across seven downstream datasets and states that GDDS Gauss achieves the best zero-shot perplexity on all seven datasets, which it attributes to semantically structured noising (Zekri et al., 22 Mar 2026). In unconditional generation from OWT, GDDS Uniform and GDDS Absorb improve the quality-diversity frontier relative to retrained uniform and masked diffusion baselines, and GDDS Absorb reaches similar entropy with far fewer decoding steps than MDM (Zekri et al., 22 Mar 2026). The SIK models achieve the best likelihoods, but their current ancestral sampler is heavier and less accurate than the absorb or uniform samplers (Zekri et al., 22 Mar 2026).

This empirical picture resonates with the GIDD finding that richer corruption families can improve sample quality and enable revision-like behavior, even if they complicate the learning problem (Rütte et al., 6 Mar 2025). GDDS extends that intuition from mask-plus-uniform mixtures to arbitrary CTMC kernels over token vocabularies (Zekri et al., 22 Mar 2026).

7. Conceptual significance, limitations, and open directions

GDDS is significant because it changes the status of discrete diffusion from a small family of masking or uniform-replacement models into a CTMC framework with arbitrary corruption geometry, exact forward sampling through uniformization, and a snapshot-based ELBO that remains compatible with standard bidirectional Transformer backbones (Zekri et al., 22 Mar 2026). The framework’s information–calibration analysis gives a formal reason why learning from single noised snapshots can outperform full-path objectives for clean-token prediction, even though snapshots contain less information than paths (Zekri et al., 22 Mar 2026).

Several limitations are explicit. For SIK, ancestral sampling still suffers from operator-approximation error and cumulative error over many steps (Zekri et al., 22 Mar 2026). The path-wise objective based on Campbell’s estimator requires a specialized two-stream XLNet-like architecture and empirically underperforms the simpler snapshot-based DDiT-style setup used for the main results (Zekri et al., 22 Mar 2026). Snapshot training discards path information, so the information path gap is nonzero by construction (Zekri et al., 22 Mar 2026). A plausible implication is that future gains may depend less on the snapshot objective itself than on improving reverse samplers and forward-kernel approximations for structured CTMCs.

In the broader literature, three neighboring lines clarify the framework’s place. First, masked diffusion can be reformulated as a continuous-time absorbing CTMC whose ELBO is a weighted cross-entropy integral; GDDS contains this case exactly (Shi et al., 2024). Second, GIDD demonstrates that interpolating beyond pure masking enables token revision and self-correction; GDDS generalizes that insight to arbitrary Markovian noising (Rütte et al., 6 Mar 2025). Third, continuous-time snapshot dynamics work such as SpIDES and CT-OT Flow shows that “from snapshots” can refer not only to tokenwise corruption levels but also to recovering full stochastic laws from coarse temporal marginals (Zhu et al., 2024, Kawano et al., 23 May 2025). This suggests that the phrase “generalized discrete diffusion from snapshots” now names both a specific large-vocabulary CTMC framework and a broader research direction concerned with reconstructing stochastic discrete-time or discretized dynamics from marginal observations.

Future directions listed in the GDDS paper include richer Semantic-Informed Kernels, improved reverse samplers for structured kernels, and applications to other discrete domains such as graphs, molecules, and code (Zekri et al., 22 Mar 2026). Within the available evidence, the core contribution is already clear: GDDS provides a mathematically unified, snapshot-based, and computationally scalable formulation of discrete diffusion that strictly generalizes prior masked and uniform constructions while achieving strong large-vocabulary language modeling performance (Zekri et al., 22 Mar 2026).