Generalized Interpolating Discrete Diffusion (GIDD)

• GIDD is a unifying framework for discrete generative modeling that generalizes continuous diffusion to arbitrary discrete spaces with flexible noise schedules.
• It employs stochastic integral representations using Poisson measures and parameterizes transition kernels to interpolate noise types like masked, uniform, and structured corruption.
• Its rigorous error bounds, τ-leaping design, and scalable variational objectives enable precise control over generation quality and computational trade-offs.

The Generalized Interpolating Discrete Diffusion (GIDD) framework is a unifying theory and toolkit for generative modeling with discrete diffusion processes. GIDD generalizes the classical (continuous-state) diffusion paradigm to arbitrary discrete state-spaces, supports flexible noising and denoising schedules, and provides both algorithmic and rigorous error guarantees for designing, analyzing, and scaling discrete generative models. By parameterizing transition kernels that interpolate between canonical noise types such as masked, uniform, and structured corruption, GIDD encompasses a vast family of Markovian and non-Markovian processes, admits exact or tractable variational lower bounds, and directly unifies discrete, continuous, and hybrid diffusion methodologies.

1. Stochastic Integral and Poisson Measure Foundations

GIDD formalizes discrete diffusion processes via Lévy-type stochastic integrals constructed from Poisson random measures with state-dependent intensity. Let $X$ be a finite state space with the counting measure $\nu(dy)=\sum_{y\in X}\delta_y$. For a predictable, nonnegative intensity $\lambda_t(y)$ satisfying integrability conditions, the associated Poisson random measure $N[\lambda](dt,dy)$ satisfies: for any disjoint time–state sets, increments are independent, and the count over $(s,t]\times B$ is Poisson with mean $\int_s^t\sum_{y\in B} \lambda_\tau(y)\nu(dy) d\tau$. The compensated measure $$\tilde N[\lambda] = N[\lambda] - \lambda_t(y)\nu(dy)dt$$ is a local martingale.

This structure enables a stochastic integral representation of the dynamics of a time-homogeneous Markov chain with generator $Q(x,y)$:

$$X_t = X_0 + \int_0^t \sum_{y\in X} (y - X_{s^-}) N[\lambda](ds,dy),$$

with $\lambda_t(y) = Q(y, X_{t^-})$ if $\nu(dy)=\sum_{y\in X}\delta_y$0 and $\nu(dy)=\sum_{y\in X}\delta_y$1 otherwise. This recovers the CTMC law and, in compensated form, admits a decomposition into drift and martingale fluctuation terms (Ren et al., 2024).

2. Interpolation Mechanisms and Unified Noise Schedules

A core feature of GIDD is its general parameterization of transition kernels via interpolation of multiple archetypal noise processes. For a discrete token $\nu(dy)=\sum_{y\in X}\delta_y$2 in vocabulary $\nu(dy)=\sum_{y\in X}\delta_y$3, and a continuous “time” parameter $\nu(dy)=\sum_{y\in X}\delta_y$4, the forward process is specified by:

$\nu(dy)=\sum_{y\in X}\delta_y$5

where $\nu(dy)=\sum_{y\in X}\delta_y$6 is the survival/mixing rate (typically $\nu(dy)=\sum_{y\in X}\delta_y$7, $\nu(dy)=\sum_{y\in X}\delta_y$8), $\nu(dy)=\sum_{y\in X}\delta_y$9, and $\lambda_t(y)$0 is a time-dependent “mixing distribution” that may interpolate between hard-masked, uniform, or other patterns (Rütte et al., 6 Mar 2025, Rütte et al., 11 Dec 2025).

The transition from $\lambda_t(y)$1 at time $\lambda_t(y)$2 to $\lambda_t(y)$3 at $\lambda_t(y)$4 is

$\lambda_t(y)$5

with explicit expressions for $\lambda_t(y)$6 and $\lambda_t(y)$7 in terms of the schedule. The schedule itself can blend, for example, an absorbing mask and uniform noise:

$\lambda_t(y)$8

where $\lambda_t(y)$9 is the absorbing mask token and $N[\lambda](dt,dy)$0 is the uniform distribution. Parametric schedules enable interpolation between masked diffusion (pure absorption), uniform diffusion (pure randomization), hybrid settings, or structured diffusion (e.g., via nearest-neighbor graphs or Gaussian kernels as in D3PMs) (Austin et al., 2021, Li et al., 17 Apr 2026, Kollovieh et al., 19 Apr 2026). This mechanism allows fine-grained control over the trade-off between sample diversity, self-correction dynamics, and error accumulation.

3. Change-of-Measure, Variational Objectives, and KL Decomposition

GIDD extends the change-of-measure theorems of continuous diffusion (Girsanov) to the discrete setting. For two Poisson random measures $N[\lambda](dt,dy)$1, $N[\lambda](dt,dy)$2 with $N[\lambda](dt,dy)$3, the Radon–Nikodym density is

$N[\lambda](dt,dy)$4

which, when a true martingale, transforms $N[\lambda](dt,dy)$5 into $N[\lambda](dt,dy)$6 under the new measure.

The expected $N[\lambda](dt,dy)$7 between true and approximate path measures $N[\lambda](dt,dy)$8 decomposes as:

$N[\lambda](dt,dy)$9

with $(s,t]\times B$0, directly connecting to the "score-entropy" loss minimized in discrete diffusion training.

In ELBO-based formulations, such as those used for language modeling, the loss function may include both per-step KL terms and surrogate Itakura–Saito distances under importance weighting:

$(s,t]\times B$1

Reverse processes are implemented via neural denoisers parameterized either by explicit posterior estimation or by CTMC-aligned exit rates and jump-directional distributions, yielding a decomposition of the KL into Poisson (timing) and categorical (direction) terms (Rütte et al., 6 Mar 2025, Li et al., 17 Apr 2026).

4. Error Bounds, τ-Leaping, and Algorithmic Design

The framework provides rigorous error analysis for discrete-time approximations such as τ-leaping. For discretizations of the backward process, the KL divergence between true and approximated path measures admits the bound:

$(s,t]\times B$2

where $(s,t]\times B$3 is the ergodicity rate, $(s,t]\times B$4 the state-space cardinality, $(s,t]\times B$5 a bound on diagonal elements, $(s,t]\times B$6 mesh size, and $(s,t]\times B$7 the approximation error for learned scores. Optimal configuration of $(s,t]\times B$8 and $(s,t]\times B$9 (mixing time and step size) achieves KL error $\int_s^t\sum_{y\in B} \lambda_\tau(y)\nu(dy) d\tau$0 in $\int_s^t\sum_{y\in B} \lambda_\tau(y)\nu(dy) d\tau$1 steps for state spaces of dimension $\int_s^t\sum_{y\in B} \lambda_\tau(y)\nu(dy) d\tau$2, guiding computation–quality tradeoffs and motivating comparison across τ-leaping and other discrete solvers (Ren et al., 2024).

Algorithmically, GIDD prescribes:

• Schedule selection to ensure sufficient mixing ($\int_s^t\sum_{y\in B} \lambda_\tau(y)\nu(dy) d\tau$3 large for $\int_s^t\sum_{y\in B} \lambda_\tau(y)\nu(dy) d\tau$4),
• Score estimator training to KL tolerance ($\int_s^t\sum_{y\in B} \lambda_\tau(y)\nu(dy) d\tau$5),
• Tuning discretization step-size $\int_s^t\sum_{y\in B} \lambda_\tau(y)\nu(dy) d\tau$6 for low cumulative discretization error ($\int_s^t\sum_{y\in B} \lambda_\tau(y)\nu(dy) d\tau$7),
• Sampler design that efficiently interpolates between masked and uniform/noise, with correction dynamics analyzable via explicit state transitions or absorbing-state mechanisms (Wang et al., 13 Feb 2026).

5. Relation to Structured, Continuous, and Hybrid Diffusion Models

GIDD both unifies and generalizes prior discrete and continuous diffusion models. In continuous state-spaces, the family recovers (by appropriate specialization) DDPMs, DDIMs, probability-flow ODE schemes, and axes-wise noise scheduling (paDDIM) (Han, 2024, Gerdes et al., 2024). The key is the interpretation of the transition mechanisms and score networks in a basis aligned with the data, allowing for per-component or per-axis noise schedules, and interpolation between fully parallel (diffusion-like) and sequential (autoregressive) regimes. In discrete state-spaces, GIDD encompasses models including:

• Uniform-corruption D3PMs,
• Absorbing/masked (BERT-like) models,
• Graph-diffusion and other structured kernels,
• Arbitrary convex mixtures of the above (Austin et al., 2021, Kollovieh et al., 19 Apr 2026, Rütte et al., 6 Mar 2025).

By varying the interpolating kernel, one recovers classical BERT, autoregressive, score-based, and hybrid models as strict special cases, with algorithmic and theoretical machinery unambiguously deriving from the unified stochastic calculus perspective.

6. Scaling Laws, Empirical Observations, and Practical Trade-offs

Scaling behavior in GIDD is sensitive to the interplay between the chosen interpolation scheme, learning rates, batch size, and total sample/computation budget. Compute and data scaling exponents vary with the interpolation between masked and uniform noise; e.g., more uniform noise exhibits scaling-larger optimal model size at fixed compute and lower data requirements, while masked diffusion is more data-hungry but less parameter-intensive. These dependencies—quantified via exponents $\int_s^t\sum_{y\in B} \lambda_\tau(y)\nu(dy) d\tau$8—are robust to hyperparameter annealing and hold up to the largest published discrete diffusion models. Empirical results confirm that GIDD’s interpolation provides sample quality and correction advantages in language modeling, molecular generation, and other tasks, while parallel self-correction dynamics benefit from hybrid or uniform corruption schedules with absorptive states enabling greater correction per step (Rütte et al., 11 Dec 2025, Wang et al., 13 Feb 2026, Kollovieh et al., 19 Apr 2026).

7. Generalizations, Open Directions, and Theoretical Guarantees

The GIDD meta-framework admits further extensions, including:

• Decoupling the mixing or resampling schedule from the Markov chain (e.g., via learned or adaptive interpolation functions),
• Exploring non-Gaussian (Lévy or nonparametric) noise for complex data,
• Component- or axis-wise learned schedules in an arbitrary transformed basis,
• Incorporation of customized priors anchored to data statistics or domain knowledge,
• Algorithmic innovations in reverse-network parameterization—Neural CTMC, absorbing-state, and multi-head models all naturally subsume GIDD noise designs with provable equivalence between tractable surrogates and variational lower bounds under standard regularity assumptions (Li et al., 17 Apr 2026).

Algorithmic, statistical, and theoretical properties of GIDD are thus linked via the unified stochastic integral construction. This enables discrete-diffusion generative models to diagnose, analyze, and optimize for sample quality, generation speed, and compute/data efficiency—providing a central organizing principle for ongoing advancements in discrete generative modeling.