Interpolating Discrete Diffusion

• Interpolating discrete diffusion is a framework that constructs continuous paths between discrete states using mixing kernels and tunable noise parameters.
• It leverages mathematical foundations—ranging from Markov processes and stochastic differential equations to geometric constructions—to interpolate between deterministic, masking, and random regimes.
• The approach is applied in language modeling, molecular graph generation, and image interpolation, offering improved error correction, robustness, and sample quality.

Interpolating discrete diffusion refers to a broad class of mathematical, statistical, and generative modeling techniques that construct continuous or controllable paths (diffusions) between discrete states or distributions. This area encompasses both theoretical frameworks—such as stochastic processes interpolating between classical and fractional diffusion—and modern generative modeling applications, where discrete diffusion chains with tunable noising and resampling mechanisms interpolate among different regimes of error correction, data corruption, and token generation. Interpolation may occur in discrete time, via explicit mixture kernels; in continuous time, using Markov chain generators or SDE limits; or via pathwise or geometric constructions in latent or data spaces.

1. Mathematical Foundations of Discrete Diffusion Interpolation

Interpolating discrete diffusion models generalize classical Markov processes on finite sets by defining transition kernels that mix the current state with other designated outcomes according to scheduling or interpolation parameters. At its core, for a finite state space $X = \{1, \ldots, K\}$, a typical interpolating kernel at time $t$ takes the form

$$q_t(x'|x) = \alpha_t\, \delta_{x'=x} + (1-\alpha_t)\, \pi_t(x'),$$

where $\alpha_t$ is a “signal” or “mixing” rate and $\pi_t$ a time-varying mixing distribution over $X$. This encapsulates uniform noising (mixing with the uniform), masking (mixing with a special symbol), as well as more expressive schedules. The evolution of marginal distributions is governed by products or exponentials of these kernels—in discrete and continuous time, respectively. This setup enables precise, tunable interpolation between deterministic identity (no noise), pure masking, full randomization, and hybrid regimes (Pauline et al., 4 Dec 2025).

2. Continuous-Time and Fractional Interpolation: Hamiltonian Models

Beyond categorical state spaces, interpolation arises in physical lattice models with conserved quantities. For example, the Hamiltonian lattice–field model introduces two independent noises:

• a "volume and energy-conserving" Brownian noise acting on triplets of sites;
• an "energy-only" Brownian noise acting on site pairs.

By controlling the relative strength of these noises through a scaling parameter $a$, one constructs a process that in the hydrodynamic limit (critical scaling $a=3/2$) interpolates between normal diffusion (heat equation) and anomalous fractional diffusion generated by a skewed $3/2$-stable Lévy process. At the critical value, the fluctuation field of the energy converges to the solution of an infinite-dimensional Ornstein–Uhlenbeck equation driven by a nonlocal operator whose Fourier symbol bridges the Laplacian and a fractional Laplacian (Bernardin et al., 2016). This provides, at the SPDE level, a mathematically rigorous interpolation between distinct universality classes of diffusive behavior.

3. Interpolating Discrete Diffusion in Generative Modeling

Modern discrete diffusion models in machine learning—especially for language and graph generation—deploy interpolating kernels to mediate a trade-off between the rigidity of masking-based processes and the flexibility of uniform noise. For instance, Generalized Interpolating Discrete Diffusion (GIDD) uses time-varying schedules for both the mixing rate and the mixing distribution $\pi_t$, enabling hybrid masking-uniform processes. This flexibility allows the model to correct previously made token errors (self-correction), a major limitation of pure masking schemes. The ELBO framework generalizes to these settings, retaining closed-form expressions for marginals and reversibility (Rütte et al., 6 Mar 2025, Rütte et al., 11 Dec 2025).

In the Interpolating Discrete Diffusion Model with controllable resampling (IDDM), transition kernels interpolate between staying at the current state, flipping to the ground-truth, and resampling from a prior, with a scalar “forget” parameter $t$0 regulating the resampling. This mechanism enables fine-grained control of error correction: at $t$1 one gets absorbing models that are unable to revise early errors; at $t$2 the model fully resamples, maximizing correction potential at the cost of stability (Kollovieh et al., 19 Apr 2026).

Block Diffusion models go further and interpolate between purely autoregressive and fully diffusive models by partitioning the sequence into blocks, enabling tuning of parallelism and modeling granularity by block size (Arriola et al., 12 Mar 2025).

4. Continuous-Discrete and Geometric Interpolations

Interpolation need not be limited to discrete chains. In coevolutionary continuous-discrete diffusion (CCDD), a joint diffusion over both a discrete token space and a continuous latent space is defined, allowing for the strengths of both modalities—discrete decoding and continuous expressivity—to be utilized simultaneously. The training loss combines ELBOs from both components, and the model architecture includes cross-modal attention or mixture-of-expert routing. This approach empirically improves perplexity and sample fidelity over pure discrete or pure continuous models (Zhou et al., 3 Oct 2025).

In probability-simplex diffusion, discrete states are embedded in a continuous logit space, diffused via Ornstein–Uhlenbeck SDEs, and projected back to the simplex by softmax, resulting in smooth interpolating paths (bridges) within the space of probability vectors—an explicit geometric interpolation between categorical distributions (Floto et al., 2023).

For image or high-dimensional data, interpolation with score-based Riemannian metrics leverages the pretrained diffusion model's score function to endow the data space with a Riemannian metric, then computes geodesics (minimizing paths) under this metric. This yields geometry-aware interpolations outperforming naive linear or spherical heuristics, grounding interpolation firmly in the learned data manifold (Saito et al., 28 Apr 2025).

5. Practical Algorithms, Variational Framework, and Empirical Observations

Key algorithmic elements across discrete-diffusion interpolation literature include:

• Construction of forward and reverse kernels analytically to preserve specified marginals;
• Optimization of scheduling (mixing, noise type, transition counts) for sample quality, parallelism, and variance minimization;
• ELBO or KL-based objectives for likelihood maximization in both discrete and continuous-time settings;
• Decoupling training from inference via parametrized or resampling-based transition families.

Empirically, interpolated kernels mitigate error accumulation inherent to pure masking models and reduce sample degradation relative to pure uniform models by controlling the resampling rate. Scaling studies reveal that hybrid or uniform noise schedules may yield superior scaling exponents or data efficiency in compute-bound or token-bound regimes (Rütte et al., 11 Dec 2025). Interpolated diffusion chains have enabled state-of-the-art results in language modeling, molecular graph generation, and improved robustness to noise and domain drift (Kollovieh et al., 19 Apr 2026, Rütte et al., 6 Mar 2025).

6. Theoretical Properties, Extensions, and Open Directions

The foundational properties of interpolating discrete diffusion include:

• Marginal consistency, ensuring that the family of forward or reverse kernels maintains the correct transition marginals at all times;
• Reversibility and the existence of a closed-form ELBO for both mixed discrete and continuous-time chains (Pauline et al., 4 Dec 2025);
• Control of expressivity and inductive bias via schedule and mixture parameterization—the space of possible interpolations subsumes pure masking, full randomization, and any smooth convex combination;
• Robustness of pathwise and kernel-based interpolation to the scaling of state space and parameterization.

Open directions under active research involve learned or data-dependent mixing schedules, integration of classifier-free or classifier-guided sampling, variance-reduced samplers, and the extension to more expressive structured noise processes or hybrid geometric-algebraic frameworks (Rütte et al., 6 Mar 2025, Kollovieh et al., 19 Apr 2026). Bridging the gap between mathematically motivated physical models and practical generative algorithms remains a fertile ground for future investigation.