Discrete Diffusion and Flow Matching

• Discrete diffusion and flow matching models are generative frameworks that learn probability distributions over categorical data using Markov processes and principles from optimal transport and information geometry.
• They leverage methodologies such as score matching, generator matching, and information-geometric embeddings to interpolate efficiently between a tractable source and the target data distribution.
• These models have practical applications in language modeling, molecule and graph generation, and speech synthesis, offering improved sampling efficiency and quality.

Discrete diffusion and flow matching models are generative frameworks for learning probability distributions over discrete or categorical state spaces, mirroring the success of diffusion and ODE-based (flow matching) models for continuous data modalities. Central to these approaches is the construction of a path of probability distributions (often over sequences, graphs, or other structured objects) that interpolates between a tractable source distribution and the empirical data distribution, enabling both efficient learning and flexible sampling. Recent research has produced a diverse set of theoretical tools, algorithmic designs, and applications, extending flow paradigms to discrete domains and unifying them with stochastic optimal transport and information geometry.

1. Mathematical Principles and Definitions

Let $\mathcal{X} = \mathcal{S}^D$ denote a finite discrete state space of interest (e.g., token sequences, graphs, molecular fragments). Discrete diffusion models typically define a continuous-time Markov chain (CTMC) or discrete-time Markov process which gradually corrupts data samples from the empirical data law $q_1(x)$ to a tractable source (e.g., uniform or masked), and learn to reverse this process either via score matching (for CTMCs) or by learning optimal transition rates.

In the flow matching paradigm, the core object is a time-dependent family of distributions $\{p_t\}_{t\in[0,1]}$, along with an associated velocity or generator field, such that the process transports $p_0$ to $q_1$. The most general formulation employs a generator (rate) matrix $Q_t(z,x)$ on $\mathcal{X}$ defining the dynamics: $$P(X(t+h)=z | X(t)=x) = \delta_{x,z} + u_t(z,x) h + o(h)\,,$$ with $u_t(z,x)\geq0$ for $z\neq x$ and column sums zero. For each $t$, $p_t$ evolves according to the discrete Liouville equation: $$\frac{d}{dt} p_t(x) = \sum_{z\in\mathcal{X}}\big[u_t(x,z)p_t(z) - u_t(z,x)p_t(x)\big]\,.$$

In continuous-state discrete flow matching (CS-DFM), probability mass vectors $\mu\in\Delta^{d-1}$ (e.g., one-hot or softened categorical distributions) are embedded into a manifold via a parametric map (e.g., $\pi^{(\alpha)}$), and Riemannian flow-matching techniques are applied in this space (Cheng et al., 14 Apr 2025).

2. Modeling Paradigms: Discrete Diffusion, Flow Matching, and Hybrid Approaches

Discrete diffusion models (e.g., D3PM, masked diffusion) employ predefined noising schedules and focus on learning denoising or reverse transition probabilities via score matching. The connection to optimal transport has been formalized: under appropriate conditions, continuous probability flow corresponds to a Monge map between intermediate marginals, and the discrete analogue achieves trajectory-wise optimality under a Kantorovich plan (Zhang et al., 2023).

Flow matching in discrete space fundamentally extends the notion of deterministic ODE flows to stochastic jump processes. Rather than score estimation, the core learning objective is generator/velocity matching, directly estimating the rates or the conditional stepwise denoisers that parameterize the flow (Gat et al., 22 Jul 2024, Wan et al., 26 Sep 2025). Discrete flows erase the need for a forward noising chain or score-based reversal, and, with appropriate uniformization and generator design, incur no truncation or discretization bias (Wan et al., 26 Sep 2025).

A variety of hybridizations have emerged:

• Discretized Rectified Flow: Discrete analogues of rectified continuous flows involve subdividing the path into variable-velocity momentum fields by recursively perturbing the velocity with noise, resulting in momentum flow matching and allowing interpolation between deterministic ODEs and stochastic CTMCs (Ma et al., 10 Jun 2025). This achieves both diversity and efficiency: deterministic segments for efficient straight-line propagation near the data, and stochastic variation to capture diversity closer to the prior.
• $\alpha$-Flow and Fisher-Flow: Utilizing information-geometric embeddings and the Fisher–Rao metric, flows are defined along geodesics in probability space (e.g., on the positive orthant of the $\ell_2$ sphere), offering global optimality with respect to generalized kinetic energy and controlling trade-offs between sample fidelity and diversity (Cheng et al., 14 Apr 2025, Davis et al., 23 May 2024).
• Minibatch Optimal Transport (OT): In settings where deterministic rectification is infeasible, discrete flow objectives can be recast as dynamic OT problems (discrete-time Benamou–Brenier), with a Kantorovich dual solved in minibatch for scalable training (Haxholli et al., 1 Nov 2024).

3. Objective Functions and Training Algorithms

Diverse objective functions have been developed, tailored to generator learning, information geometry, and optimal transport interpretations:

• Velocity/generator matching (CTMC): For general discrete flow models, the pathwise KL divergence between true and model path measures yields an integral of Bregman divergences over time: $$L(Q^0, \widehat{Q}) = \mathbb{E}^{Q^0}\left[ \int_0^1 \sum_{z\neq X(s)} D_F(u^0_s(z,X(s))\,\|\,\hat{u}_s(z,X(s)))\,ds \right]\,,$$ where $D_F(a\|b) = a\log(a/b) - a + b$ (Wan et al., 26 Sep 2025).
• Information-geometric objectives: For CS-DFM, the training loss becomes

$$L^{(\alpha)}(\theta) = \mathbb{E}_{t,\mu_0,\mu_1} \| v_\theta(x_t, t) - \dot{x}_t \|_g^2\,,$$

where $v_\theta$ is the projected velocity in the appropriate geometry (Cheng et al., 14 Apr 2025).

• Minibatch OT objective: The dynamic transport cost reduces to a minibatch Kantorovich cost over the coupling $\pi(x_0, x_1)$,

$$\mathcal{D}_{\text{dyn}} = \sum_{x_0, x_1} c(x_0, x_1)\pi(x_0,x_1)$$

with $c(x_0,x_1)$ typically Hamming or embedding distance, making the flow globally optimal under convex interpolant conditional laws (Haxholli et al., 1 Nov 2024).

Training proceeds via stochastic gradient descent using Monte Carlo samples of paths or rates, with regularization and early stopping to control estimation and approximation errors (Wan et al., 26 Sep 2025).

4. Sampling Algorithms and Efficiency

Flow-matching models furnish closed-form formulas for sampling velocities, often in terms of learned posteriors (denoisers or noise-predictors). For convex-interpolant flows in discrete space: $$u_t(x_i \mid z) = \frac{k_t}{1 - k_t} \big[ p_{1|t}(x_i \mid z) - \delta_{z_i}(x_i) \big]\,,$$ where $k_t$ is the noise scheduler and $p_{1|t}$ is the predicted one-step posterior (Gat et al., 22 Jul 2024).

The resulting sampling algorithm is highly parallelizable: each variable (or sequence position, graph component, etc.) is sampled independently given the current state and the estimated marginal velocities. Adaptive and corrector steps further enhance coverage and empirical performance.

Discrete generator-matched flows can be exactly sampled using Poisson uniformization; this removes time-discretization bias present in previous score-based approaches (Wan et al., 26 Sep 2025). Momentum flow approaches leverage sub-path Euler solvers, trading off the number of function evaluations (NFE) for fidelity and diversity, with empirical results showing strong sample quality at far fewer NFE than score-based discrete diffusion (Ma et al., 10 Jun 2025).

Guidance mechanisms, including classifier and energy guidance, have been generalized to discrete flows using exact density ratios, surpassing previously prevalent first-order Taylor approximations in both accuracy and computational efficiency (Wan et al., 26 Sep 2025).

5. Theoretical Analysis and Connections

Theoretical foundations link discrete flows to optimal transport (both Monge and Kantorovich), information geometry (Fisher–Rao, $\alpha$-divergences), and stochastic calculus:

• Pathwise KL bounds: Nonasymptotic error analyses decompose the total error into stochastic estimation, generator approximation, and early-stopping bias, with discrete flows incurring no truncation error unlike finite-horizon diffusions (Wan et al., 26 Sep 2025).
• Information geometry: $\alpha$-geodesic flows are globally optimal for generalized kinetic energy, yielding a single knob ($\alpha$) for fidelity-diversity control (Cheng et al., 14 Apr 2025, Davis et al., 23 May 2024).
• Discrete probability flows as optimal transport: For CTMCs, every infinitesimal segment of the probability flow corresponds to a local Kantorovich minimizer; globally, the dynamics realize the Monge map under mild conditions (Zhang et al., 2023).
• Evaluation metrics and perplexity bounds: KL-derived upper bounds allow estimation of negative log-likelihood and perplexity for non-autoregressive models, supporting model selection without full generation (Haxholli et al., 1 Nov 2024).

Discrete flow models can be seen as interpolating between deterministic rectified flows and stochastic discrete diffusion, enabling a continuum of efficiency and diversity regimes (Ma et al., 10 Jun 2025).

6. Applications and Empirical Results

Discrete diffusion and flow matching models have demonstrated strong empirical performance across discrete domains:

• Language modeling and code generation: Discrete flow matching achieves competitive perplexity and functional pass@k metrics compared to autoregressive baselines, with significant speedup due to non-autoregressive, parallel sampling (Gat et al., 22 Jul 2024).
• Molecular and graph generation: Fragment-level discrete flows (FragFM) with coarse-to-fine latent decoding outperform atom-level diffusion and flow models in validity and property-aware molecule generation (Lee et al., 19 Feb 2025). Graph flows like DeFoG achieve state-of-the-art quality on molecular and synthetic benchmarks with 10–20$\times$ fewer sampling steps (Qin et al., 5 Oct 2024).
• Speech and TTS: DiFlow-TTS and Drax apply discrete flows to speech synthesis and ASR, leveraging factorized token representations, audio-conditioned probability paths, and parallel decoding to combine strong accuracy with order-of-magnitude speedups (Nguyen et al., 11 Sep 2025, Navon et al., 5 Oct 2025).
• Evaluation and practical design: Minibatch OT flows are preferred when deterministic mapping is intractable, and embedding- or Hamming-based costs can be exploited for specialized task objectives (Haxholli et al., 1 Nov 2024). Guidance, early stopping, and hybrid schedules are effective in discrete settings (Wan et al., 26 Sep 2025).

7. Future Directions and Open Problems

Key priorities in ongoing research include:

• Task-specific path design: Beyond convex interpolants, paths adapted to empirical error distributions, or data-dependent bridges, may further close the generalization gap (Navon et al., 5 Oct 2025).
• Adaptive geometry: Dynamic or learned $\alpha$-schedules, Fisher–Rao metric tuning, or hybrid continuous-discrete geometries offer enhanced modeling capacity (Cheng et al., 14 Apr 2025, Davis et al., 23 May 2024).
• Hybrid models: Combining flow matching with diffusion or autoregressive methods, e.g., using OT-trained discrete flows as the initialization or pre-flow for discrete diffusion, has shown promise for achieving optimal speed-quality trade-offs (Haxholli et al., 1 Nov 2024).
• Scalable evaluation and tuning: Efficient KL and perplexity bounds, as well as error analyses via Girsanov-type results, inform optimization of training and decoding strategies (Haxholli et al., 1 Nov 2024, Wan et al., 26 Sep 2025).
• Theoretical refinement: Deeper connections to stochastic optimal transport, conditions for uniqueness of discrete Monge maps, and convergence analysis for minibatch OT training remain important open questions (Zhang et al., 2023, Wan et al., 26 Sep 2025).

Discrete diffusion and flow matching constitute a unified, versatile class of non-autoregressive generative models for categorical data, grounded in optimal transport, stochastic process theory, and information geometry. Their rapidly widening empirical and theoretical scope continues to advance the state-of-the-art across structured discrete domains.