Binomial flows: Denoising and flow matching for discrete ordinal data

Abstract: Flow-based generative modeling in continuous spaces exploit Tweedie's formula to express the denoiser (learned in training) as a score function (used in sampling). In contrast, this relation has been largely missing in the discrete setting where common approaches focus on learning discrete scores and rates. In this work we close this gap for discrete non-negative ordinal data by introducing Binomial flows. Our framework provides a simple recipe for training a discrete diffusion model which simultaneously denoises, samples, and estimates exact likelihoods. We verify our methodology on synthetic examples and obtain competitive results on real-world data sets.

• The paper introduces a novel binomial thinning framework that extends Tweedie’s formula to discrete ordinal data for exact likelihood estimation.
• It defines a discrete denoiser based on conditional expectations, enabling efficient parameterization and reverse-time sample synthesis via binomial flows.
• Empirical results on synthetic and CIFAR-10 benchmarks demonstrate strong likelihood recovery and superior generative performance compared to prior discrete diffusion models.

Binomial Flows: Denoising and Flow Matching for Discrete Ordinal Data

Introduction and Motivation

The paper "Binomial flows: Denoising and flow matching for discrete ordinal data" (2605.00360) presents a principled framework for generative modeling of non-negative discrete ordinal data via a binomial thinning-based discrete diffusion process. Modern generative models, particularly continuous normalizing flows and diffusion models, leverage connections between denoising objectives and score-based generative processes, most notably through Tweedie’s formula. However, the extension of such frameworks to discrete data has been hampered by the absence of a clear analogue to Gaussian noise and Tweedie's formula in the discrete setting.

The authors address this limitation by demonstrating that binomial thinning serves as the appropriate noise mechanism for discrete ordinal variables, enabling the direct definition and training of denoisers in analogy with the continuous case. This approach also provides a discrete Tweedie’s formula, facilitating likelihood estimation, sample generation, and theoretical analysis with strong parallels to continuous-data score models.

Binomial Flows: Formulation and Algorithmic Framework

For data supported on $\mathbb{N}^d$, the proposed method constructs a Markov bridge from pure noise (all zeros) to the data distribution by successively “thinning” counts according to binomial distributions—with underlying success probability $t/T$ interpolating from noise ($t=0$) to data ($t=T$). The critical insight is that, given a clean sample $x_T$, a noisy version $x_t$ is generated componentwise by $x_t^i \sim \mathrm{Binomial}(x_T^i, t/T)$.

Let $X_T$ be the data variable, and $X_t$ be the thinned variable at time $t$:

• At $t/T$0: $t/T$1 is always zero;
• At $t/T$2: $t/T$3 is the data;
• For $t/T$4: $t/T$5.

The denoiser $t/T$6 is defined as the conditional expectation $t/T$7. As in the continuous case, the denoiser is shown to uniquely minimize a Bregman divergence-based loss between the recovered and true data, enabling efficient parameterization and training.

Discrete Tweedie’s Formula: The authors prove that the discrete Tweedie’s formula holds: the rate function for the underlying sampling process is exactly

$t/T$8

which mirrors the relationship between the denoiser and score in the Ornstein-Uhlenbeck diffusion in continuous space.

Figure 1: Target dataset, generated samples using Binomial Flows, and true target PMFs for four representative synthetic distributions.

Poisson-Föllmer Process: Sampling and Theoretical Structure

Binomial flows naturally induce a time-inhomogeneous continuous-time Markov chain termed the Poisson-Föllmer process. This process starts at all zeros and increments via Poisson-type arrivals, but the jump intensities are functions of the current state determined by the denoiser. The process is constructed such that its endpoint at $t/T$9 exactly samples from the target data distribution, and its bridges correspond to binomial thinned paths.

Bridge characterization: Conditional on $t=0$0, the forward path $t=0$1 is obtained via binomial thinning. The reverse-time process induces a nontrivial bridge Markov chain, but crucially, the denoiser $t=0$2 specifies the required rate function to reverse the process and synthesize samples.

Likelihood Estimation and Exact Training Identities

A key result is an explicit identity for the data likelihood in terms of the denoiser, analogous to continuous diffusion score-matching estimators but exact in the discrete case:

$t=0$3

where $t=0$4 is the Bregman divergence for the discrete setting. This facilitates both maximum-likelihood training and precise likelihood evaluation for generative modeling.

Figure 2: True and estimated log-likelihoods for Poisson, ZIP, and Yule-Simon test distributions—demonstrating high-fidelity likelihood recovery via Binomial flows.

Empirical Results: Synthetic and Image Data

The framework is applied to a variety of challenging low-dimensional synthetic distributions (Poisson mixtures, heavy-tailed, multimodal, see Figure 1), where the model achieves tight negative log-likelihoods converging closely to ground truth. Importantly, the approach is directly scalable to high-dimensional image data via the EDM adaptation and DDPM++ model architecture.

On the unconditional CIFAR-10 benchmark, Binomial Flow achieves a Fréchet Inception Distance (FID) of 2.94, outperforming prior discrete-diffusion baselines such as Blackout (4.58), LTJ (4.80), and approaching the best continuous-data models.

Figure 3: Unconditional CIFAR-10 samples generated using the Poisson-Föllmer process with EDM preconditioning.

The method leverages EDM-style preconditioning, time parameterizations, and the $t=0$5-leaping numerical sampler for efficient high-dimensional synthesis; ablations show the importance of non-uniform time discretization and noise-level sampling.

Analysis of Denoising Dynamics and Parameterization

Salient for diagnostics and interpretability, the binomial-flow denoiser interpolates smoothly between noise and signal as time progresses, reconstructing context and global structure even at very early steps (see denoiser outputs and denoising trajectories).

Figure 4: Denoising at early time $t=0$6. Left: noisy binomial observations; Middle: denoiser predictions; Right: ground-truth clean images.

Figure 5: Comparison of denoising trajectories for uniform discretization in time $t=0$7 (top) versus noise level $t=0$8 (bottom), highlighting advantages of logarithmic time parameterization for early-stage coverage.

A study of layer scaling and baseline predictors for the denoising task demonstrates that the neural denoiser’s improvement over affine baselines is peaked at intermediate noise levels, motivating non-uniform sampling schemes during training for maximum statistical efficiency.

Figure 6: Distribution of noise level sampling; the normalized improvement curve informs optimal data curriculum for denoiser training.

Comparison to Related Works

Whereas previous approaches either focus on discrete score/rate learning or employ Poisson-based perturbations (which do not recover the data at a finite time horizon), Binomial flows recover the data exactly at $t=0$9 and never overshoot the support. In contrast to Blackout, which also uses binomial thinning but lacks the discrete Tweedie’s formula, the current framework enables exact likelihoods and sample generation with arbitrary Bregman losses. This provides a direct discrete analogue to state-of-the-art continuous flow-matching/diffusion models and strengthens the theoretical and empirical connection between score-based modeling and generative processes in discrete spaces.

Implications and Future Directions

Practically, Binomial flows provide a robust framework for likelihood-based generative modeling and discrete-data synthesis without quantization artifacts, bridging the theoretical gap with continuous methods. The derived explicit denoiser likelihood identity is particularly useful for principled evaluation and model selection.

Theoretically, these results emphasize the importance of the correspondence between discrete and continuous generative processes, laying groundwork for further advancements such as:

• Sharper time discretization and inference schedules for computational efficiency
• The role of denoiser martingale properties for improved training or consistent generation
• Systematic extensions to structured discrete data (e.g., molecules, text sequences)
• Explorations of guidance methods in large-scale diffusion with discrete support

Conclusion

This work rigorously establishes binomial thinning as the discrete counterpart to Gaussian noise in denoising diffusion models for non-negative ordinal data, enabling the extension of continuous-data theoretical tools, such as Tweedie’s formula, to the discrete setting. The resulting Binomial flows framework demonstrates state-of-the-art performance among discrete generative models and provides strong theoretical and practical tools for future developments in discrete score-based generative modeling.