Uniform-State Diffusion Models

• Uniform-State Diffusion Models (USDMs) are generative models that transform data into a uniform distribution across discrete or continuous domains using Markovian noise processes.
• They employ uniformization techniques in both CTMC for discrete data and SDEs for continuous data, enabling exact sampling without time-discretization error.
• USDMs offer rapid mixing, favorable convergence rates, and practical applications in language modeling, vision tasks, and semiparametric diffusion analysis.

Uniform-State Diffusion Models (USDMs) constitute a class of generative models in which the forward process iteratively transforms data into a uniform, structureless distribution over a discrete or continuous state space. This framework unifies and extends the theory of score-based diffusion to categorical and discrete domains, providing both algorithmic tools and theoretical guarantees for domains such as language, graphs, discrete vectors, and semiparametric diffusions. In discrete settings, USDMs exploit continuous-time Markov chain (CTMC) “noising” and reverse denoising processes, leveraging the exact “uniformization” technique for algorithmic efficiency and statistical accuracy. In the continuous domain, they encompass SDE-driven processes mapped via uniformization transformations for flexible semiparametric modeling. Uniform-state kernels are central, enforcing full-mixing and ergodicity. USDMs admit exact and tractable likelihoods, avoid time discretization artifacts, and under regularity conditions achieve favorable convergence rates in both KL-divergence and total variation.

1. Mathematical Foundations of Uniform-State Diffusion

The USDM framework is founded on the concept of progressively corrupting and reconstructing data via Markovian noise schedules. In the discrete case, for a finite state space $\mathcal X$ (e.g., $\mathcal X=\{0,1\}^d$ or $\mathcal X=V^L$ for a vocabulary $V$), the forward process is a time-inhomogeneous CTMC with a generator $Q(t)\in\mathbb R^{|\mathcal X|\times|\mathcal X|}$ satisfying $Q_{x,x}(t) = -\sum_{y\neq x} Q_{x,y}(t)$ and $Q_{x,y}(t)\ge 0$ for $x\neq y$. The canonical example is the independent-flip generator on the Boolean hypercube: $$Q_{x,y} = \begin{cases} 1 & y = x+e_i \ -d & y = x \ 0 & \text{otherwise} \end{cases}$$ uniformly flipping any coordinate, yielding rapid mixing toward the uniform law (Chen et al., 12 Feb 2024). In discrete time, the uniform-replacement kernel is

$$q(x_t = i \mid x_{t-1}=j) = \alpha_t\, \delta_{i,j} + (1-\alpha_t)\tfrac{1}{K}$$

with time-dependent “keep” probability $\alpha_t\in[0,1]$ (Pauline et al., 4 Dec 2025, Austin et al., 2021).

For continuous domains, USDMs arise when an underlying scalar SDE $dX_t = \mu(X_t)dt + \sigma(X_t)dW_t$ is “uniformized” by applying the stationary cumulative distribution function $F$, i.e., $U_t = F(X_t), U_t\in [0,1]$. Then $U_t$ itself evolves as a time-inhomogeneous SDE with drift and diffusion computed from the original coefficients and $F$'s derivatives (Bu et al., 2020).

2. Forward and Reverse Processes: Uniformization and Denoising

Forward (Noising) Process

• Discrete: The forward process iterates randomizing transitions governed by the CTMC generator $Q(t)$. Uniformization yields an equivalent description in terms of discrete embedded Markov chains, with random transition times distributed as a Poisson process; specifically, the law at time $T$ is exactly the expectation over all possible sequences of embedded jumps: $$p(T) = p(0)\,e^{\int_0^T Q(s) ds} = \mathbb{E}_{N, \tau_1\ldots\tau_N}\left[ p(0) P(\tau_1)\ldots P(\tau_N)\right]$$ where $P(t)=I+Q(t)/\lambda$ for a suitable rate $\lambda\ge\sup_{x,t}(-Q_{x,x}(t))$ (Chen et al., 12 Feb 2024).
• Semiparametric Diffusions: Uniformization transforms the unknown marginal of arbitrary one-dimensional diffusions to a uniform distribution on $[0,1]$, decoupling the parametric copula dynamics from the nonparametric marginal (Bu et al., 2020).

Reverse (Denoising) Process

• Discrete: The time-reversed process is again a CTMC, but with time-inhomogeneous generator $Q^\dagger_{x,y}(t) = Q_{y,x}(t)\cdot p_y(t)/p_x(t)$. The key object is the ratio $c_{x,y}(t)=p_y(t)/p_x(t)$, typically approximated by a learned score $\mathbf{s}_{x,y}(t)$ (Chen et al., 12 Feb 2024). The learning objective is a pathwise KL divergence involving the local Bregman divergence: $$\ell(c,s) = \sum_{y\ne x} Q_{y,x}(t)\left[ -c_{x,y} + s_{x,y} + c_{x,y}\log\frac{c_{x,y}}{s_{x,y}} \right]$$
• Discrete ELBO and Parameterization: For models over symbols, the negative ELBO has a sum over steps of $D_{KL}$ terms between the true reverse-time posterior $q(x_{t-1}|x_t, x_0)$ and the learned reverse chain $p_\theta(x_{t-1}|x_t)$, plus cross-entropy or denoising terms for stabilization (Pauline et al., 4 Dec 2025, Austin et al., 2021).
• Continuous: The reverse SDE for $U_t = F(X_t)$ has coefficients $\mu_\Upsilon(u)$ and $\sigma_\Upsilon(u)$ derived by applying Itō's lemma and inverting via $x=F^{-1}(u)$ (Bu et al., 2020).

3. Algorithmic Schemes and Exact Sampling

The hallmark of discrete USDMs is that sampling from the time-reversed process is exact via uniformization:

1. Initialize from the uniform prior.
2. For each time-interval, draw the number of jumps as $M\sim \text{Poisson}(\lambda\Delta t)$.
3. At each jump time, select a jump according to the normalized learned score $s_{x}(t)$ or stay put with remaining probability.
4. Propagate forward, then return the endpoint state.

This exact simulation removes the time-discretization error present in continuous SDE-based diffusion samplers (Chen et al., 12 Feb 2024). In large-scale language and vision USDMs, sampling begins from the uniform distribution over tokens or pixels and proceeds by reverse steps, using neural network parameterizations of the conditional denoising distributions at each time point (Sahoo et al., 12 Jun 2025, Zhu et al., 27 Oct 2025).

For semiparametric copula models, likelihood-based estimators (PMLE, sieve-MLE) and kernel-smoothing approaches for drift and diffusion are both practical and theoretically justified (Bu et al., 2020).

4. Theoretical Guarantees and Comparison to Continuous Diffusion

Error analysis for discrete USDMs yields favorable scaling:

• KL and TV Bounds: Under a score-entropy accuracy assumption $\epsilon$ and a uniform rate bound, the nonstationary error is $$KL(p(\delta)||\hat p^\dagger(T-\delta)) \lesssim d e^{-T} + T\epsilon$$ and total variation distance is $$TV(p(0), \hat p^\dagger(T-\delta)) \le (1-e^{-d\delta}) + \sqrt{d e^{-T} + T \epsilon}$$ (Chen et al., 12 Feb 2024). Setting $$T\sim \log (d/\epsilon),\,\delta \sim \sqrt{\epsilon}/d$$ gives $O(\sqrt{\epsilon})$ TV and $O(\epsilon)$ KL, with $O(d \log(d/\epsilon))$ computational complexity.
• No Time-Discretization Error: Uniformization ensures that the generator is simulated exactly, in contrast to the Euler–Maruyama or other discretizations in continuous SDE-based models (Chen et al., 12 Feb 2024, Pauline et al., 4 Dec 2025).
• Dimension and Accuracy Scaling: Discrete USDMs achieve $O(\log(1/\epsilon))$ accuracy scaling and essentially linear $O(d\log(d/\epsilon))$ scaling in dimension, outperforming the $O(\epsilon^{-2})$, $O(d\cdot\mathrm{polylog}(1/\epsilon))$ scaling of $\mathbb R^d$ SDE methods (Chen et al., 12 Feb 2024).
• Spectral Gap and Mixing: Uniform-state kernels lead to rapid spectral mixing, with explicit rates in both discrete and continuous time, and the uniform law as stationary distribution (Pauline et al., 4 Dec 2025).

5. Practical Applications and Empirical Performance

Discrete Data Generation

USDMs have been deployed in text (language modeling), vision (pixel-wise symbol models), and categorical structured prediction:

• The “Duo” method exploits the connection between Gaussian diffusion and discrete USDMs to import curriculum learning (tempered softmax relaxation, lowering variance and doubling convergence speed), and discrete consistency distillation (enabling fast, few-step generation matching full-step ancestral quality). Duo achieves PPL competitive with autoregressive models and order-of-magnitude sampling acceleration (Sahoo et al., 12 Jun 2025).
• Simpler denoising-only losses (that penalize only positions corrupted in the forward process) match ELBO-trained USDMs in sample quality while greatly improving efficiency and few-step stability. Further, contrastive negative gradients in the loss function specifically enhance generation quality after a handful of denoising steps by discouraging wrong token mass (Zhu et al., 27 Oct 2025).
• Empirical results on language datasets (LM1B, OpenWebText) show that USDM-based transformers with selective denoising or “Duo” distillation outperform prior non-autoregressive and uniform diffusion baselines in perplexity, while enabling very rapid sampling.

Semiparametric Diffusion Models

Uniformization enables a transparent semiparametric framework for real-valued time series: one posits a parametric SDE, transforms into the uniform domain, estimates the drift and diffusion nonparametrically, then recovers the underlying copula structure. This provides near-parametric efficiency and strong empirical fit for challenging financial time series, such as VIX (Bu et al., 2020).

Limitations

In continuous $\mathbb R^d$ settings, replacing Gaussian with uniform additive noise (as in USDMs) yields catastrophic sample degradation and unstable score estimation, as the uniform density is piecewise constant and lacks the smoothing properties of the Gaussian kernel. Uniform noise induces very poor FID ($\sim 274$ vs $\sim 4.4$ Gaussian on CIFAR-10 at 100 steps), making it unsuitable for continuous-valued diffusion applications (Jolicoeur-Martineau et al., 2023).

6. Comparative Analysis: USDMs, Masked Diffusion, and Structured Kernels

A distinguishing feature of USDMs is the use of the uniform prior, fully symmetric mixing, and the absence of absorbing or masking states. In contrast:

• Masked Diffusion LLMs (MDLMs): Use an absorbing [MASK] state. Once a token is masked, it cannot be unmasked; this leads to poor performance with few reverse steps and inhibits distillation (Zhu et al., 27 Oct 2025, Sahoo et al., 12 Jun 2025).
• Structured/Embedding Kernels: Uniform-state is the most “diffusive” and least structured kernel; alternatives like discretized Gaussian or embedding-nearest-neighbor kernels allow local structure to be preserved and can enhance denoising, but lose ergodicity and may require careful design (Austin et al., 2021).
• Self-Correction: USDMs, having no absorbing state, allow each position to be corrected at every generation step, which is essential for enabling consistency-based distillation and efficient few-step synthesis (Sahoo et al., 12 Jun 2025).

The table below compares key aspects of USDMs, MDLMs, and Gaussian SDE models:

Model Class	Forward Noising	Reverse Process	Stationarity	Self-Correction	Sampling Complexity
USDM	Uniform CTMC/kernel	Exact by uniformization	Uniform	Yes	$O(d\log d)$
MDLM	Absorbing state ([MASK])	Absorbing reversal	Masked	No	Typically higher
Gaussian SDE	Additive Gaussian	Euler/EM discretization	Gaussian	Yes	$O(d\,\mathrm{polylog}(1/\epsilon))$

USDMs are uniquely positioned among these alternatives in discrete domains for rapid mixing, exact sampling, and compatibility with distillation.

7. Future Directions and Open Questions

Current research highlights several frontiers for USDMs:

• Discrete Probability-Flow ODEs: Extending deterministic sampling methods (e.g., DDIM) to discrete state spaces by developing explicit probability-flow ODE analogues remains unsolved (Sahoo et al., 12 Jun 2025).
• Score Matching Parameterizations: Improved neural architectures and parameterizations for more expressive, low-variance score estimation or drift prediction in complex discrete domains are an active topic (Zhu et al., 27 Oct 2025).
• Extensions to Graphs and Structures: Uniformization over nontrivial combinatorial objects, such as graphs or sets, is theoretically supported but little explored practically.
• Theoretical Analysis of Distillation/Self-Correction: Quantifying the convergence properties and statistical efficiency of few-step distillation and bias–variance trade-offs in curriculum strategies is ongoing (Sahoo et al., 12 Jun 2025).
• Alternative Noise Schedules and Kernels: While uniform-state kernels are attractive for their symmetry, alternatives that balance structure retention and full mixing may yield practical gains in data types with strong local dependencies (Austin et al., 2021).

A plausible implication is that unifying continuous, discrete, and semiparametric USDM frameworks will further bridge the performance and scalability gap between diffusion-based and autoregressive generative models in discrete data modeling.

• (Chen et al., 12 Feb 2024) Convergence Analysis of Discrete Diffusion Model: Exact Implementation through Uniformization
• (Bu et al., 2020) Diffusion Copulas: Identification and Estimation
• (Pauline et al., 4 Dec 2025) Foundations of Diffusion Models in General State Spaces: A Self-Contained Introduction
• (Sahoo et al., 12 Jun 2025) The Diffusion Duality
• (Zhu et al., 27 Oct 2025) Simple Denoising Diffusion LLMs
• (Austin et al., 2021) Structured Denoising Diffusion Models in Discrete State-Spaces
• (Jolicoeur-Martineau et al., 2023) Diffusion models with location-scale noise