Uniform Diffusion Models (UDM)

• Uniform Diffusion Models (UDM) are generative models that apply uniform corruption kernels to progressively convert data into noise and recover it via parameterized reverse dynamics.
• In discrete settings, UDMs enable parallel token sampling and leverage leave-one-out denoising to achieve competitive scaling and performance on language and structured data tasks.
• In continuous domains, UDMs employ additive uniform noise, yielding invariant reverse dynamics that support efficient text-to-image generation and other modality applications.

Uniform Diffusion Models (UDM) are a family of generative models that implement stochastic data-to-noise and noise-to-data transitions, using a uniform corruption kernel on discrete or continuous state spaces. UDMs have emerged as a central paradigm in both large-scale discrete generative modeling (notably for language and structured data) and as a theoretical alternative to traditional Gaussian-based continuous diffusion models. In the discrete case, UDMs have enabled directly parallel sampling, tractable reverse posteriors, and competitive scaling properties for LLMs. In the continuous case, they exemplify the broad class of non-Gaussian, non-normal noise distributions, providing insights into invariance properties of diffusion-based generative modeling.

1. Mathematical Formulation and Forward Process

In the discrete setting, UDMs operate on a finite alphabet 𝒱 of size $K$, with data $x_0 \in \mathcal{V}^D$, where $D$ is the sequence or data dimension. The forward process is defined as a time-indexed Markov chain:

$$q(x_t \mid x_{t-1}) = (1-\beta_t)\mathbf{1}[x_t = x_{t-1}] + \frac{\beta_t}{K}\mathbf{1}[x_t \neq x_{t-1}]$$

where $\beta_t \in (0,1)$ is a user-specified (possibly time-dependent) noise rate. At each timestep, each token is independently preserved with probability $1-\beta_t$ or replaced by a uniformly random symbol with probability $\beta_t$ (Wang et al., 20 Apr 2026).

The marginal law after $t$ steps is of the form:

$$q(x_t \mid x_0) = \alpha_t \mathbf{1}[x_t = x_0] + (1-\alpha_t)\,\mathrm{Uniform}_K(x_t)$$

for

$\alpha_t = \prod_{s=1}^t (1-\beta_s)$

This uniform corruption process is symmetric and treats all tokens identically, in contrast to masked or structured noise kernels.

In continuous data spaces, UDMs implement additive noise with uniform increments, i.e.,

$x_0 \in \mathcal{V}^D$0

with corresponding variance constraint and transitions (Jolicoeur-Martineau et al., 2023, Li, 2024).

2. Reverse Denoising Dynamics and Sampling

The core learning task is to parameterize and learn the reverse chain, which reconstructs clean data from uniform noise. In the discrete case, this is a categorical distribution parameterized by a neural network over token positions:

$x_0 \in \mathcal{V}^D$1

where $x_0 \in \mathcal{V}^D$2 are per-token logits.

The true reverse posterior has an explicit Bayesian form due to the uniformity of transitions. For $x_0 \in \mathcal{V}^D$3:

$x_0 \in \mathcal{V}^D$4

Common parameterizations include direct categorical outputs, clean-token proxies, and, as shown in recent work, leave-one-out (LOO) denoisers that predict $x_0 \in \mathcal{V}^D$5 at position $x_0 \in \mathcal{V}^D$6 conditioned on all other positions in $x_0 \in \mathcal{V}^D$7 except $x_0 \in \mathcal{V}^D$8 (Gourevitch et al., 21 May 2026).

Sampling proceeds by drawing $x_0 \in \mathcal{V}^D$9 from the reverse kernel, recursively for $D$0, with $D$1 initialized as pure uniform noise.

In continuous domains, the score-based or moment-matching framework for UDMs may predict the mean (and sometimes variance) of the noise innovation, with reverse transitions implemented via parameterized uniform (or mixed) distributions (Jolicoeur-Martineau et al., 2023, Li, 2024).

3. Objective Functions and Losses

UDMs are trained primarily through variational inference, maximizing an evidence lower bound (ELBO) which—under uniform transitions— admits substantial simplifications. The general objective is:

$D$2

For discrete UDMs, the cross-entropy of the clean data token at each position with the predicted denoiser is sufficient and, when restricted to only genuinely corrupted positions, recovers the ELBO up to a constant shift (Zhu et al., 27 Oct 2025). Recent work demonstrates that in uniform discrete settings, the LOO parameterization aligns the plug-in reverse kernel with the ELBO, while the standard per-token denoising loss targets the true denoiser, revealing a gap with implications for both generative quality and sampling efficiency (Gourevitch et al., 21 May 2026).

In the continuous uniform setting, the loss is typically a method-of-moments MSE between the true and predicted uniform noise, as score-matching techniques are not directly applicable due to the constant interior score field (Jolicoeur-Martineau et al., 2023).

4. Algorithmic Properties, Scaling, and Sampling Strategies

UDMs support parallel generation since each token is updated independently. Uniform transitions simplify the transition matrices and favor highly parallel hardware implementations—every token can be processed without order-dependence at each step (Wang et al., 20 Apr 2026, Ye et al., 17 Jun 2026). Sampling can leverage standard ancestral decoders or adaptive selection heuristics:

• Ancestral sampling: At each reverse timestep, every token is sampled from its predicted categorical distribution.
• Confidence/adaptive sampling: Select a subset of positions with the largest logit differences to update, inducing an emergent commitment order and yielding practical speedups in generation (Ye et al., 17 Jun 2026).
• Absorbing-state reformulation: UDM can be re-expressed via an auxiliary variable $D$3 assigning each token a “target” state, allowing UDM to be simulated via masked-diffusion-like dynamics with per-token carryover and remasking (Gourevitch et al., 21 May 2026).

The UDM scaling laws reveal distinct properties: UDMs are more parameter-efficient and less data-efficient than masked diffusions—optimal model size scales as $D$4 and optimal data as $D$5 (at compute $D$6) (Rütte et al., 11 Dec 2025, Sahoo et al., 16 Feb 2026). UDMs thus display favorable scaling in data-limited regimes and parity with AR models at very large scale. Notably, the 10B-parameter UDM achieves validation ELBO matching state-of-the-art AR trends (Rütte et al., 11 Dec 2025).

5. Empirical Performance and Comparative Results

Discrete UDMs at scale (e.g., Sumi-7B, 1.5T tokens) achieve competitive results with AR and masked diffusion models on knowledge, reasoning, and (especially) code synthesis benchmarks, outperforming on tasks aligned with their data mixture and generation paradigm (Ye et al., 17 Jun 2026). On MMLU, RACE, and TruthfulQA, Sumi-7B matches or outperforms Falcon-7B and Llama 2-7B at similar budgets. In mathematical reasoning (GSM8K, BBH) and coding (HumanEval), UDMs exhibit particular strength.

In downstream text-to-image generation, UDM-GRPO (UDM with Group Relative Policy Optimization for RL-based reward maximization) sets a new benchmark with GenEval accuracy rising from 69% to 96% on compositional reasoning, and OCR accuracy from 8% to 57% (Wang et al., 20 Apr 2026). UDM-GRPO resolves prior RL instability via two algorithmic insights: using the final clean sample as the action and reconstructing the forward process trajectory, which stably aligns reward signals with pretraining.

On standard image generation (e.g., CIFAR-10), uniform-noise DMs (continuous UDM) have nearly identical FID to Gaussian models ($D$7 vs $D$8) but achieve slightly higher bits-per-dimension ($D$9 vs $$q(x_t \mid x_{t-1}) = (1-\beta_t)\mathbf{1}[x_t = x_{t-1}] + \frac{\beta_t}{K}\mathbf{1}[x_t \neq x_{t-1}]$$0), with samples exhibiting blockier texture (Li, 2024). In purely discrete settings, UDMs have shown competitive generative perplexity and diversity, with contrastive-denoising enhancements yielding further improvements (Zhu et al., 27 Oct 2025).

6. Theoretical Considerations, Limitations, and Advances

A core theoretical result is the invariance principle for continuous UDMs: as the step size $$q(x_t \mid x_{t-1}) = (1-\beta_t)\mathbf{1}[x_t = x_{t-1}] + \frac{\beta_t}{K}\mathbf{1}[x_t \neq x_{t-1}]$$1, the reverse-time SDE becomes invariant to the increment distribution, meaning Gaussian and uniform increments yield identical limiting reverse dynamics (Li, 2024). Thus, in the small-step regime, the choice of noise law can be viewed as a free parameter for regularization or sample diversity.

For discrete UDMs, a central insight is the distinction between denoiser and leave-one-out parameterizations when optimizing the ELBO. Masked diffusion (absorbing state) enjoys structural alignment between cross-entropy and ELBO, but uniform diffusion requires LOO dependency analysis to ensure correct generative dynamics. Absorbing-state reformulations (AUDM) allow UDM to match, and sometimes surpass, masked diffusion on standard LM tasks (Gourevitch et al., 21 May 2026).

Limitations include higher inference cost (requiring $$q(x_t \mid x_{t-1}) = (1-\beta_t)\mathbf{1}[x_t = x_{t-1}] + \frac{\beta_t}{K}\mathbf{1}[x_t \neq x_{t-1}]$$2 steps, though parallelizable) and slightly degraded performance in commonsense-heavy benchmarks when compared to AR models. UDMs may lag behind masked diffusion in very small-scale or data-rich pretraining regimes. Boundary artifacts may occur in continuous-data UDM with uniform noise due to the lack of heavy tails (Jolicoeur-Martineau et al., 2023).

7. Applications, Extensions, and Open Problems

UDMs are now established as a viable foundation for large-scale language modeling, text-to-image generation, discrete data synthesis, and serve as testbeds for architectural and algorithmic advances such as adaptive sampling, self-correction, and RL reward optimization (Ye et al., 17 Jun 2026, Wang et al., 20 Apr 2026).

Future directions include optimal tradeoffs for number of denoising steps $$q(x_t \mid x_{t-1}) = (1-\beta_t)\mathbf{1}[x_t = x_{t-1}] + \frac{\beta_t}{K}\mathbf{1}[x_t \neq x_{t-1}]$$3 and parallel update cardinality $$q(x_t \mid x_{t-1}) = (1-\beta_t)\mathbf{1}[x_t = x_{t-1}] + \frac{\beta_t}{K}\mathbf{1}[x_t \neq x_{t-1}]$$4, more sample-efficient fine-tuning under complex reward schedules, and principled combinations of UDM with other noise schedules (e.g., interpolating, masked-uniform hybrids) (Rütte et al., 11 Dec 2025, Sahoo et al., 16 Feb 2026). Understanding context-length scaling, developing efficient predictor-corrector samplers, and closing the empirical gap in certain benchmark domains remain open.

A table summarizing distinguishing properties of uniform, masked, and Gaussian diffusion setups appears below:

Property	Discrete UDM	Masked Diffusion	Gaussian Diffusion
Forward kernel	Uniform corruption	Absorbing w/ [MASK]	Additive Gaussian
Reverse parameterization	Denoiser, LOO, plug-in	Denoiser, plug-in (coincide)	Score-based
Exact posterior	Closed-form	Closed-form	Closed-form
Sampling dynamics	Fully parallelizable	Mask-parallelizable	Serial, ODE/Score-based
Empirical scaling	Parameter-optimal, data-poor	Data-optimal at small scale	Data-optimal, continuous
Limiting SDE/CTMC	Markov chain w/ uniform rates	Masked chain	Itô SDE

This comprehensive theoretical and empirical body establishes UDMs as a foundational approach in both discrete and continuous generative modeling, with ongoing research focusing on parameterization, sampling strategies, scaling laws, and application to new modalities (Pauline et al., 4 Dec 2025, Ye et al., 17 Jun 2026, Gourevitch et al., 21 May 2026).