Denoising Generative Models

• Denoising generative models are a rigorous framework that uses coupled noising and reverse processes to recover clean data from structured corruption.
• They leverage techniques such as score matching, Langevin dynamics, and Itô SDEs to accurately model complex, high-dimensional data distributions.
• Practical implementations include denoising autoencoders, diffusion models, and transformer-based architectures, achieving state-of-the-art results in image synthesis and inverse problems.

Denoising generative models constitute a mathematically rigorous and empirically successful framework for data generation, density estimation, and Bayesian inference in high-dimensional spaces. The core idea is to introduce structured noise (often Gaussian, but also generalizable to non-Gaussian processes) into data and learn, directly or indirectly, the conditional distribution—or its associated score field—that allows mapping corrupted instances back toward their clean precursors. The broad family includes denoising autoencoders, score-based diffusion models, and restoration-based generative frameworks, achieving state-of-the-art results across image synthesis, inverse problems, and scientific inference.

1. Mathematical Foundations of Denoising Generative Models

At their heart, denoising generative models rely on the definition of two coupled stochastic processes: a forward (noising) Markov chain transforming data into noise, and a reverse (denoising) process mapping samples from noise back to data. In the discrete-time setting, the forward diffusion process is typically defined as

$$q(x_{1:T}\mid x_0) = \prod_{t=1}^T q(x_t\mid x_{t-1}), \quad q(x_t\mid x_{t-1}) = \mathcal{N}(\sqrt{1-\beta_t} x_{t-1}, \beta_t I)$$

where $\{\beta_t\}_{t=1}^T$ is a schedule of small variances and $x_0 \sim p_\text{data}$ is a sample from the data distribution (Deja et al., 2022).

The reverse process, parameterized by neural networks, attempts to approximate the time-reversed chain,

$$p_\theta(x_{t-1}\mid x_t) = \mathcal{N}(\mu_\theta(x_t, t), \Sigma_\theta(x_t, t)),$$

with $\theta$ optimized via variational lower bounds (ELBO) that decompose into tractable conditional Kullback-Leibler divergences between forward and learned backward transitions (Deja et al., 2022, Benton et al., 2022).

Continuous-time analogs are typically constructed as Itô SDEs,

$$d\mathbf{x}_t = f(t, \mathbf{x}_t)dt + g(t)d\mathbf{w}_t,$$

with the reverse SDE involving the current score of the marginal density, $$\mathbf{s}_\theta(\mathbf{x}, t) = \nabla_{\mathbf{x}} \log p_t(\mathbf{x})$$ (Zhong et al., 14 Oct 2025).

A central analytical technique is denoising score matching, whereby the score function of the target density (or its Gaussian convolution) is estimated. This enables direct construction of sampling algorithms via Langevin dynamics or by simulating the reverse SDE (Block et al., 2020, Vargas et al., 2023).

2. Denoising Objectives, Score Estimation, and Theoretical Guarantees

The denoising autoencoder (DAE) objective, for $X \sim p$ and corruptions $\varepsilon \sim N(0, \sigma^2 I)$, is defined as

$$L_{\mathrm{DAE}}(r) = \mathbb{E}_{X, \varepsilon} \|r(X + \varepsilon) - X\|^2.$$

Minimization yields $$r^*(y) = y + \sigma^2 \nabla_y \log p_{\sigma^2}(y)$$, where $p_{\sigma^2}$ is the Gaussian-smoothed density. The corresponding score-matching objective is

$$L_{\mathrm{DSM}}(s) = \mathbb{E}_{Y \sim p_{\sigma^2}} \|s(Y) - \nabla \log p_{\sigma^2}(Y)\|^2,$$

with the equivalence $s(y) = (r(y) - y)/\sigma^2$ at the optimum (Block et al., 2020, Loaiza-Ganem et al., 2022).

For continuous and discrete spaces, or more general Feller Markov processes (including manifold and combinatorial structures), a generalized score-matching and ELBO framework can be constructed, unifying diffusion-based and other denoising generative models (Benton et al., 2022).

Sampling from the learned model can be accomplished via overdamped Langevin dynamics with plug-in score estimates, with non-asymptotic convergence guarantees in Wasserstein and MMD distances established for finite-sample score estimation (Block et al., 2020, Vargas et al., 2023).

Recent work on the regularity of data reveals that the optimal denoiser (Tweedie's estimator) is not always the minimum MSE estimator. For regular densities, a "half-denoiser" using only half the Tweedie correction attains lower Wasserstein and MMD error than full denoising, while for singular/low-dimensional or Dirac-support data, full denoising is required for support recovery and overcomes the curse of dimensionality (Beyler et al., 17 Mar 2025).

3. Model Structures, Architectural Variants, and Extensions

Denoising generative models admit considerable architectural variability. The classical choice is a fully convolutional U-Net acting either in pixel or latent space. Transformer-based architectures have recently been shown to be effective, especially when large patch sizes and direct clean-image prediction ("x-prediction") leverage the data manifold structure (Li et al., 17 Nov 2025).

The choice of predictive target (noise, clean data, or "velocity") in training loss significantly impacts performance, particularly in high-dimensional settings where only x-prediction avoids catastrophic degradation due to off-manifold expansion (Li et al., 17 Nov 2025). The model may also operate in latent spaces, with an explicit generator–denoiser split beneficial for stability, transfer, and interpretability (Deja et al., 2022).

Non-isotropic noise models, heavy-tailed score-matched noise kernels, and non-Gaussian forward processes (e.g., Gamma, Poisson) have been developed to accommodate domain-specific noise characteristics and overcome limitations of the isotropic Gaussian assumption (Voleti et al., 2022, Deasy et al., 2021, Xie et al., 2023). Theoretical results confirm that the denoising score-matching objective holds for both Gaussian and generalized normal/laplace-like noise kernels, provided weak regularity conditions—specifically the almost-everywhere differentiability of the noise distribution (Deasy et al., 2021).

Restoration-based generative models, formulated as MAP estimation with implicit and explicit learned priors, bridge the gap between traditional image restoration and generative modeling, supporting multi-scale and arbitrary forward degradations at low computational cost (Choi et al., 2023).

4. Practical Algorithms, Training Procedures, and Sampling

Training denoising generative models typically proceeds via stochastic minimization of the denoising or score-matching loss over mini-batches, randomly sampling noise levels or forward chain steps per sample.

A variety of sampling methods exist:

• Diffusion Reversal: Discrete or continuous reverse dynamics (SDE/ODE), conditioned on learned denoising or score fields (Vargas et al., 2023, Benton et al., 2022).
• Langevin Dynamics: Plug-in discrete-time Markov chains using estimated scores, with non-asymptotic error bounds in Wasserstein distance (Block et al., 2020).
• Denoising GANs: Replacing Gaussian reverse kernels with learned multimodal (GAN-based) transitions enables dramatic reduction in the number of required sampling steps (from $\sim$1000 to as few as 4–8) (Xiao et al., 2021).
• One-Step Distillation: Denoising score distillation (DSD) compresses a diffusion model (even when trained on noisy-only data) into a single-step generator, improving quality and speed via implicit regularization of the generated covariance (Chen et al., 10 Mar 2025).
• Plug-and-Play Priors: Generative priors learned in this framework can be used as proximal operators in classical inverse problem solvers, with effective results shown on denoising, super-resolution, and colorization (Choi et al., 2023, Cardoso et al., 2023).

Network architecture, output space choice (predicting clean or noised data), and training loss parameterization have substantial impact on stability, sample quality, and scaling to high dimensions (Li et al., 17 Nov 2025).

5. Empirical Results, Limitations, and Guidance

Denoising generative models achieve state-of-the-art sample quality and mode coverage on benchmarks such as ImageNet (e.g., FID<2 in pixel space using large-patch Transformers with x-prediction) (Li et al., 17 Nov 2025), and massive acceleration via step-reduction (up to $\sim$2000$\times$) without loss of fidelity when using GAN-based (multimodal) denoising kernels (Xiao et al., 2021). Denoising models trained with only noisy data, when distilled via DSD, achieve both higher quality and much faster generation than the original teacher (Chen et al., 10 Mar 2025).

Empirical investigations also reveal:

• Mild or even negative effects when noise-conditioning is omitted in modern diffusion/score-based architectures, except in strictly deterministic ODE samplers where error accumulates without stochasticity; noise-unconditional training is often robust for high-dimensional data under smooth sampling schedules (Sun et al., 18 Feb 2025).
• Model choice of denoiser parameter $\alpha$ (full vs. half correction) should be carefully matched to the regularity of the data; hybrid $\alpha$ or adaptive schedules may be needed in practice (Beyler et al., 17 Mar 2025).
• Restoration-based models with explicit learned priors and multi-scale (super-resolution) degradations match or outperform standard diffusion samplers at a tiny fraction of the cost (Choi et al., 2023).
• Bayesian inference in complex structured domains (e.g., ECG signal recovery, mixture models, non-Euclidean manifolds) can be achieved by pairing the denoising generative prior with SMC or MCMC-based posterior sampling, retaining uncertainty quantification and downstream task performance (Cardoso et al., 2023, Benton et al., 2022).

A representative table of denoising strategy effectiveness (for VAE and flow models, FID$\downarrow$ on MNIST/SVHN/CIFAR-10) is summarized below (Loaiza-Ganem et al., 2022):

Model	MNIST	FMNIST	SVHN	CIFAR-10
VAE	197.4	188.9	311.5	270.3
ND-VAE	199.9	185.7	317.8	264.5
TD-VAE	199.1	190.4	310.9	263.9
CD-VAE	197.4	195.8	290.0	262.4
Flow	137.2	110.5	231.9	222.7
ND-Flow	103.2	72.3	222.0	222.9
TD-Flow	105.6	70.6	224.2	222.8
CD-Flow	87.4	73.3	206.0	225.4

Notably, simply adding noise often improves performance, but denoising corrections do not universally yield improvements, highlighting the need for data- and architecture-aware design.

6. Extensions, Challenges, and Future Directions

Denoising generative models are being extended along multiple axes:

• Support for generic noise models (Gamma, Poisson, general Markov kernels), non-Euclidean domains, and manifold structures (Xie et al., 2023, Benton et al., 2022).
• Theoretical analysis of convergence, expressivity, and finite-sample errors, including new approximation results for the Föllmer drift and Schrödinger bridge connections (Vargas et al., 2023).
• Addressing practical challenges such as "noise shift" (mismatch between pre-defined and realized noise during sampling), which can be mitigated by explicit noise-awareness guidance terms that ensure sampler trajectories remain consistent with intended schedules (Zhong et al., 14 Oct 2025).
• Robustness to high-dimensional capacity mismatch, guiding architectural choices (x-prediction, bottlenecks, elimination of pre-conditioning) as key for scaling diffusion models to large-image or structured data (Li et al., 17 Nov 2025).
• Plug-and-play integration of generative models as priors in Bayesian and inverse-problem pipelines, including ECG, MRI, and other scientific domains (Cardoso et al., 2023, Choi et al., 2023).

Open research directions include learning or adapting forward process parameters, optimal denoiser schedules (potentially data-driven adaptive $\alpha$), theoretical calibration of estimator regularity for wisest denoiser selection, and broadening generative application domains beyond images to science, language, and general probabilistic inference.

7. References to Key Works

• Generalized DAEs and consistency of pseudo-Gibbs chains for arbitrary corruption and reconstruction: (Bengio et al., 2013)
• Score-matched and restoration-based models: (Block et al., 2020, Choi et al., 2023, Benton et al., 2022)
• Optimal denoising strategies (full vs. half denoising): (Beyler et al., 17 Mar 2025)
• Heavy-tailed score matching: (Deasy et al., 2021)
• Diffusion GANs for accelerated sampling: (Xiao et al., 2021)
• Transformer-based pixel diffusion and the case for x-prediction: (Li et al., 17 Nov 2025)
• Noise conditioning analysis: (Sun et al., 18 Feb 2025)
• Handling noise shift during sampling: (Zhong et al., 14 Oct 2025)
• Extensions to domain-specific noise/forward models: (Xie et al., 2023, Voleti et al., 2022)
• Denoising score distillation for noisy-data training: (Chen et al., 10 Mar 2025)
• Bayesian and inverse problem applications: (Cardoso et al., 2023)

Denoising generative models thus form a cohesive, extensible, and deeply analyzed family of probabilistic models, whose theoretical foundations and algorithmic refinements are rapidly evolving and informing the state of modern generative methodology.