Denoising Score Matching Loss

• Denoising Score Matching (DSM) is a loss function that learns energy-based and score-based models by matching gradients from noise-corrupted data to analytically tractable Gaussian scores.
• DSM reduces computational complexity by bypassing intractable Hessian terms and leveraging noise-conditional score networks with multi-scale noise schedules.
• DSM underpins state-of-the-art generative diffusion models and robust inverse problem solvers, offering scalable surrogate objectives with strong theoretical guarantees.

Denoising Score Matching (DSM) is a statistical learning paradigm for fitting unnormalized density models—especially energy-based models (EBMs) and score-based diffusion models—via regression onto analytically tractable scores after explicit noise injection. DSM delivers a scalable surrogate to classical Fisher-divergence-based score matching, sidestepping intractable terms involving Hessian traces by comparing the learned model's score field to an explicit conditionally known reference under fixed additive Gaussian noise. Its adoption underpins modern generative diffusion modeling, robust inverse problem solvers, and deep energy-based purification pipelines, while also motivating significant theoretical advances in sample complexity, optimization guarantees, and extensions to latent manifolds.

1. Formal Definition and Mathematical Structure

Let $p_d(x)$ be a target density on $\mathbb{R}^D$, and $q_\theta(x)$ a parameterized or implicitly-defined energy-based model. To avoid the computational intractability of the classical Fisher-divergence-based score matching objective, DSM introduces isotropic Gaussian corruption: $$p(\tilde x \mid x) = \mathcal{N}(\tilde x; x, \sigma^2 I).$$ Define the noised marginal

$$\tilde p_d(\tilde x) = \int p_d(x) p(\tilde x \mid x) \mathrm{d}x.$$

The DSM loss is the Fisher divergence between the noised data law $\tilde p_d$ and the equally noised model law $\tilde q_\theta$, which—for Gaussian corruption—admits

$$\mathcal{L}_\mathrm{DSM}(\theta) = \frac{1}{2} \mathbb{E}_{x\sim p_d,\,\tilde x\sim p(\tilde x \mid x)} \left\| \nabla_{\tilde x} \log p(\tilde x \mid x) - s_{q_\theta}(\tilde x) \right\|_2^2,$$

where $$s_{q_\theta}(\tilde x) = \nabla_{\tilde x} \log q_\theta(\tilde x)$$.

For Gaussian noise, $$\nabla_{\tilde x}\log p(\tilde x \mid x ) = (x-\tilde x)/\sigma^2$$, so the loss reduces to

$\mathbb{R}^D$0

with $\mathbb{R}^D$1.

DSM is most commonly implemented with a noise-conditional score network, and modern treatments integrate over a prescribed noise-level schedule $\mathbb{R}^D$2, e.g., log-uniform on $\mathbb{R}^D$3 (Zhang et al., 2023, Yoon et al., 2021, Jolicoeur-Martineau et al., 2020).

2. Relationship to Fisher Divergence and Classical Score Matching

Classical score matching seeks to fit the score field $\mathbb{R}^D$4 directly via Fisher divergence: $\mathbb{R}^D$5 This requires evaluating a Hessian trace involving model parameters, which becomes cubic in $\mathbb{R}^D$6, imposing computational barriers.

Vincent's denoising formulation circumvents this by convolving both density and model with Gaussian noise and matching the smoothed scores. Under mild smoothness,

$\mathbb{R}^D$7

where $\mathbb{R}^D$8 is the Gaussian kernel (Zhang et al., 2023). As $\mathbb{R}^D$9, the original score matching objective is recovered, but in this singular limit the gradients diverge—a crucial practical consideration.

3. Formal Inconsistency at Fixed-Noise and Theoretical Resolution

For any fixed $q_\theta(x)$0, DSM is inconsistent for $q_\theta(x)$1: minimizing the loss only guarantees that the learned model matches the noised data distribution, not the clean data law. Specifically,

$q_\theta(x)$2

so that

$q_\theta(x)$3

a convolution with the Gaussian kernel. The model thus corresponds to a blurred version of the true data density. Inverting this convolution to recover $q_\theta(x)$4 is intractable in general for high-dimensional data (Zhang et al., 2023).

Zhang et al. propose a practical two-stage workaround: train at fixed $q_\theta(x)$5 (accepting the approximate nature), then sample from $q_\theta(x)$6 using moment-matching Gibbs sampling, exploiting Tweedie’s formula to recover the posterior mean and covariance. This procedure targets the true data law despite DSM’s intrinsic training-time inconsistency.

4. Algorithmic Implementation and Extensions

The DSM loss admits highly efficient stochastic minibatch estimation: $q_\theta(x)$7 with $q_\theta(x)$8, and $q_\theta(x)$9 sampled from a prescribed schedule (Yoon et al., 2021, Kobler et al., 2023, Jolicoeur-Martineau et al., 2020).

Modern practice prefers a multi-scale, noise-conditional approach:

• Draw $$p(\tilde x \mid x) = \mathcal{N}(\tilde x; x, \sigma^2 I).$$0 noise levels $$p(\tilde x \mid x) = \mathcal{N}(\tilde x; x, \sigma^2 I).$$1, typically in geometric progression.
• For each batch item, sample $$p(\tilde x \mid x) = \mathcal{N}(\tilde x; x, \sigma^2 I).$$2, $$p(\tilde x \mid x) = \mathcal{N}(\tilde x; x, \sigma^2 I).$$3, and $$p(\tilde x \mid x) = \mathcal{N}(\tilde x; x, \sigma^2 I).$$4.
• Reweight by $$p(\tilde x \mid x) = \mathcal{N}(\tilde x; x, \sigma^2 I).$$5 (balancing importance across scales) (Yoon et al., 2021).

Extensions and generalizations include:

• Structured (non-diagonal) covariance forward processes (“Whitened Score”), avoiding matrix inversion (Alido et al., 15 May 2025).
• SURE-Score: joint denoising and score learning from noisy data via the SURE principle with explicit divergence estimation (Aali et al., 2023).
• Self-supervised and manifold DSMs (Rao–Blackwellized, GDSM), crucial when fully clean data or ambient access is unavailable (Tu et al., 8 May 2025, Rawal, 25 May 2026).
• High-order DSM to control ODE log-likelihood gaps in score-based diffusion modeling (Lu et al., 2022).

5. Statistical Properties, Limitations, and Sample Complexity

DSM is statistically advantageous compared to classical score matching, especially for multimodal distributions or data concentrated on low-dimensional manifolds. Diffusion-based DSM achieves near-optimal estimation rates in the intrinsic data dimension, not the ambient space, overcoming the curse of dimensionality (Yakovlev et al., 30 Dec 2025). Under suitable assumptions, both implicit and denoising score matching achieve minimax rates

$$p(\tilde x \mid x) = \mathcal{N}(\tilde x; x, \sigma^2 I).$$6

where $$p(\tilde x \mid x) = \mathcal{N}(\tilde x; x, \sigma^2 I).$$7 is the manifold dimension and $$p(\tilde x \mid x) = \mathcal{N}(\tilde x; x, \sigma^2 I).$$8 the regularity.

Limits include:

• Irregularity and memorization: In the low-noise regime, the empirical DSM minimizer becomes highly oscillatory (sharp transitions between clusters), causing memorization of the training set. Large learning rates act as an implicit regularizer, preventing full memorization (Wu et al., 5 Feb 2025).
• Hyperparameter dependence: The choice of noise-level schedule and (in advanced forms) weighting function can significantly impact convergence and gradient variance. Heuristic weighting ($$p(\tilde x \mid x) = \mathcal{N}(\tilde x; x, \sigma^2 I).$$9) is widely used and often optimal in practice (Zhang et al., 3 Aug 2025).
• Blurring at fixed-noise: The default DSM estimator always fits the Gaussian-blurred density, not the clean law; explicit sampling corrections or refined deconvolution are required for generative applications (Zhang et al., 2023).

6. Practical Impact, Applications, and Extensions

DSM is foundational in state-of-the-art score-based generative models, denoising diffusion probabilistic models, and recent energy-based purification pipelines. Notable applications include:

• Score-based generative modeling: DSM underpins SDE- and ODE-based samplers, achieving state-of-the-art sample quality across imaging benchmarks (Lu et al., 2022, Jolicoeur-Martineau et al., 2020).
• Inverse problems: SURE-Score and GDSM enable training from only noisy/partial observations and permit self-supervised learning in MRI and channel estimation (Aali et al., 2023, Tu et al., 8 May 2025).
• Robustness and adversarial purification: DSM-trained EBMs provide fast purification outperforming MCMC-based methods (Yoon et al., 2021).
• Estimating local intrinsic dimension, learning on manifolds: DSM loss gives a tight lower bound to the local intrinsic dimension, and advances in manifold DSM enable efficient, bias-corrected modeling of densities on submanifolds (Yeats et al., 14 Oct 2025, Rawal, 25 May 2026).

7. Theoretical Developments and Ongoing Research

Current research focuses on:

• Theoretical guarantees of ODE-based and diffusion-based generative samplers, relying on the quality of DSM-trained scores and their Hessians (Yakovlev et al., 30 Dec 2025).
• Alternative losses such as Target Score Matching (TSM), which interpolate between DSM and direct regression on the clean score to handle low-noise regimes where DSM's variance explodes (Bortoli et al., 2024).
• Statistical error bounds via advanced concentration inequalities and Rademacher complexity for stochastic optimization under unbounded loss, formalizing uniform laws of large numbers for DSM (Birrell, 12 Feb 2025).
• Rigorous characterization of the generalization–memorization tradeoff in randomized feature models, revealing that oversampling noise can provoke memorization even without overparameterization (George et al., 1 Feb 2025).
• High-order DSM (second and third moment-matching losses) to close the log-likelihood gap in score-based diffusion models and empirically improve density modeling (Lu et al., 2022).

DSM's algorithmic versatility, tractable estimation, and flexible extensions render it fundamental in the statistical learning of complex high-dimensional data distributions, with significant ongoing influence on both theory and high-impact applications.

---

References:

• (Zhang et al., 2023) “Moment Matching Denoising Gibbs Sampling”
• (Alido et al., 15 May 2025) “Whitened Score Diffusion: A Structured Prior for Imaging Inverse Problems”
• (Yoon et al., 2021) “Adversarial purification with Score-based generative models”
• (Yakovlev et al., 30 Dec 2025) “Implicit score matching meets denoising score matching: improved rates of convergence and log-density Hessian estimation”
• (Aali et al., 2023) “Solving Inverse Problems with Score-Based Generative Priors learned from Noisy Data”
• (Tu et al., 8 May 2025) “Score-based Self-supervised MRI Denoising”
• (Zhang et al., 3 Aug 2025) “Why Heuristic Weighting Works: A Theoretical Analysis of Denoising Score Matching”
• (George et al., 1 Feb 2025) “Denoising Score Matching with Random Features: Insights on Diffusion Models from Precise Learning Curves”
• (Lu et al., 2022) “Maximum Likelihood Training for Score-Based Diffusion ODEs by High-Order Denoising Score Matching”
• (Kobler et al., 2023) “Learning Gradually Non-convex Image Priors Using Score Matching”
• (Wu et al., 5 Feb 2025) “Taking a Big Step: Large Learning Rates in Denoising Score Matching Prevent Memorization”
• (Yeats et al., 14 Oct 2025) “A Connection Between Score Matching and Local Intrinsic Dimension”
• (Bortoli et al., 2024) “Target Score Matching”
• (Schwienhorst et al., 21 May 2026) “Diffusion-based Denoising Beats Vanilla Score Matching in Parameter Estimation: A Theoretical Explanation”
• (Rawal, 25 May 2026) “Rao-Blackwellized Score Matching on Manifolds”
• (Olga et al., 2021) “Denoising Score Matching with Random Fourier Features”
• (Thiry et al., 2024) “Classification-Denoising Networks”
• (Jolicoeur-Martineau et al., 2020) “Adversarial score matching and improved sampling for image generation”