---
title: Multiscale Denoising Score Matching
url: https://www.emergentmind.com/topics/multiscale-denoising-score-matching
type: topic
---

# Multiscale Denoising Score Matching

Multiscale denoising score matching is the class of score-based learning procedures in which data are perturbed by multiple noise levels and a model is trained to represent the score, denoised estimate, or an equivalent energy/log-density quantity of the corresponding smoothed distributions. In its canonical form, one corrupts data with Gaussian noise at a ladder of scales, learns a noise-conditioned model, and then traverses those scales from coarse to fine during generation or restoration. In high dimension, the multiscale construction is motivated by the fact that single-noise denoising score matching learns only on a thin shell around the data manifold, whereas multiple scales distribute supervision over both ambient space and structural scale [1910.07762]. Subsequent work established that the effectiveness of this framework depends not only on the training objective but also on sampler consistency, final denoising readout, and the handling of few-step or variable-step annealing schedules [2009.05475, 2104.03725].

## 1. Canonical formulation

The standard multiscale setting begins with Gaussian corruption
\[
q_\sigma(\tilde{x}\mid x)=\mathcal{N}(\tilde{x}\mid x,\sigma^2 I),
\]
a clean-data distribution \(p(x)\), and either a discrete ladder \(\{\sigma_i\}_{i=1}^L\) or a continuum of scales. A score network \(s_\theta(\tilde{x},\sigma)\) is then trained to estimate the score of the perturbed data distribution \(q_\sigma(\tilde{x})\). A widely used denoising score matching objective is
\[
\frac{1}{2}\,\mathbb{E}_{p(\tilde{x},x,\sigma)} \left[ \left\| \sigma\, s_\theta(\tilde{x},\sigma)+\frac{\tilde{x}-x}{\sigma} \right\|_2^2 \right],
\]
with \(p(\tilde{x},x,\sigma)=q_\sigma(\tilde{x}\mid x)\,p(x)\,p(\sigma)\), where \(p(\sigma)\) is often uniform over a discrete set of noise levels [2009.05475].

A geometric schedule is standard:
\[
\sigma_1 > \sigma_2 > \cdots > \sigma_L,
\]
with \(\sigma_1\) large enough to cover the support of the data and \(\sigma_L\) small enough to be nearly imperceptible. In one common parameterization, the scales form a geometric progression. On CIFAR-10, one reported choice is \(\sigma_1=50\), \(\sigma_L=0.01\), \(L=232\); on LSUN-Churches, \(\sigma_1=140\), \(\sigma_L=0.01\), \(L=788\) [2009.05475]. The hierarchy here is in noise scales rather than in latent layers.

Two parameterization regimes appear in the literature. In the conditional form, the model explicitly takes \((\tilde{x},\sigma)\) as input. In an unconditional form, one can define
\[
s_\theta(\tilde{x},\sigma_i)=\frac{s_\theta(\tilde{x})}{\sigma_i},
\]
so the network is not explicitly conditioned on \(\sigma\), although the effective score remains scale-dependent. The conditional formulation is the more direct expression of multiscale denoising score matching; the unconditional formulation is analytically useful but can misestimate the score away from the harmonic mean of the noise levels [2009.05475].

A closely related energy-based formulation replaces direct score prediction by a scalar energy \(E(x;\theta)\) or \(f_\theta(x,\sigma)\). In an energy-based model,
\[
p_m(x)\propto e^{-E(x)},
\qquad
\nabla_x \log p_m(x) = -\nabla_x E(x).
\]
This makes multiscale denoising score matching compatible with explicit energy models, while preserving the denoising interpretation of the learned vector field [1910.07762].

## 2. Why multiple scales are necessary

The foundational argument for multiscale DSM in high dimension is geometric. If data concentrate near a lower-dimensional manifold \(\mathcal{X}\subset\mathbb{R}^d\), then isotropic Gaussian corruption at one fixed scale places most noisy samples on a thin shell around that manifold. Because Gaussian norm concentrates near \(\sqrt{d}\sigma\), a single-noise model learns the score primarily on that shell and says much less about regions either closer to or farther from the manifold. This is sufficient for local denoising but not for generation from random initialization [1910.07762].

From this perspective, multiple scales serve two roles. First, they fill in ambient space more continuously by overlapping shells of different radii. Second, they separate coarse and fine structure: larger noise levels capture longer-range geometry, whereas smaller noise levels recover local detail. The empirical conclusion drawn in energy-based modeling is that no single \(\sigma\) suffices for high-quality generation from scratch; low noise captures fine detail but not global structure, and high noise captures global structure but loses detail [1910.07762].

A complementary theoretical interpretation uses the energy
\[
F(y,t)=-\log p_t(y),
\]
where \(p_t\) is the data distribution after Gaussian smoothing with variance \(t\). For sufficiently large noise variance, this smoothed negative log density becomes convex under bounded-domain Gaussian-mixture assumptions. That produces a graduated non-convexity picture: large \(t\) yields coarse, easier, more convex energies, while small \(t\) recovers sharper and more non-convex structure [2302.10502]. This suggests that multiscale DSM is not only a robustness device but also a continuation mechanism.

The same large-noise/small-noise tradeoff appears in log-density estimation and anomaly detection. Small \(\sigma\) is closer to the original density but may have support too concentrated around training data; large \(\sigma\) broadens support and stabilizes estimation but distorts the target distribution. In multiscale log-density estimation for video anomaly detection, this single-\(\sigma\) choice is described as “an unwelcome compromise,” motivating a shared \(\sigma\)-conditioned model trained over a range of scales [2403.14497]. When the clean target score is known, low-noise instability can be attacked more directly: Target Score Matching replaces the usual DSM target by the clean score and is explicitly designed for favorable behavior at small noise levels [2402.08667].

## 3. Annealed sampling and scale traversal

Training across multiple noise levels is only one half of the framework; the other half is the sampler. In the standard setting, sampling starts at high noise,
\[
x_0 \sim \mathcal N(0,\sigma_1 I),
\]
and progressively refines the sample down to \(\sigma_N\) using the learned score. Three schemes are especially prominent: annealed Langevin sampling (ALS), consistent annealed sampling (CAS), and predictor-corrector (PC) sampling [2104.03725].

ALS uses
\[
x_i \leftarrow x_{i-1} + \alpha\, s_\theta(x_{i-1},\sigma_i) + \sqrt{2\alpha}\, z_i,
\qquad
z_i \sim \mathcal N(0,I),
\]
with a scale-dependent step size. In the formulation reviewed for DSM-trained models,
\[
\alpha_i = \epsilon \frac{\sigma_i^2}{\sigma_L^2},
\]
and the update can be rewritten through the denoised estimate
\[
H_\theta(x,\sigma)=x+\sigma^2 s_\theta(x,\sigma)
\]
as
\[
x \leftarrow (1-\eta)x + \eta H_\theta(x,\sigma_i) + \sqrt{2\eta}\,\sigma_i z.
\]
This makes explicit that annealed score-based sampling is also a multiscale denoising process [2009.05475].

CAS was introduced to correct a specific inconsistency in finite-step ALS. Under ALS, the sample carried from one noise level to the next may not actually have the variance assumed by the score network at that level. CAS couples denoising and re-noising more tightly:
\[
x_i \leftarrow x_{i-1} + \eta \sigma_i^2 s_\theta(x_{i-1},\sigma_i) + \beta \sigma_{i+1} z_i,
\]
with
\[
\beta = \sqrt{1-\left(\frac{1-\eta}{\gamma}\right)^2},
\qquad
\gamma=\frac{\sigma_{i+1}}{\sigma_i}.
\]
A practical stability region follows from the requirement that \(\beta\) remain real and from the denoising recursion itself:
\[
1-\gamma \le \eta \le 1.
\]
Because \(\gamma\) depends on the discretization, \(\eta\) should not be tuned independently of the number of sampling steps. A practical reparameterization is
\[
\eta = 1-\gamma^{\epsilon_c}, \qquad \epsilon_c \ge 1,
\]
which traverses the full admissible interval automatically and remains valid when the number of steps changes [2104.03725].

The boundary cases of this parameterization are informative. At \(\epsilon_c=1\), CAS becomes deterministic:
\[
x_i \leftarrow \gamma x_{i-1} + (1-\gamma)H(x_{i-1},\sigma_i),
\]
with no additional stochasticity beyond initialization. As \(\epsilon_c\to\infty\), CAS approaches
\[
x_i \leftarrow H(x_{i-1},\sigma_i) + \sigma_{i+1} z_i,
\]
a denoise-then-re-noise recursion. The case \(\epsilon_c=2\) yields an ALS-like update and also approximates the PC predictor step; in the limit \(\gamma\to 1\), the discrepancy disappears and CAS becomes the predictor part of PC [2104.03725].

An additional practical ingredient is the final denoising readout
\[
x_{\text{final}} = H_\theta(x,\sigma_L)=x+\sigma_L^2 s_\theta(x,\sigma_L),
\]
sometimes described as the expected denoised sample. This changes only the terminal readout, not the training objective, but it can substantially affect evaluation. On CIFAR-10 with the Song et al. architecture, non-adversarial non-consistent ALS at \(n_\sigma=1\) improved from FID \(36.3\) without denoising to \(13.3\) with denoising, and CAS at the same step budget gave \(14.7/12.3\) for non-denoised/denoised output [2009.05475]. The general conclusion is that multiscale DSM performance depends jointly on the learned score, the consistency of the annealing path, and the terminal denoising map.

## 4. Energy, log-density, and joint scale models

Although the most familiar formulation uses a vector-valued score network, multiscale DSM also supports scalar energy or log-density models. In one line of work, a joint energy \(R_L(y,t)\) is learned over image and variance, with
\[
\nabla_y R_L(y,t;\theta) \approx -\nabla_y \log p_t(y).
\]
The corresponding objective contains the standard denoising term across scales together with an explicit regularization term in the variance direction \(t\), so the model is not merely a bank of per-scale scores but a continuously parameterized joint prior over \((y,t)\) [2302.10502]. This formulation makes the continuation interpretation explicit and allows fixed, learned, or jointly optimized schedules in inverse problems.

A related but distinct construction is multiscale log-density estimation. MULDE trains a scalar network \(f_\theta(\tilde{x},\sigma)\) so that
\[
\nabla_{\tilde{x}} f_\theta(\tilde{x},\sigma) \approx -\nabla_{\tilde{x}}\log q_\sigma(\tilde{x}),
\]
hence
\[
f_\theta(\tilde{x},\sigma)\approx -\log q_\sigma(\tilde{x})+C_\sigma.
\]
Training samples \(\sigma\) log-uniformly from a range of scales, weights the denoising term by \(\lambda(\sigma)=\sigma^2\), and adds a regularizer
\[
\beta f_\theta(x,\sigma)^2
\]
to align the otherwise arbitrary additive constants across scales [2403.14497]. At test time, the vector
\[
[f_\theta(x,\sigma_1),\ldots,f_\theta(x,\sigma_L)]
\]
is treated as a multiscale descriptor and modeled with a Gaussian mixture model. This is still multiscale denoising score matching in substance, but its output statistic is a multiscale energy trajectory rather than a direct generative score field.

Energy-based MDSM for high-dimensional image modeling takes yet another route. There, one learns a single scalar energy \(E(x;\theta)\) associated with one smoothed density \(p_{\sigma_0}\), while multiple corruption scales are used only in the expectation that defines the training loss:
\[
L(\theta) = \sum_{\sigma \in \{\sigma_1 \cdots \sigma_K\} \mathbb{E}_{q_\sigma(\tilde{x}\mid x)p(x)} \, l(\sigma)\, \left\| x-\tilde{x}+\sigma_0^2 \nabla_{\tilde{x}}E(\tilde{x};\theta) \right\|^2,
\qquad
l(\sigma)=\frac{1}{\sigma^2}.
\]
In this view, the model is not conditioned on the noise level at inference; the multiscale structure is used to spread score supervision across space during training [1910.07762].

## 5. Generalizations of targets, corruption families, and geometry

One important extension changes the regression target rather than the sampler. In Target Score Matching, if the clean target score \(\nabla \log p_X(x)\) is known, the noisy score
\[
\nabla \log p_Y(y)
\]
can be written as a posterior expectation of the clean score rather than the score of the corruption kernel:
\[
\nabla \log p_Y(y)= \int \nabla \log p_X(x)\, p_{X|Y}(x\mid y)\, dx.
\]
For scaled additive models this becomes
\[
\nabla \log p_Y(y)=\alpha^{-1}\int \nabla \log p_X(x)\,p_{X|Y}(x\mid y)\,dx.
\]
The associated regression loss has the same population minimizer as DSM but very different variance properties: DSM is unfavorable at low noise, while TSM is favorable there and unfavorable at high noise. Across a full schedule, the paper advocates scale-dependent mixtures of DSM and TSM rather than a universal replacement [2402.08667].

Another extension changes the corruption family. Heavy-tailed denoising score matching replaces Gaussian corruption by the generalized normal family, proving that the usual DSM simplification remains valid when the relevant functions are differentiable almost everywhere rather than everywhere. In multiscale form, the architecture remains the same—a noise-conditioned score network over \(\sigma_1>\cdots>\sigma_K\)—but the corruption score target becomes that of the generalized normal distribution, and the high-dimensional norm law is no longer \(\chi^2\) but generalized Gamma. The paper’s main geometric claim is that generalized normal corruption produces thicker, skewed shells in high dimension, motivating a quantile-matched noise-scaling algorithm for annealed Langevin initialization [2112.09788].

A third extension concerns geometry rather than noise family. When the latent distribution is supported on a smooth embedded manifold \(M\subset\mathbb{R}^D\), ambient Gaussian DSM develops a low-noise singularity. For the raw tangent target \(T_\sigma\), the conditional variance satisfies
\[
\mathrm{Var}(T_\sigma \mid \pi(X)=z)=\frac{d}{\sigma^2}+O(1),
\]
so the supervision noise diverges as \(\sigma\to 0^+\). Conditioning on the nearest-point projection \(\pi(X)\) yields the Rao-Blackwellized target
\[
r_\sigma(z)=\mathbb E[T_\sigma\mid \pi(X)=z],
\]
which is the unique \(L^2\)-optimal predictor among estimators depending only on \(\pi(X)\). Its small-noise expansion is
\[
r_\sigma(z) = \nabla_M\log q(z) + \sigma^2\bigl[b_q(z)+g_M^{\mathrm{ext}}(z)\bigr] + o(\sigma^2),
\]
where \(b_q\) is an intrinsic Tweedie term and \(g_M^{\mathrm{ext}}\) is an extrinsic curvature correction involving the Weingarten and Ricci operators [2605.25567]. This makes precise the sense in which low-noise multiscale DSM on manifold-supported data is not merely biased but singular unless the tangent target is Rao-Blackwellized.

More abstractly, denoising diffusion can be generalized from Gaussian processes on \(\mathbb{R}^d\) to denoising Markov models on general state spaces. In that framework, time still indexes multiple corruption levels, but the learned object is no longer necessarily an ordinary gradient score; it becomes a generator-dependent generalized logarithmic derivative. Euclidean and manifold diffusions recover true scores, while discrete spaces yield ratio-based or discrete-gradient analogues [2211.03595]. This suggests that “multiscale denoising score matching” is broader than Gaussian Euclidean DSM, even when its most developed implementations remain Gaussian.

## 6. Applications, empirical impact, and open issues

The original motivation for multiscale DSM was high-dimensional generation, and that remains a central application. Energy-based MDSM was proposed precisely because single-noise DSM could denoise but could not generate well from random initialization in high dimensions, while multiscale training plus annealed Langevin dynamics yielded image synthesis competitive with strong energy-based and adversarial baselines [1910.07762]. Subsequent work on image generation showed that final denoising readout, CAS, hybrid adversarial-plus-score objectives, and stronger architectures materially improved the practical standing of DSM-based generative models [2009.05475].

The framework has also migrated into discriminative settings. MULDE uses multiscale DSM to learn negative log-density surrogates of video features across a range of Gaussian smoothings and then combines the per-scale outputs with a Gaussian mixture model for video anomaly detection [2403.14497]. A later video anomaly detector keeps the classical multi-noise DSM backbone,
\[
\frac{1}{L} \sum_{i=1}^{L} \lambda(\sigma_i)\left[\frac{1}{2}\mathbb{E}\left\|s_\theta(\tilde{x},\sigma_i)+\frac{\tilde{x}-x}{\sigma_i^2}\right\|_2^2\right],
\qquad
\lambda(\sigma_i)=\sigma_i^2,
\]
but reinterprets the learned score as anomaly evidence rather than as a sampling vector field, augmenting it with scene conditioning, motion weighting, and an autoregressive denoise-rescore inference loop [2506.23282].

Self-supervised MRI denoising provides an ambient-data variant. Corruption2Self introduces a generalized DSM loss in which already noisy observations are further corrupted and a time-conditioned model is trained so that
\[
h_{\theta^*}(X_t,t)=\mathbb E[X_{t_{\text{target}}}\mid X_t].
\]
The training is genuinely multiscale because a single conditioned network is optimized over a continuum of additional corruption levels, and a reparameterization
\[
\sigma_\tau^2 = \sigma_t^2 - \sigma_{t_{\text{data}}}^2
\]
is introduced to stabilize optimization and improve convergence [2505.05631]. This suggests that multiscale DSM is not limited to sample generation; it also supplies a self-supervised route to denoising when clean targets are unavailable.

The literature is equally clear about the limitations. One short note on CAS emphasizes that it does not propose a new DSM loss, a new training weighting across noise levels, or a new theory of optimal noise schedules; it is a contribution about sampling and sampler tuning [2104.03725]. More broadly, evidence for some proposals is illustrative rather than exhaustive, often confined to one modality or one architectural family. Low-noise behavior remains delicate: DSM is statistically poor there in known-score settings [2402.08667], and ambient DSM on manifolds exhibits an explicit \(d/\sigma^2\) variance singularity [2605.25567]. Even when training is well understood, generation can remain sampler-limited rather than representation-limited [1910.07762].

A final line of work addresses training efficiency rather than target design. Local curvature smoothing with Stein’s identity replaces the Jacobian trace in classical score matching by an inner product under local Gaussian perturbation, yielding a trace-free objective compatible with the same time-conditioned training pipelines used in diffusion models. The method is presented as learning the true score under local smoothing, with empirical performance comparable to DSM and better than sliced-score alternatives in several image-generation settings [2412.03962]. This suggests that future work on multiscale DSM may continue to separate into three partly independent questions: what target is learned at each scale, how scales are traversed at inference time, and how score learning across scales is optimized computationally.

In synthesis, multiscale denoising score matching is best understood as a family rather than a single algorithm. Its common structure is a scale-indexed smoothing of the data distribution and a shared model that ties together the resulting denoising or score-estimation problems across scales. Within that family, the main technical axes are the corruption law, the regression target, the sampler, and the geometric assumptions under which low-noise limits are interpreted. The enduring insight is that scale is not an incidental hyperparameter: it is the organizing variable that makes denoising score matching tractable, generative, and extensible.

Source: https://www.emergentmind.com/topics/multiscale-denoising-score-matching