Multi-scale Denoising Score Matching (MS-DSM)

• Multi-scale Denoising Score Matching (MS-DSM) is a training methodology that leverages multiple noise scales to condition energy-based models for improved score estimation across high-dimensional data.
• It employs deep residual networks with noise-level conditioning, enabling robust sample synthesis, effective inpainting, and accurate outlier detection.
• The method integrates annealed Langevin dynamics with formal convergence guarantees, achieving competitive generative performance and reduced overfitting compared to traditional approaches.

Multi-scale Denoising Score Matching (MS-DSM) is a training methodology for energy-based models (EBMs) that employs denoising score matching across multiple noise scales to obtain reliable, high-dimensional generative models. MS-DSM addresses the fundamental limitations of classical single-noise-scale denoising score matching—enabling robust sample synthesis, inpainting, denoising, and outlier detection—by appropriately conditioning the score estimator across a range of corruption levels (Li et al., 2019, Block et al., 2020).

1. Formal Objective and Loss Function

MS-DSM trains an energy-based model of the form

$p_\theta(x) \propto \exp(-E_\theta(x))$

by matching its score $\nabla_x \log p_\theta(x)$ to a data-driven target score pertaining to noisy samples. Specifically, for each noise scale $\sigma_i$, noisy samples $\tilde x$ are generated via an isotropic Gaussian corruption $$q_{\sigma_i}(\tilde x|x) = \mathcal{N}(\tilde x; x, \sigma_i^2 I)$$. The conditional score network $s_\theta(\tilde x, \sigma_i)$ is trained to match the denoising score:

$$\nabla_{\tilde x} \log q_{\sigma_i}(\tilde x|x) = -\frac{\tilde x - x}{\sigma_i^2}$$

Aggregating the loss over $K$ noise levels yields the MS-DSM objective:

$$L_{\text{MS-DSM}}(\theta) = \sum_{i=1}^K w(\sigma_i) \mathbb{E}_{p_{\text{data}}(x)}\mathbb{E}_{q_{\sigma_i}(\tilde{x}|x)} \|s_\theta(\tilde x, \sigma_i) - \nabla_{\tilde x} \log q_{\sigma_i}(\tilde x|x)\|_2^2$$

where $w(\sigma_i) = 1/\sigma_i^2$ balances variance across scales. Equivalently, if $$s_\theta(\tilde x, \sigma_i) \coloneqq \sigma_i^2 \nabla_{\tilde x} E_\theta(\tilde x)$$, the loss can be reshaped as

$$L(\theta) = \sum_{i=1}^K w(\sigma_i)\mathbb{E}_{p_{\text{data}}(x)}\mathbb{E}_{q_{\sigma_i}(\tilde{x}|x)}\|x - \tilde{x} + \sigma_0^2 \nabla_{\tilde x} E_\theta(\tilde x)\|^2$$

with $\sigma_0$ as a reference noise scale (Li et al., 2019).

2. High-dimensional Score Coverage and the Failure of Single-scale DSM

In high-dimensional spaces, data is typically concentrated near a low-dimensional manifold. Injecting fixed-variance Gaussian noise causes the corrupted samples to concentrate on a narrow shell at radius $\sim \sqrt{D} \sigma$ from the manifold, due to measure concentration. Training DSM with a single $\sigma$ matches the score only on this thin shell, leaving the score function poorly defined outside. Consequently, sample generation initialized from random noise, which is typically far outside this shell, becomes unstable and fails to converge.

MS-DSM remedies this by training across a broad spectrum of noise scales $\{\sigma_i\}$:

• Large $\sigma_i$: Teach the score network the coarse gradients far from the data manifold.
• Small $\sigma_i$: Provide the fine gradients close to the manifold.

The joint $\sigma$-conditional score estimator $s_\theta(\tilde x, \sigma)$ thus covers the entire ambient space, ensuring that annealed sampling processes remain stable and effective regardless of starting point (Li et al., 2019).

3. Architecture and Conditioning on Noise Level

Empirical implementations deploy a deep residual network backbone, with hyperparameters tuned to dataset complexity. For grayscale $32\times32$ images, a 12-block ResNet of width 64 is used; for RGB data (e.g., CIFAR-10), 18 blocks of width 128 are employed. ELU (Exponential Linear Unit) activations and the exclusion of batch normalization within the blocks are standard.

The output energy head is quadratic, designed as $$E_\mathrm{out}(h) = (a^Th+b_1)(c^Th+b_2) + d^T(h\circ h) + b_3$$, allowing energy scaling proportionally to the input noise variance $\sigma^2$.

Noise-scale conditioning is achieved by appending either the scalar $\sigma$ or an embedded feature $\phi(\sigma)$ to feature maps at various network layers, either via concatenation along the channel axis or through featurewise scaling and shifting. This enables $s_\theta(\cdot, \sigma)$ to adaptively encode noise-level-dependent information (Li et al., 2019).

4. Annealed Langevin Dynamics for Sampling

After training, sample synthesis is performed by annealed Langevin dynamics. For a temperature sequence $\{T_t\}$ decreasing from $T_{\max}$ to $T_{\min}\approx 1$, and step sizes $\epsilon_t$, samples are updated as follows:

$$x_{t+1} = x_t + \frac{\epsilon_t^2}{2} s_\theta(x_t, \sigma_0) + \epsilon_t \sqrt{T_t}\, z_t, \quad z_t \sim \mathcal{N}(0, I)$$

A final deterministic "denoising jump,"

$$x_{\text{final}} = x_{\text{last}} - \sigma_0^2 \nabla_x E_\theta(x_{\text{last}})$$

sharpens the synthetic sample. This sampling procedure blends the benefits of tempering (coarse-to-fine exploration) with stochastic Langevin steps, leveraging the network's global score coverage (Li et al., 2019).

A generic pseudocode, as described in (Li et al., 2019), is as follows:

Initialize x₀ ∼ N(0,I) for t = 0…T_steps−1: T = schedule(t) ε = step_size(t) score = s_θ(x_t, σ₀) x_{t+1} = x_t + ½ ε² * score + ε * sqrt(T) * randn() x_out = x_{T_steps} − σ₀² * ∇_x E_θ(x_{T_steps})

5. Theoretical Guarantees and Multi-scale Sampling Analysis

The MS-DSM framework admits formal convergence guarantees when coupled with denoising auto-encoder (DAE) or DSM-based score estimation. For a data density $p(x)$, the Gaussian-smoothed density $p_\sigma(x)$ and its score $\nabla \log p_\sigma(x)$ can be reliably estimated via DSM using finite-sample training. Under standard dissipativity and Lipschitz assumptions, the Wasserstein-2 distance between the resulting Langevin-annealed samples $\mu_K$ and the data distribution $p$ satisfies bounds of the form:

$$W_2(\mu_K, p) \leq \sigma \sqrt{d} + A(\eta, \tau) + \sqrt{c_{LS}(\sigma^2) KL(\mu_0\|p_\sigma)} e^{-\tau / c_{LS}(\sigma^2)} + C(\tau, \epsilon)$$

where $c_{LS}(\sigma^2)$ is the log-Sobolev constant, and $C(\tau, \epsilon) \sim (\epsilon \tau + ...)^{1/4}$ collects score estimation errors (Block et al., 2020). Warm starts, scale-specific mixing parameters, and network Rademacher complexity are recommended for optimal convergence, per the practical guidelines in (Block et al., 2020).

6. Empirical Performance and Applications

MS-DSM achieves competitive generative modeling benchmarks. On unconditional CIFAR-10 generation (32$\times$32 RGB), the model attains an Inception Score of $8.31$ and FID of $31.7$, comparable to strong GANs (e.g., SNGAN IS=$8.22$, FID=$21.7$) and competitive with NCSN (IS$\approx 8.9$, FID$\approx25.3$). On "3-channel MNIST," it covers approximately $966/1000$ possible modes, matching GAN baselines in mode coverage. Generated samples, including those on MNIST and Fashion-MNIST, demonstrate sharp, coherent structure.

MS-DSM reduces overfitting, as measured by nearest-neighbor retrieval, yielding novel, non-replicated samples. Reverse AIS log-likelihood upper bounds reach approximately $7.0$ bits/dim on CIFAR-10 (subject to estimator variability) (Li et al., 2019).

7. Extensions: Denoising, Inpainting, Outlier Detection

MS-DSM models can directly perform image inpainting by clamping known pixel regions and executing annealed Langevin dynamics with added Gaussian noise in those regions; the reconstructions are plausible and color-consistent. The score network also offers an empirical Bayes denoiser:

$$\hat x = \tilde x + \sigma^2 s_\theta(\tilde x, \sigma)$$

allowing test-time denoising without prior knowledge of $\sigma$. For out-of-distribution detection, models output unnormalized energies $E_\theta(x)$, but—as with most deep generative models—may mis-rank OOD data; alternative strategies using denoising loss or reconstruction error as novelty scores have mixed efficacy (Li et al., 2019).

In sum, MS-DSM equips denoising score matching with a spectrum of noise levels, yielding robust, globally valid score estimators and enabling EBMs with strong generative, restorative, and anomaly-detection capabilities in high dimensions (Li et al., 2019, Block et al., 2020).