Tightening the Score Matching Gap for Diffusion Models

Abstract: Diffusion models (DMs) are a state-of-the-art generative method to approximately sample from an unknown distribution. Their training and evaluation primarily rely on an Evidence Lower Bound (ELBO), which relates the Kullback-Leibler (KL) divergence of model samples to the score matching loss along the path, which serves as a tractable surrogate. The difference between sample quality and the score matching loss produced by this bound leads to the \emph{score matching gap}, which is known to be tight in the worst-case but not descriptive of sample quality in general. In this work, we provide a theoretical analysis of this gap, developing tighter bounds for three metrics: KL divergence, reverse KL divergence, and Wasserstein distance, effectively exploiting the regularity of the class of score estimators. Our results suggest that the quality of the score approximation has more impact on closing the score matching gap for low noise scales. To obtain these bounds, our key technical insight is to exploit the contraction properties of the backward processes. In particular, we rely on entropy flows, logarithmic Sobolev inequalities and reflection couplings, rigorously linking the ergodicity of the Langevin diffusion to the score matching gap problem.

• The paper rigorously derives time-decaying upper bounds for the forward KL, reverse KL, and Wasserstein distances in diffusion models.
• It employs regularity assumptions on the score network, leveraging pseudo-Lipschitz and dissipativity properties to reduce the score matching gap.
• Empirical validations on synthetic and CIFAR-10 datasets demonstrate that low-noise score errors disproportionately affect generative quality.

Tightening the Score Matching Gap for Diffusion Models

Motivation and Context

Diffusion models (DMs)—including score-based generative models (SGMs)—are prominent generative techniques that learn distributions via time-reversed stochastic processes. Training and evaluation of DMs typically rely on the evidence lower bound (ELBO), which frames the estimation task as a variational inference problem, with the (forward) Kullback-Leibler (KL) divergence between the data and the model upper bounded by a (weighted) score matching loss. The divergence between the true KL and this loss is labeled the score matching gap. While previous analyses establish worst-case tightness, the gap is unpredictable in practical scenarios, undermining score matching loss as a faithful proxy for generative quality.

The objective of "Tightening the Score Matching Gap for Diffusion Models" (2607.04442) is to rigorously characterize and reduce this gap. The authors introduce a systematic approach to deriving sharper upper bounds for forward KL, reverse KL, and Wasserstein distances by leveraging regularity properties of the score estimator and exploiting dynamical contractive properties of the backward (generative) SDE. This work advances both theoretical understanding and practical evaluation methodologies for DMs.

Theoretical Developments

Revisiting the Score Matching Gap

The canonical result, following [song2021maximum], asserts that for the forward process $X_t$ (typically an Ornstein-Uhlenbeck SDE) with data distribution $\mu$, parameterized score $s^\theta$, and large terminal time $T$, we have

$$\mathrm{KL}(\mu \| \nu_T) \leq \frac{1}{4} \int_0^T \mathbb{E}\left[ |s(t, X_t) - s^\theta(t, X_t)|^2 \right] \,dt + \mathrm{KL}(\nu_T \| \gamma^d)$$

where $\gamma^d$ is standard Gaussian and $\nu_T$ is the model distribution at time $T$. This relation's looseness motivates a pursuit of tighter, more informative bounds.

Negative Result: On the Inescapability of the Gap

The analysis begins with a negative assertion: in the absence of additional structure (e.g., regularity of $s^\theta$), the classical bound is tight up to negligible error for worst-case constructions, as demonstrated by a Schrödinger bridge argument.

Tightening the Bound: Regularity and Contraction

By imposing regularity—specifically, a one-sided Lipschitz or dissipativity property—on the score network, the authors derive new time-dependent, decaying weightings for the contributions to the score matching loss at different noise levels (elapsed times in the forward process). This is achieved by entropy flow and logarithmic Sobolev arguments:

• Forward KL Bound with Decay: For a score network $s^\theta$ satisfying a pseudo-Lipschitz condition with constant $\mu$0, the improved bound is

$\mu$1

where the weight $\mu$2 is an explicit, time-decreasing function determined by $\mu$3, and $\mu$4 is a similarly improved initialization error term.

Figure 1: Decay term $\mu$5 for the tightened KL upper bound under various regularity assumptions.

This decay prioritizes small-time (low-noise) errors, aligning with empirical best practices emphasizing fine-scale accuracy in DDPM training schedules.

• Dissipativity Enhancement: When the drift is dissipative at a distance, an explicit exponential bound on the score matching loss is obtained, improving classical ELBO convergence rates.
• Reverse KL and Wasserstein Bounds: The analysis extends to the reverse KL and $\mu$6 distances. Notably, for the reverse KL, a log-Sobolev assumption on the data (as opposed to the estimator) suffices:

$\mu$7

where the decay $\mu$8 now depends only on the concentration properties of $\mu$9.

Figure 2: Decay term for the reverse KL divergence with varying log-Sobolev constants.

For Wasserstein-1 and Wasserstein-2, reflection coupling and dissipativity arguments enable similar decays, and explicit equivalence relationships between the relevant weighted metrics and standard $s^\theta$0 are established.

Figure 3: Sensitivity of the KL divergence and score matching loss to score perturbations at different time steps. The proposed time-dependent weighting aligns more closely with the KL sensitivity compared to conventional ELBO weighting.

Empirical Validation

Synthetic and CIFAR-10 Experiments

Empirical studies validate that deviations in the score approximation at low noise scales have disproportionate influence on the gap, in agreement with the theory:

• In a toy circle dataset, the per-time-step score matching losses at low noise more closely track true KL divergence during training.

Figure 4: KL divergence and test score matching loss over training for a toy dataset; low-noise intervals better correlate with KL progression.

• On CIFAR-10 with a U-Net-based DDPM, the regularity (pseudo-Lipschitz and dissipativity) assumptions are evaluated directly.

Figure 5: Empirical evaluation of dissipativity and smoothness—$s^\theta$1—for a trained U-Net at various $s^\theta$2.

• Perturbing the score function at different times further reveals that localized error at low noise results in grainy, locally corrupted images, while perturbations at higher noise have more global, semantic effects.

Figure 6: CIFAR-10 samples generated with score perturbations at various time steps; early perturbations damage local structure, later ones affect global content.

• Weighted score matching loss curves, using the new decays, better capture the true evolution of sample quality.

  Figure 7: Weighted test score matching loss against epochs.

Implications and Perspectives

Theoretically, these results establish that the canonical ELBO/score matching upper bound for DMs can be improved in expectation with problem-relevant regularity on the estimator or the data. The decaying weights rigorously formalize the intuition that minute errors at early reverse integration steps (near the data manifold) have outsize impact. This observation connects prevailing heuristics for training-weight scheduling with robust estimation guarantees, while also providing refined convergence rates for both KL and Wasserstein metrics.

Practically, the findings support the use of time-decayed evaluations over naive score matching loss, and, by quantifying initialization contributions, enable more trustworthy model selection and factual performance attribution (disentangling score approximation from discretization error). The regularity-verification and perturbation analyses conducted on high-dimensional DDPMs further suggest these theoretical insights apply in large-scale vision domains.

For future work, tightening log-Sobolev estimates or exploiting alternative coupling strategies may further close the gap. The convergence analysis approach here also invites integration with recent advances in generalization and sample complexity theory for DMs. Empirical extensions to other architectures and modalities beyond images are another promising direction.

Conclusion

The paper offers a comprehensive theoretical and empirical treatment of the score matching gap in diffusion models, identifying its inherent limitations and introducing principled regularity-based improvements leading to time-decaying upper bounds for key divergences and distances. These results bridge the gap between theory and practice, providing actionable guidance for model evaluation, estimator design, and future analysis of generative modeling dynamics.