Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors

Abstract: Score-based diffusion models are typically trained by minimizing the $L^2$ score matching error, and standard theoretical analyses rely on this quantity to bound the sampling discrepancy between the learned and target distributions. We show the $L^2$ score error is not the right intrinsic measure of marginal distributional quality: a learned diffusion model can incur arbitrarily large $L^2$ score error while perfectly matching the target distribution. By decomposing score errors into a gradient and a solenoidal component (a Helmholtz-Hodge decomposition), we identify the geometric reason behind this: only the gradient component enters the marginal Fokker-Planck dynamics, while the solenoidal component is structurally invisible. We make this precise in three results. First, building on the corrected geometry, we prove an impossibility result: no monotone function of the $L^2$ score error can uniformly lower bound any divergence between the learned and target distributions. Second, we derive an upper bound on the Kullback-Leibler divergence that depends only on the observable gradient component of the error, tightening the standard Girsanov bound and identifying its looseness as the cost of operating on path-space rather than marginal-space dynamics. Third, we give a tractable estimator of the gradient component via a dual Sobolev identity, which is shown to empirically correlate substantially better with sample quality than the full $L^2$ error.

• The paper shows that the full L2 score error can be misleading because solenoidal errors do not affect the generated marginal distribution.
• It introduces a Helmholtz-Hodge decomposition to separate observable gradient components from invisible solenoidal parts that are irrelevant for marginal evolution.
• Empirical results reveal a strong correlation (>0.95) between the gradient components and FID, supporting geometry-aware diagnostics and training adjustments.

Geometric Decomposition of Score Matching Errors in Diffusion Models

Motivation and Background

Score-based diffusion models are widely adopted for generative modeling by approximating an unknown target distribution $p^\star$ via iterative noise perturbations and reverse-time sampling. Denoising Score Matching (DSM), together with its extensions, supplies the prevailing score-estimation objectives. Conventional theory links the $L^2$ score matching error to sample quality by supplying upper bounds on distributional divergences (e.g., KL) between the generated and true distributions. However, the paper "Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors" (2606.06179) establishes that this $L^2$ score error is not intrinsic to marginal sampling quality.

Helmholtz-Hodge Decomposition and Marginal Observability

The central geometric insight is that the score estimation error $e_s$ can be orthogonally decomposed into a pure gradient (observable) component and a solenoidal (invisible) component via the Helmholtz-Hodge decomposition. The gradient part, $\Pi_G e_s$, tangibly affects the marginal distribution via the Fokker-Planck dynamics, while the solenoidal component, $\Pi_{G^\perp} e_s$, is filtered out and does not contribute to marginal evolution.

Figure 1: Visualization of solenoidal versus gradient score errors: solenoidal components (center) do not impact the generated distribution, unlike gradient-like errors (right), despite similar $L^2$ scores.

The paper rigorously proves that a model may incur arbitrarily large $L^2$ score error (due exclusively to solenoidal components) while perfectly matching the target distribution in terms of marginals. Thus, any monotone function of the $L^2$ score error fails to provide a meaningful uniform lower bound on divergences between learned and true distributions.

Observable-Only Bounds and Tightening Girsanov

The derivation replaces classical bounds (which depend on the full $L^2$ norm) with one strictly determined by the projection onto gradients: $L^2$0 The observable gradient component exclusively governs marginal discrepancies; ignoring the solenoidal parts tightens the bound and identifies the slack in standard Girsanov-based arguments as stemming from path-space versus marginal-space analysis.

Figure 2: Spearman rank correlation between FID and error metrics; the gradient component aligns significantly better with sample quality versus the full error.

Figure 3: Comparison between classical Girsanov and geometry-aware bounds; the classical bound saturates at the solenoidal floor, while the geometry-aware bound keeps decreasing.

Empirical results demonstrate that across dataset and capacity variations (Fashion-MNIST, CIFAR-10), the observable (gradient) component correlates strongly ($L^2$1) with FID, outperforming the full score error norm which may have correlations as low as $L^2$2.

Unobservable Error Saturation During Training

DSM objectives do not penalize solenoidal error components, so after initial phases where the observable gradient components decrease substantially, invisible error components persist and dominate the residual score error.

Figure 4: DSM fails to reduce solenoidal (invisible) error components during training; gradient errors decline while invisible errors plateau.

Figure 5: Full decomposition of score error across model sizes, showing persistent solenoidal plateaus even as gradient errors are minimized.

This separation undermines the reliability of full $L^2$3 score error as a proxy for sample quality; only the geometry-aware gradient projection yields a valid indicator.

Computational Estimation and Diagnostics

The paper proposes a tractable dual Sobolev variational reformulation for estimating $L^2$4 without explicit projection, by gradient ascent on scalar-valued critic networks. This estimator can be used as a standalone diagnostic, efficiently computable during or post training, and is robust across architectures and capacity tiers.

Theoretical and Practical Implications for AI

The findings imply that standard DSM loss is wasteful, penalizing error modes that are structurally unobservable in generated samples. The principal consequence is a refined theoretical taxonomy of score errors:

• Only gradient modes affect marginal generation, as dictated by Fokker-Planck dynamics.
• Solenoidal errors, prevalent in unconstrained architectures (U-Nets, Transformers), inflate classical bounds and decouple learning curves from actual sample quality.

Practically, this motivates the adoption of geometry-aware diagnostics during score network development, and suggests future research directions:

• Integrating gradient component estimation into training objectives, potentially via min-max schemes where critic networks guide the score network toward observable improvements.
• Further analysis of endpoint observability—identifying which gradient errors persist or dissipate en route to sampled outputs, especially in high-dimensional settings.
• Extensions to constrained score parametrizations (energy-based models, input-convex architectures), where the Helmholtz-Hodge gap vanishes.

Theoretically, the decomposition aligns with Wasserstein-2 geometry, identifying $L^2$5 as the tangent space element and connecting to optimal transport regularization and manifold-based modeling.

Conclusion

This work redefines the analysis of score-based diffusion models by identifying the geometric structure of score estimation errors. Observable gradient components, not the full $L^2$6 error, exclusively determine marginal quality. This insight produces strictly tighter sample quality bounds, motivates robust computational diagnostics, and reframes both theoretical understanding and practical training of diffusion models. Future advances should exploit this geometric perspective for improved generative modeling, regularization, and principled architecture design in AI.