Generalization in diffusion models arises from geometry-adaptive harmonic representations (2310.02557v3)

Abstract: Deep neural networks (DNNs) trained for image denoising are able to generate high-quality samples with score-based reverse diffusion algorithms. These impressive capabilities seem to imply an escape from the curse of dimensionality, but recent reports of memorization of the training set raise the question of whether these networks are learning the "true" continuous density of the data. Here, we show that two DNNs trained on non-overlapping subsets of a dataset learn nearly the same score function, and thus the same density, when the number of training images is large enough. In this regime of strong generalization, diffusion-generated images are distinct from the training set, and are of high visual quality, suggesting that the inductive biases of the DNNs are well-aligned with the data density. We analyze the learned denoising functions and show that the inductive biases give rise to a shrinkage operation in a basis adapted to the underlying image. Examination of these bases reveals oscillating harmonic structures along contours and in homogeneous regions. We demonstrate that trained denoisers are inductively biased towards these geometry-adaptive harmonic bases since they arise not only when the network is trained on photographic images, but also when it is trained on image classes supported on low-dimensional manifolds for which the harmonic basis is suboptimal. Finally, we show that when trained on regular image classes for which the optimal basis is known to be geometry-adaptive and harmonic, the denoising performance of the networks is near-optimal.

• The paper shows that diffusion model generalization stems from geometry-adaptive harmonic representations which align DNN inductive biases with optimal denoising functions.
• The study demonstrates that DNN denoisers using GAHBs achieve near-optimal performance on curated datasets while struggling with low-dimensional or shuffled image data.
• The findings pave the way for future research on refining model architectures and algorithms to either leverage or overcome GAHB-induced biases in high-dimensional data modeling.

Generalization in Diffusion Models Arises from Geometry-Adaptive Harmonic Representation

Overview

This paper presents a thorough examination of generalization capabilities within diffusion models, attributing these capabilities to what the authors term "geometry-adaptive harmonic representation" (GAHB). By employing high-dimensional data, the paper reveals the models' proficiency in surpassing the limitations imposed by the curse of dimensionality. Meanwhile, the investigation uncovers the models' tendency to avoid memorizing datasets, thus fostering a broader understanding of their generalization nature. Furthermore, the discovery of the denoising deep neural networks' (DNNs) bias towards GAHBs underscores a crucial step towards comprehending the intrinsic properties enabling efficient and effective high-dimensional data modeling.

Denoising and Generalization

Two denoising DNNs, trained on distinct non-overlapping dataset subsets, exhibited remarkable consistency in denoising function and generated image quality, thereby indicating strong generalization. This was perceived especially notable given the minimal training set size relative to the networks' capacity and image dimensions. The paper posits that this generalization derives from a synergistic alignment of the DNN inductive biases with the intricate properties of image data distributions.

Geometry-Adaptive Harmonic Bases (GAHBs)

Investigations into the operational mechanism of DNN denoisers on photographic images identified a significant operation mode: performing shrinkage in an orthonormal basis characterized by harmonic functions, aptly named geometry-adaptive harmonic bases. This revelation not only highlighted the DNN denoisers' affinity for GAHBs but also set the stage for further exploration into classes of images where the GAHBs shine in terms of optimal basis selection.

Inductive Bias and Optimal Denoising Performance

The paper meticulously benchmarks the DNN denoisers against known optimal denoising conditions across various image classes, including synthetic datasets expressly designed to test the hypothesis concerning GAHBs. This in-depth analysis firmly establishes that when GAHBs align well with the underlying optimal denoising basis of the data, DNN denoisers reach near-optimal performance.

Suboptimal Performance Indications

Conversely, the paper furnishes compelling evidence that deviations from the GAHB-induced inductive bias result in suboptimal denoising. This was particularly evident in image scenarios not suitably represented by GAHBs, such as images formed from low-dimensional manifolds and shuffled pixel datasets, where the DNN denoisers faltered in meeting the optimal denoising threshold.

Implications and Future Directions

The findings from this paper significantly advance the understanding of diffusion models' generalization mechanisms, particularly highlighting the role of geometry-adaptive harmonic bases. By delineating the bounds of efficacy through the lens of inductive biases towards GAHBs, the research paves the way for refined model designs that either harness or transcend these biases for enhanced performance across a broader spectrum of high-dimensional data scenarios. Future research avenues might include further dissecting the architectural and algorithmic underpinnings of DNNs that promote the observed inductive biases and extending this framework to other generative models beyond diffusion-based setups.

Conclusion

Through a combination of empirical validation and theoretical insight, this paper elucidates the foundational role of geometry-adaptive harmonic representation in driving the generalization observed in diffusion models. By bridging the gap between inductive biases and optimal basis representations, the research sets a new cornerstone in the understanding of diffusion models' operational efficacy, marking a significant milestone in the journey towards mastering high-dimensional probabilistic modeling.