Denoising Diffusions with Optimal Transport: Localization, Curvature, and Multi-Scale Complexity (2411.01629v1)

Abstract: Adding noise is easy; what about denoising? Diffusion is easy; what about reverting a diffusion? Diffusion-based generative models aim to denoise a Langevin diffusion chain, moving from a log-concave equilibrium measure $\nu$, say isotropic Gaussian, back to a complex, possibly non-log-concave initial measure $\mu$. The score function performs denoising, going backward in time, predicting the conditional mean of the past location given the current. We show that score denoising is the optimal backward map in transportation cost. What is its localization uncertainty? We show that the curvature function determines this localization uncertainty, measured as the conditional variance of the past location given the current. We study in this paper the effectiveness of the diffuse-then-denoise process: the contraction of the forward diffusion chain, offset by the possible expansion of the backward denoising chain, governs the denoising difficulty. For any initial measure $\mu$, we prove that this offset net contraction at time $t$ is characterized by the curvature complexity of a smoothed $\mu$ at a specific signal-to-noise ratio (SNR) scale $r(t)$. We discover that the multi-scale curvature complexity collectively determines the difficulty of the denoising chain. Our multi-scale complexity quantifies a fine-grained notion of average-case curvature instead of the worst-case. Curiously, it depends on an integrated tail function, measuring the relative mass of locations with positive curvature versus those with negative curvature; denoising at a specific SNR scale is easy if such an integrated tail is light. We conclude with several non-log-concave examples to demonstrate how the multi-scale complexity probes the bottleneck SNR for the diffuse-then-denoise process.

• The paper establishes that score functions serve as optimal transport maps to achieve effective denoising in reverse diffusion processes.
• The study demonstrates how curvature modulates localization uncertainty, directly impacting the denoising performance.
• The work introduces a multi-scale complexity framework that links theoretical bounds to practical improvements through SNR scheduling.

An Academic Overview of "Denoising Diffusions with Optimal Transport: Localization, Curvature, and Multi-Scale Complexity"

The paper "Denoising Diffusions with Optimal Transport: Localization, Curvature, and Multi-Scale Complexity" by Liang et al. presents a rigorous exploration into the mathematical foundations and implications of diffusion-based generative models. Through the perspective of optimal transport theory, the authors explore the mechanisms underlying the denoising processes of these models, framing the discussion around the concepts of localization and curvature within multi-scale complexity landscapes.

Diffusion models are increasingly prevalent in generative tasks, particularly where the probability distributions in question are complex or multi-modal. The primary challenge addressed is the reversal of a diffusion process: transitioning from a log-concave terminal measure, often Gaussian, to a potentially non-log-concave initial measure. This process leverages a score function, which serves as the optimal backward map in minimizing transportation cost, thereby denoising the data in the reverse-time diffusion process.

Major Contributions

1. Optimal Transport and Score Functions: The authors establish that score functions are inherently connected to the optimal transport maps required for effective denoising. This ties the process of reverting diffusion chains directly to transportation cost minimization.
2. Curvature and Localization: A key contribution is the discovery of how curvature impacts localization uncertainty—essentially the variability in predicting past states from current ones. The curvature function controls the localization by influencing the conditional variance, which is crucial for understanding denoising efficacy.
3. Multi-Scale Complexity: The paper introduces a framework for analyzing the diffuse-then-denoise process at various signal-to-noise ratio (SNR) scales. The multi-scale complexity is characterized not by worst-case scenarios, but by average-case curvature complexity, allowing a more nuanced view on where diffusion models succeed or face challenges.

Numerical Results and Theoretical Claims

Through a theoretical framework grounded in Wasserstein metrics, the paper asserts that the curvature, quantified at multiple scales, determines the difficulty of the denoising task. The analyses reveal that while forward diffusion tends to contract distribution distances, backward denoising exhibits at times expansion, dictated by the curvature sign, magnitude, and its complex, multi-layered behavior across scales.

The authors provide theoretical bounds on the contraction and expansion rates of forward and backward processes, respectively, demonstrating that overall process efficacy — denoted as the net contraction of distance — is influenced by the interplay of forward and backward dynamics. Notably, expansion isn't always detrimental; sometimes adding noise and subsequent denoising results in net gains, a phenomenon governed by curvature complexity distinct from traditional metrics like total variation.

Implications and Future Directions

The findings have significant theoretical implications, advancing our understanding of diffusion models beyond classical log-concave assumptions. Practically, this work suggests pathways to optimize diffusion processes through SNR scheduling, potentially improving generative performance by exploiting nuanced non-log-concavities at different scales.

Future developments may include applying these insights to learnings in neural network-augmented diffusion models, where score functions are approximated computationally. Further exploration could lead to improved algorithms for complex generative tasks, ultimately broadening the capacity of AI systems to model diverse data landscapes.

In essence, the paper lays a robust theoretical groundwork for future innovations in diffusion-based generative modeling, emphasizing not just the capabilities, but the complexities inherent in reverting noisy transformations across a spectrum of real-world scenarios.