Denoising Diffusion Probabilistic Models

Abstract: We present high quality image synthesis results using diffusion probabilistic models, a class of latent variable models inspired by considerations from nonequilibrium thermodynamics. Our best results are obtained by training on a weighted variational bound designed according to a novel connection between diffusion probabilistic models and denoising score matching with Langevin dynamics, and our models naturally admit a progressive lossy decompression scheme that can be interpreted as a generalization of autoregressive decoding. On the unconditional CIFAR10 dataset, we obtain an Inception score of 9.46 and a state-of-the-art FID score of 3.17. On 256x256 LSUN, we obtain sample quality similar to ProgressiveGAN. Our implementation is available at https://github.com/hojonathanho/diffusion

• The paper introduces a novel denoising diffusion probabilistic model that reverses a Gaussian diffusion process to generate high-quality images.
• It leverages variational inference and denoising score matching, achieving an inception score of 9.46 and an FID of 3.17 on CIFAR10.
• Experimental results indicate that diffusion models excel as lossy compressors and offer a competitive alternative to traditional generative frameworks.

Denoising Diffusion Probabilistic Models

The paper "Denoising Diffusion Probabilistic Models" introduces a novel approach to image synthesis using diffusion probabilistic models, a subclass of latent variable models inspired by concepts from nonequilibrium thermodynamics. These models employ a parameterized Markov chain trained via variational inference to reverse a diffusion process that incrementally injects Gaussian noise into data, ultimately aiming to generate high-quality samples. The method showcases promising results on image datasets such as CIFAR10 and CelebA-HQ, exhibiting competitive metrics against alternative generative model classes.

Methodology

The core idea revolves around the diffusion model's ability to reverse a diffusion process where Gaussian noise is systematically added to input data through a forward process. The corresponding reverse process removes noise to regenerate the original data. This process offers a novel perspective by leveraging denoising score matching intertwined with Langevin dynamics, enabling the derivation of a simplified variational bound for training purposes.

Diffusion and Reverse Process

Diffusion models operate by structuring the transition of data points through a sequence of noisy states governed by learned Gaussian distributions:

$$q(x_t|x_{t-1}) \sim \mathcal{N}(x_t; \sqrt{1-\beta_t}x_{t-1}, \beta_t I)$$

Here, $\beta_t$ represents the variance schedule and ensures minute Gaussian perturbations.

The equivalence between denoising score matching and Langevin dynamics enables efficient training of these reverse transitions, represented by:

$$_\theta(x_t, t) = \frac{1}{\sqrt{\alpha_t}}\left( x_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}} \epsilon_\theta(x_t,t)\right)$$

where $\alpha_t$ involves scaling terms and $_\theta$ denotes a neural network approximation.

Figure 1: Generated samples on CelebA-HQ $256 \times 256$ (left) and unconditional CIFAR10 (right).

Training Objectives

The proposed approach involves a simplified training objective:

$$L_\mathrm{simple}(\theta) \approx \mathbb{E}_{t, x_0, \epsilon} \left\| \epsilon - \epsilon_\theta(\sqrt{\bar\alpha_t} x_0 + \sqrt{1-\bar\alpha_t}\epsilon, t) \right\|^2$$

This emphasizes denoising over a range of noise levels, allowing the model to implicitly learn efficient denoising transitions.

Experimental Evaluation

Performance on CIFAR10 and LSUN

The models were tested on CIFAR10, achieving impressive results with an inception score of 9.46 and an FID of 3.17. This signifies superior sample quality compared to most existing models, including class conditional counterparts.

Figure 2: Unconditional CIFAR10 generated samples.

Lossless and Lossy Compression Insights

Despite notable sample quality, diffusion models exhibit non-competitive likelihoods compared to other likelihood-based models, implying their efficacy as lossy compressors. The variational bound suggests that a large proportion of coding length is devoted to minute image details, further asserting their advantageous role in lossy compression scenarios.

Applications and Implications

Diffusion models highlight a generalized form of autoregressive modeling without rigid data order constraints. They hold potential for multi-scale lossy compression applications and pose conceptual links with energy-based models, offering pathways for novel theoretical and practical insights.

Future Developments

Future exploration could expand diffusion models' applicability across various domains and explore their integration with other generative models, enhancing robustness and versatility in data generation and modeling tasks.

Conclusion

The paper illustrates diffusion probabilistic models as a viable and competitive generative modeling framework. Their technical robustness is evident in the impressive image synthesis results. Albeit their nascent stage, diffusion models hold promise for broad applications in data generation, compression, and beyond.