Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Compensation Sampling for Improved Convergence in Diffusion Models (2312.06285v1)

Published 11 Dec 2023 in cs.CV and cs.AI

Abstract: Diffusion models achieve remarkable quality in image generation, but at a cost. Iterative denoising requires many time steps to produce high fidelity images. We argue that the denoising process is crucially limited by an accumulation of the reconstruction error due to an initial inaccurate reconstruction of the target data. This leads to lower quality outputs, and slower convergence. To address this issue, we propose compensation sampling to guide the generation towards the target domain. We introduce a compensation term, implemented as a U-Net, which adds negligible computation overhead during training and, optionally, inference. Our approach is flexible and we demonstrate its application in unconditional generation, face inpainting, and face de-occlusion using benchmark datasets CIFAR-10, CelebA, CelebA-HQ, FFHQ-256, and FSG. Our approach consistently yields state-of-the-art results in terms of image quality, while accelerating the denoising process to converge during training by up to an order of magnitude.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. Compositional transformers for scene generation. Advances in Neural Information Processing Systems, 34:9506–9520, 2021.
  2. Cold Diffusion: Inverting arbitrary image transforms without noise. arXiv preprint arXiv:2208.09392, 2022.
  3. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. arXiv preprint arXiv:2206.07309, 2022a.
  4. Analytic-DPM: An analytic estimate of the optimal reverse variance in diffusion probabilistic models. In International Conference on Learning Representations, 2022b.
  5. Conditional image generation with score-based diffusion models. arXiv preprint arXiv:2111.13606, 2021.
  6. Facial structure guided GAN for identity-preserved face image de-occlusion. In International Conference on Multimedia Retrieval, pages 46–54, 2021.
  7. Perception prioritized training of diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11472–11481, 2022.
  8. Soft Diffusion: Score matching with general corruptions. Transactions on Machine Learning Research (TMLR), 2023.
  9. Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  10. Occlusion-aware GAN for face de-occlusion in the wild. In IEEE International Conference on Multimedia and Expo (ICME), pages 1–6, 2020.
  11. Taming transformers for high-resolution image synthesis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12873–12883, 2021.
  12. Generative diffusion prior for unified image restoration and enhancement. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9935–9946, 2023.
  13. Generative adversarial nets. In Advances in Neural Information Processing Systems, 2014.
  14. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Advances in neural information processing systems, 30, 2017.
  15. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  16. Keys to better image inpainting: Structure and texture go hand in hand. In IEEE/CVF Winter Conference on Applications of Computer Vision, pages 208–217, 2023.
  17. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations, 2018.
  18. A style-based generator architecture for generative adversarial networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401–4410, 2019.
  19. Analyzing and improving the image quality of StyleGAN. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110–8119, 2020.
  20. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00364, 2022.
  21. Soft Truncation: A universal training technique of score-based diffusion model for high precision score estimation. arXiv preprint arXiv:2106.05527, 2021.
  22. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  23. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
  24. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
  25. Entropy-driven sampling and training scheme for conditional diffusion generation. arXiv preprint arXiv:2206.11474, 2022a.
  26. MAT: Mask-aware transformer for large hole image inpainting. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10758–10768, 2022b.
  27. Q-diffusion: Quantizing diffusion models. In IEEE/CVF International Conference on Computer Vision, pages 17535–17545, 2023.
  28. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778, 2022a.
  29. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022b.
  30. Deep learning face attributes in the wild. In IEEE International Conference on Computer Vision, pages 3730–3738, 2015.
  31. DPM-Solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775–5787, 2022a.
  32. GLaMa: Joint spatial and frequency loss for general image inpainting. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1301–1310, 2022b.
  33. Repaint: Inpainting using denoising diffusion probabilistic models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11461–11471, 2022.
  34. Understanding deep image representations by inverting them. In IEEE Conference on Computer Vision and Pattern Recognition, pages 5188–5196, 2015.
  35. On distillation of guided diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14297–14306, 2023.
  36. A morphology focused diffusion probabilistic model for synthesis of histopathology images. In IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2000–2009, 2023.
  37. Generating images with sparse representations. arXiv preprint arXiv:2103.03841, 2021.
  38. DiffuseVAE: Efficient, controllable and high-fidelity generation from low-dimensional latents. arXiv preprint arXiv:2201.00308, 2022.
  39. Diffusion autoencoders: Toward a meaningful and decodable representation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10619–10629, 2022.
  40. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
  41. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241, 2015.
  42. A U-net based discriminator for generative adversarial networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8207–8216, 2020.
  43. D2C: Diffusion-decoding models for few-shot conditional generation. Advances in Neural Information Processing Systems, 34:12533–12548, 2021.
  44. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256–2265, 2015.
  45. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
  46. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
  47. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
  48. Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data. In International Conference on Machine Learning, pages 9206–9216, 2020.
  49. Resolution-robust large mask inpainting with Fourier convolutions. In IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2149–2159, 2022.
  50. NVAE: A deep hierarchical variational autoencoder. Advances in Neural Information Processing Systems, 33:19667–19679, 2020.
  51. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34:11287–11302, 2021.
  52. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  53. Patch diffusion: Faster and more data-efficient training of diffusion models. arXiv preprint arXiv:2304.12526, 2023a.
  54. Diffusion-GAN: Training GANs with diffusion. In International Conference on Learning Representations, 2023b.
  55. Tackling the generative learning trilemma with denoising diffusion GANs. arXiv preprint arXiv:2112.07804, 2021.
  56. PFGM++: Unlocking the potential of physics-inspired generative models. arXiv preprint arXiv:2302.04265, 2023a.
  57. Stable target field for reduced variance score estimation in diffusion models. In International Conference on Learning Representations, 2023b.
  58. Segmentation-reconstruction-guided facial image de-occlusion. In IEEE International Conference on Automatic Face and Gesture Recognition (FG), pages 1–8, 2023.
  59. Generative image inpainting with contextual attention. In IEEE Conference on Computer Vision and Pattern Recognition, pages 5505–5514, 2018.
  60. Face de-occlusion with deep cascade guidance learning. IEEE Transactions on Multimedia, 2022.
  61. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conference on Computer Vision and Pattern Recognition, pages 586–595, 2018.
  62. Large scale image completion via co-modulated generative adversarial networks. arXiv preprint arXiv:2103.10428, 2021.
  63. Unpaired image-to-image translation using cycle-consistent adversarial networks. In IEEE International Conference on Computer Vision, pages 2223–2232, 2017.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets