Papers
Topics
Authors
Recent
Search
2000 character limit reached

Fixed Point Diffusion Models

Published 16 Jan 2024 in cs.CV, cs.AI, and cs.LG | (2401.08741v1)

Abstract: We introduce the Fixed Point Diffusion Model (FPDM), a novel approach to image generation that integrates the concept of fixed point solving into the framework of diffusion-based generative modeling. Our approach embeds an implicit fixed point solving layer into the denoising network of a diffusion model, transforming the diffusion process into a sequence of closely-related fixed point problems. Combined with a new stochastic training method, this approach significantly reduces model size, reduces memory usage, and accelerates training. Moreover, it enables the development of two new techniques to improve sampling efficiency: reallocating computation across timesteps and reusing fixed point solutions between timesteps. We conduct extensive experiments with state-of-the-art models on ImageNet, FFHQ, CelebA-HQ, and LSUN-Church, demonstrating substantial improvements in performance and efficiency. Compared to the state-of-the-art DiT model, FPDM contains 87% fewer parameters, consumes 60% less memory during training, and improves image generation quality in situations where sampling computation or time is limited. Our code and pretrained models are available at https://lukemelas.github.io/fixed-point-diffusion-models.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Fixed point theory and applications. Cambridge university press, 2001.
  2. Protein structure and sequence generation with equivariant denoising diffusion probabilistic models. arXiv preprint arXiv:2205.15019, 2022.
  3. Donald G. Anderson. Iterative procedures for nonlinear integral equations. J. ACM, 12(4):547–560, 1965.
  4. Ffhq-uv: Normalized facial uv-texture dataset for 3d face reconstruction. ArXiv, abs/2211.13874, 2022a.
  5. Deep equilibrium models. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 688–699, 2019.
  6. Multiscale deep equilibrium models. Advances in Neural Information Processing Systems, 33:5238–5250, 2020.
  7. Deep equilibrium optical flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 620–630, 2022b.
  8. Deep equilibrium optical flow estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 610–620. IEEE, 2022c.
  9. C. G. Broyden. A class of methods for solving nonlinear simultaneous equations. Mathematics of Computation, 19(92):577–593, 1965.
  10. Learning neural event functions for ordinary differential equations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
  11. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  12. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems, pages 8780–8794, 2021.
  13. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
  14. JFB: jacobian-free backpropagation for implicit networks. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022, pages 6648–6656. AAAI Press, 2022.
  15. Datacomp: In search of the next generation of multimodal datasets. CoRR, abs/2304.14108, 2023.
  16. Torchdeq: A library for deep equilibrium models. https://github.com/locuslab/torchdeq, 2023.
  17. On training implicit models. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 24247–24260, 2021.
  18. One-step diffusion distillation via deep equilibrium models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  19. Deep equilibrium architectures for inverse problems in imaging. IEEE Transactions on Computational Imaging, 7:1123–1133, 2021.
  20. Fixed point theory. Springer, 2003.
  21. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6626–6637, 2017.
  22. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, pages 6840–6851, 2020.
  23. V.I. Istratescu. Fixed Point Theory: An Introduction. Springer Dordrecht, Dordrecht, 1 edition, 1981. eBook Packages Springer Book Archive.
  24. Progressive growing of gans for improved quality, stability, and variation. ArXiv, abs/1710.10196, 2017.
  25. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00364, 2022.
  26. Neural controlled differential equations for irregular time series. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  27. Neural sdes as infinite-dimensional gans. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, pages 5453–5463. PMLR, 2021.
  28. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
  29. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pages 1106–1114, 2012.
  30. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2019.
  31. Gradient-based learning applied to document recognition. Proc. IEEE, 86(11):2278–2324, 1998.
  32. Common diffusion noise schedules and sample steps are flawed. CoRR, abs/2305.08891, 2023.
  33. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations, 2021.
  34. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.00927, 2022.
  35. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021.
  36. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, pages 16784–16804, 2022.
  37. Scalable diffusion models with transformers. CoRR, abs/2212.09748, 2022.
  38. Deep equilibrium approaches to diffusion models. Advances in Neural Information Processing Systems, 35:37975–37990, 2022.
  39. Subformer: Exploring weight sharing for parameter efficiency in generative transformers. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4081–4090, 2021.
  40. High-resolution image synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  41. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015 - 18th International Conference Munich, Germany, October 5 - 9, 2015, Proceedings, Part III, 2015.
  42. W. Rudin. Principles of Mathematical Analysis. McGraw-Hill, 3 edition, 1976.
  43. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2021.
  44. LAION-400M: open dataset of clip-filtered 400 million image-text pairs. CoRR, abs/2111.02114, 2021.
  45. LAION-5B: an open large-scale dataset for training next generation image-text models. In NeurIPS, 2022.
  46. Denoising diffusion implicit models. In International Conference on Learning Representations, 2020.
  47. Generative modeling by estimating gradients of the data distribution. Proc. NeurIPS, 32, 2019.
  48. Improved techniques for training score-based generative models. Proc. NeurIPS, 33:12438–12448, 2020.
  49. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
  50. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008, 2017.
  51. J. Wallis. A Treatise of Algebra, Both Historical and Practical. John Playford, 1685.
  52. Learning fast samplers for diffusion models by differentiating through sample quality. In International Conference on Learning Representations, 2021.
  53. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. ArXiv, abs/1506.03365, 2015.
  54. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204.13902, 2022.
Citations (3)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 3 tweets with 42 likes about this paper.

Sign Up for Free

Reddit

  1. [R] Fixed Point Diffusion Models (9 points, 1 comment)