Papers
Topics
Authors
Recent
Search
2000 character limit reached

Memory in Plain Sight: Surveying the Uncanny Resemblances of Associative Memories and Diffusion Models

Published 28 Sep 2023 in cs.LG, cs.AI, and math.DS | (2309.16750v2)

Abstract: The generative process of Diffusion Models (DMs) has recently set state-of-the-art on many AI generation benchmarks. Though the generative process is traditionally understood as an "iterative denoiser", there is no universally accepted language to describe it. We introduce a novel perspective to describe DMs using the mathematical language of memory retrieval from the field of energy-based Associative Memories (AMs), making efforts to keep our presentation approachable to newcomers to both of these fields. Unifying these two fields provides insight that DMs can be seen as a particular kind of AM where Lyapunov stability guarantees are bypassed by intelligently engineering the dynamics (i.e., the noise and step size schedules) of the denoising process. Finally, we present a growing body of evidence that records DMs exhibiting empirical behavior we would expect from AMs, and conclude by discussing research opportunities that are revealed by understanding DMs as a form of energy-based memory.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (111)
  1. Deep Unsupervised Learning using Nonequilibrium Thermodynamics, November 2015.
  2. Generative Modeling by Estimating Gradients of the Data Distribution. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  3. Improved Techniques for Training Score-Based Generative Models. In Advances in Neural Information Processing Systems, volume 33, pages 12438–12448. Curran Associates, Inc., 2020.
  4. Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems, volume 33, pages 6840–6851. Curran Associates, Inc., 2020.
  5. Diffusion Models Beat GANs on Image Synthesis. In Advances in Neural Information Processing Systems, November 2021.
  6. Hierarchical Text-Conditional Image Generation with CLIP Latents, April 2022.
  7. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. In Advances in Neural Information Processing Systems, May 2022.
  8. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  9. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models, March 2022.
  10. Midjourney, 2023.
  11. Video Diffusion Models. In Advances in Neural Information Processing Systems, May 2022.
  12. DiffDock: Diffusion Steps, Twists, and Turns for Molecular Docking. In The Eleventh International Conference on Learning Representations, September 2022.
  13. DiffWave: A Versatile Diffusion Model for Audio Synthesis. In International Conference on Learning Representations, October 2020.
  14. WaveGrad: Estimating Gradients for Waveform Generation. In International Conference on Learning Representations, October 2020.
  15. Grad-tts: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning, pages 8599–8608. PMLR, 2021.
  16. Latent diffusion energy-based model for interpretable text modeling. In Proceedings of International Conference on Machine Learning (ICML), July 2022.
  17. William James. The Principles of Psychology, Vol I. The Principles of Psychology, Vol I. Henry Holt and Co, New York, NY, US, 1890.
  18. Large associative memory problem in neurobiology and machine learning. In International Conference on Learning Representations, 2021.
  19. David McAllester. On the Mathematics of Diffusion Models, February 2023.
  20. FFJORD: Free-Form Continuous Dynamics for Scalable Reversible Generative Models. In International Conference on Learning Representations, September 2018.
  21. Score-Based Generative Modeling through Stochastic Differential Equations. In International Conference on Learning Representations, October 2020.
  22. Phillip Lippe. Tutorial 7: Deep Energy-Based Generative Models, September 2021.
  23. Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14453–14463, 2023.
  24. Relating transformers to models and neural representations of the hippocampal formation. In International Conference on Learning Representations, 2022.
  25. Pattern completion and disruption characterize contextual modulation in mouse visual cortex, March 2023.
  26. Brain-Diffuser: Natural scene reconstruction from fMRI signals using generative latent diffusion, March 2023.
  27. Incorporating natural language into vision models improves prediction and understanding of higher visual cortex, September 2022.
  28. Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence? In Workshop on Reincarnating Reinforcement Learning at ICLR 2023, March 2023.
  29. Building transformers from neurons and astrocytes. Proceedings of the National Academy of Sciences, 120(34):e2219150120, 2023.
  30. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  31. Hopfield Networks is All You Need. In International Conference on Learning Representations, February 2022.
  32. Energy Transformer, February 2023.
  33. Diffusion Models: A Comprehensive Survey of Methods and Applications, October 2022.
  34. Eric J. Ma. A Pedagogical Introduction to Score Models, April 2022.
  35. Calvin Luo. Understanding Diffusion Models: A Unified Perspective, August 2022.
  36. Denoising Diffusion-based Generative Modeling: Foundations and Applications.
  37. James Thorton and De Bortoli. What’s the score? – Review of latest Score Based Generative Modeling papers., 2023.
  38. Humayun Karim Sulehria and Ye Zhang. Hopfield neural networks: A survey. In Proceedings of the 6th Conference on 6th WSEAS Int. Conf. on Artificial Intelligence, Knowledge Engineering and Data Bases, volume 6, pages 125–130. Citeseer, 2007.
  39. A. G. Hanlon. Content-Addressable and Associative Memory Systems a Survey. IEEE Transactions on Electronic Computers, EC-15(4):509–521, August 1966.
  40. A Study on Associative Neural Memories. International Journal of Advanced Computer Science and Applications (IJACSA), 1(6), February 2012.
  41. Neural Distributed Autoassociative Memories: A Survey. Kibern. vyčisl. teh., 2017(2(188)):5–35, June 2017.
  42. T. Poggio. From Associative Memories to Deep Networks. 2021.
  43. Dmitry Krotov. A new frontier for hopfield networks. Nature Reviews Physics, pages 1–2, 2023.
  44. Auto-Encoding Variational Bayes. December 2013.
  45. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In Proceedings of the 31st International Conference on Machine Learning, pages 1278–1286. PMLR, June 2014.
  46. Angus Turner. Diffusion Models as a kind of VAE, June 2021.
  47. NICE: Non-linear Independent Components Estimation, April 2015.
  48. Density estimation using Real NVP. In International Conference on Learning Representations, November 2016.
  49. Interpreting diffusion score matching using normalizing flow. In ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, June 2021.
  50. Score-Based Generative Classifiers. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, December 2021.
  51. Should EBMs model the energy or the score? In Energy Based Models Workshop - ICLR 2021, April 2021.
  52. Adding conditional control to text-to-image diffusion models, 2023.
  53. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pages 574–584. PMLR, 2020.
  54. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
  55. Least Squares Estimation Without Priors or Supervision. Neural computation, 23:374–420, February 2011.
  56. Pascal Vincent. A Connection Between Score Matching and Denoising Autoencoders. Neural Computation, 23(7):1661–1674, July 2011.
  57. Consistency Models. In International Conference for Machine Learning, January 2023.
  58. Elucidating the Design Space of Diffusion-Based Generative Models. In Advances in Neural Information Processing Systems, May 2022.
  59. Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise, August 2022.
  60. Patrick Kidger. On Neural Differential Equations, February 2022.
  61. Dmitry Krotov. Hierarchical Associative Memory, July 2021.
  62. A Universal Abstraction for Hierarchical Hopfield Networks. In The Symbiosis of Deep Learning and Differential Equations II, October 2022.
  63. Geoffrey E. Hinton. Training Products of Experts by Minimizing Contrastive Divergence. Neural Computation, 14(8):1771–1800, August 2002.
  64. J J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8):2554–2558, April 1982.
  65. David J. C. MacKay and David J. C. Mac Kay. Information Theory, Inference and Learning Algorithms. Cambridge University Press, September 2003.
  66. John Hopfield. Neurons With Graded Response Have Collective Computational Properties Like Those of Two-State Neurons. Proceedings of the National Academy of Sciences of the United States of America, 81:3088–92, June 1984.
  67. Dense associative memory for pattern recognition. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
  68. Y. Abu-Mostafa and J. St. Jacques. Information capacity of the Hopfield model. IEEE Transactions on Information Theory, 31(4):461–464, July 1985.
  69. Storing infinite numbers of patterns in a spin-glass model of neural networks. Physical Review Letters, 55(14):1530, 1985.
  70. Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141–142, 2012.
  71. On a Model of Associative Memory with Huge Storage Capacity. J Stat Phys, 168(2):288–299, July 2017.
  72. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 609–616, Montreal Quebec Canada, June 2009. ACM.
  73. Louis A. Jaeckel. An alternative design for a sparse distributed memory. July 1989.
  74. Pentti Kanerva. Sparse Distributed Memory. MIT press, 1988.
  75. Memory Networks, November 2015.
  76. Biological learning in key-value memory networks. Advances in Neural Information Processing Systems, 34:22247–22258, 2021.
  77. Key-Value Memory Networks for Directly Reading Documents, October 2016.
  78. Boltzmann Machines: Constraint Satisfaction Networks That Learn. Carnegie-Mellon University, Department of Computer Science Pittsburgh, PA, 1984.
  79. A learning algorithm for Boltzmann machines. Cognitive science, 9(1):147–169, 1985.
  80. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pages 5253–5270, 2023.
  81. Score-based Denoising Diffusion with Non-Isotropic Gaussian Noise Models, November 2022.
  82. Non Gaussian Denoising Diffusion Models, June 2021.
  83. All are Worth Words: A ViT Backbone for Score-based Diffusion Models. In NeurIPS 2022 Workshop on Score-Based Methods, November 2022.
  84. Your ViT is Secretly a Hybrid Discriminative-Generative Diffusion Model, August 2022.
  85. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi, editors, Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Lecture Notes in Computer Science, pages 234–241, Cham, 2015. Springer International Publishing.
  86. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, October 2020.
  87. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
  88. CIFAR-10 (Canadian Institute for Advanced Research).
  89. Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models. Proc Mach Learn Res, 162:15561–15583, July 2022.
  90. Simplicial hopfield networks. In The Eleventh International Conference on Learning Representations, 2023.
  91. Can a Fruit Fly Learn Word Embeddings?, March 2021.
  92. Modern Hopfield Networks and Attention for Immune Repertoire Classification. In Advances in Neural Information Processing Systems, volume 33, pages 18832–18845. Curran Associates, Inc., 2020.
  93. Dense associative memory is robust to adversarial inputs. Neural computation, 30(12):3151–3167, 2018.
  94. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023.
  95. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500–22510, 2023.
  96. An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion, August 2022.
  97. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1–11, 2023.
  98. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42(4):1–10, 2023.
  99. MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation, February 2023.
  100. SEGA: Instructing Diffusion using Semantic Dimensions, January 2023.
  101. Denoising Diffusion Implicit Models, October 2022.
  102. Progressive Distillation for Fast Sampling of Diffusion Models. In International Conference on Learning Representations, October 2021.
  103. Fast Sampling of Diffusion Models via Operator Learning. In NeurIPS 2022 Workshop on Score-Based Methods, November 2022.
  104. Adam: A Method for Stochastic Optimization, January 2017.
  105. On the limited memory BFGS method for large scale optimization. Mathematical Programming, 45(1-3):503–528, August 1989.
  106. Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations, March 2020.
  107. Redesigning the transformer architecture with insights from multi-particle dynamical systems. Advances in Neural Information Processing Systems, 34:5531–5544, 2021.
  108. ODE transformer: An ordinary differential equation-inspired model for sequence generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8335–8351, 2022.
  109. A Neural ODE Interpretation of Transformer Layers. In The Symbiosis of Deep Learning and Differential Equations II, November 2022.
  110. Scaling Laws for Neural Language Models, January 2020.
  111. On the Opportunities and Risks of Foundation Models, July 2022.
Citations (5)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 20 likes about this paper.

Sign Up for Free