Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Resistive Memory-based Neural Differential Equation Solver for Score-based Diffusion Model (2404.05648v1)

Published 8 Apr 2024 in cs.AR, cs.AI, cs.ET, and cs.NE

Abstract: Human brains image complicated scenes when reading a novel. Replicating this imagination is one of the ultimate goals of AI-Generated Content (AIGC). However, current AIGC methods, such as score-based diffusion, are still deficient in terms of rapidity and efficiency. This deficiency is rooted in the difference between the brain and digital computers. Digital computers have physically separated storage and processing units, resulting in frequent data transfers during iterative calculations, incurring large time and energy overheads. This issue is further intensified by the conversion of inherently continuous and analog generation dynamics, which can be formulated by neural differential equations, into discrete and digital operations. Inspired by the brain, we propose a time-continuous and analog in-memory neural differential equation solver for score-based diffusion, employing emerging resistive memory. The integration of storage and computation within resistive memory synapses surmount the von Neumann bottleneck, benefiting the generative speed and energy efficiency. The closed-loop feedback integrator is time-continuous, analog, and compact, physically implementing an infinite-depth neural network. Moreover, the software-hardware co-design is intrinsically robust to analog noise. We experimentally validate our solution with 180 nm resistive memory in-memory computing macros. Demonstrating equivalent generative quality to the software baseline, our system achieved remarkable enhancements in generative speed for both unconditional and conditional generation tasks, by factors of 64.8 and 156.5, respectively. Moreover, it accomplished reductions in energy consumption by factors of 5.2 and 4.1. Our approach heralds a new horizon for hardware solutions in edge computing for generative AI applications.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (75)
  1. Memory, imagination, and predicting the future: A common brain mechanism? \JournalTitleThe Neuroscientist 20, 220–234 (2014).
  2. Brooks, T. et al. Video generation models as world simulators. \JournalTitleOpenAI (2024).
  3. Denoising diffusion probabilistic models. \JournalTitleAdvances in neural information processing systems 33, 6840–6851 (2020).
  4. Elucidating the design space of diffusion-based generative models. \JournalTitlearXiv preprint arXiv:2206.00364 (2022).
  5. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10684–10695 (2022).
  6. Chen, J. et al. Pixart-sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. \JournalTitlearXiv preprint arXiv:2403.04692 (2024).
  7. Kazerouni, A. et al. Diffusion models for medical image analysis: A comprehensive survey. \JournalTitlearXiv preprint arXiv:2211.07804 (2022).
  8. Diffusion models: A comprehensive survey of methods and applications. \JournalTitlearXiv preprint arXiv:2209.00796 (2022).
  9. Fan, W. et al. Generative diffusion models on graphs: Methods and applications. \JournalTitlearXiv preprint arXiv:2302.02591 (2023).
  10. Adding conditional control to text-to-image diffusion models. \JournalTitlearXiv preprint arXiv:2302.05543 (2023).
  11. Diffusion models in nlp: A survey. \JournalTitlearXiv preprint arXiv:2303.07576 (2023).
  12. Dreamfusion: Text-to-3d using 2d diffusion. \JournalTitlearXiv preprint arXiv:2209.14988 (2022).
  13. Song, Y. et al. Score-based generative modeling through stochastic differential equations. \JournalTitlearXiv preprint arXiv:2011.13456 (2020).
  14. Consistency models. \JournalTitlearXiv preprint arXiv:2303.01469 (2023).
  15. Generative modeling by estimating gradients of the data distribution. \JournalTitleAdvances in Neural Information Processing Systems 32 (2019).
  16. Improved techniques for training score-based generative models. \JournalTitleAdvances in neural information processing systems 33, 12438–12448 (2020).
  17. Ghimire, S. et al. Geometry of score based generative models. \JournalTitlearXiv preprint arXiv:2302.04411 (2023).
  18. Meng, C. et al. Sdedit: Guided image synthesis and editing with stochastic differential equations. \JournalTitlearXiv preprint arXiv:2108.01073 (2021).
  19. Neural ordinary differential equations. \JournalTitleAdvances in neural information processing systems 31 (2018).
  20. Liu, X. et al. Neural sde: Stabilizing neural ode networks with stochastic noise. \JournalTitlearXiv preprint arXiv:1906.02355 (2019).
  21. Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit. \JournalTitlearXiv preprint arXiv:1905.09883 (2019).
  22. The end of moore’s law: A new beginning for information technology. \JournalTitleComputing in science & engineering 19, 41–50 (2017).
  23. Ye, W. et al. A 28-nm rram computing-in-memory macro using weighted hybrid 2t1r cell array and reference subtracting sense amplifier for ai edge inference. \JournalTitleIEEE Journal of Solid-State Circuits (2023).
  24. Li, C. et al. Analogue signal and image processing with large memristor crossbars. \JournalTitleNature electronics 1, 52–59 (2018).
  25. Gradient-based neuromorphic learning on dynamical rram arrays. \JournalTitleIEEE Journal on Emerging and Selected Topics in Circuits and Systems 12, 888–897 (2022).
  26. Wang, R. et al. Implementing in-situ self-organizing maps with memristor crossbar arrays for data mining and optimization. \JournalTitleNature communications 13, 2289 (2022).
  27. Sun, Z. et al. A full spectrum of computing-in-memory technologies. \JournalTitleNature Electronics 6, 823–835 (2023).
  28. Lu, C. et al. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. \JournalTitlearXiv preprint arXiv:2211.01095 (2022).
  29. Fast ode-based sampling for diffusion models in around 5 steps. \JournalTitlearXiv preprint arXiv:2312.00094 (2023).
  30. Flow matching for generative modeling. \JournalTitlearXiv preprint arXiv:2210.02747 (2022).
  31. Flow straight and fast: Learning to generate and transfer data with rectified flow. \JournalTitlearXiv preprint arXiv:2209.03003 (2022).
  32. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, 8162–8171 (PMLR, 2021).
  33. Pseudo numerical methods for diffusion models on manifolds. \JournalTitlearXiv preprint arXiv:2202.09778 (2022).
  34. Sde-net: Equipping deep neural networks with uncertainty estimates. \JournalTitlearXiv preprint arXiv:2008.10546 (2020).
  35. Wan, W. et al. A compute-in-memory chip based on resistive random-access memory. \JournalTitleNature 608, 504–512, DOI: 10.1038/s41586-022-04992-8 (2022).
  36. Wang, Z. et al. Fully memristive neural networks for pattern classification with unsupervised learning. \JournalTitleNature Electronics 1, 137–145 (2018).
  37. James, A. P. A hybrid memristor-cmos chip for ai. \JournalTitleNature Electronics 2, 268–269, DOI: 10.1038/s41928-019-0274-6 (2019). Ik8ab Times Cited:11 Cited References Count:7.
  38. In-memory computing with resistive switching devices. \JournalTitleNature electronics 1, 333–343 (2018).
  39. The missing memristor found. \JournalTitleNature 453, 80–83, DOI: 10.1038/nature06932 (2008).
  40. Memristive dynamics enabled neuromorphic computing systems. \JournalTitleScience China Information Sciences 66, 200401 (2023).
  41. Li, C. et al. Long short-term memory networks in memristor crossbar arrays. \JournalTitleNature Machine Intelligence 1, 49–57 (2019).
  42. Lanza, M. et al. Memristive technologies for data storage, computation, encryption, and radio-frequency communication. \JournalTitleScience 376, eabj9979, DOI: doi:10.1126/science.abj9979 (2022).
  43. Yan, X. et al. Reconfigurable mixed-kernel heterojunction transistors for personalized support vector machine classification. \JournalTitleNature Electronics 6, 862–869 (2023).
  44. Wang, Z. et al. Resistive switching materials for information processing. \JournalTitleNature Reviews Materials 5, 173–195 (2020).
  45. Zhang, X. et al. An artificial spiking afferent nerve based on mott memristors for neurorobotics. \JournalTitleNature communications 11, 51 (2020).
  46. Dynamical memristors for higher-complexity neuromorphic computing. \JournalTitleNature Reviews Materials 7, 575–591 (2022).
  47. Electrochemical-memristor-based artificial neurons and synapses—fundamentals, applications, and challenges. \JournalTitleAdvanced Materials 35, 2301924 (2023).
  48. Memory devices and applications for in-memory computing. \JournalTitleNature nanotechnology 15, 529–544 (2020).
  49. Memristive crossbar arrays for brain-inspired computing. \JournalTitleNature materials 18, 309–323 (2019).
  50. Moon, J. et al. Temporal data classification and forecasting using a memristor-based reservoir computing system. \JournalTitleNature Electronics 2, 480–487 (2019).
  51. Retransformer: Reram-based processing-in-memory architecture for transformer acceleration. In Proceedings of the 39th International Conference on Computer-Aided Design, 1–9 (2020).
  52. A comprehensive review on emerging artificial neuromorphic devices. \JournalTitleApplied Physics Reviews 7 (2020).
  53. Classifier-free diffusion guidance. \JournalTitlearXiv preprint arXiv:2207.12598 (2022).
  54. Wang, D. et al. Stochastic Emerging Resistive Memories for Unconventional Computing. In Advanced Memory Technology: Functional Materials and Devices, DOI: 10.1039/BK9781839169946-00240 (Royal Society of Chemistry, 2023).
  55. Tolerating noise effects in processing-in-memory systems for neural networks: A hardware–software codesign perspective. \JournalTitleAdvanced Intelligent Systems 4, 2200029 (2022).
  56. Wei, T. et al. Three-dimensional reconstruction of conductive filaments in hfox-based memristor. \JournalTitleAdvanced Materials 35, 2209925 (2023).
  57. Cai, F. et al. Power-efficient combinatorial optimization using intrinsic noise in memristor hopfield neural networks. \JournalTitleNature Electronics 3, 409–418 (2020).
  58. Modell, A. H. Imagination and the meaningful brain (mit Press, 2003).
  59. Rao, M. et al. Thousands of conductance levels in memristors integrated on cmos. \JournalTitleNature 615, 823–829 (2023).
  60. A 40nm analog-input adc-free compute-in-memory rram macro with pulse-width modulation between sub-arrays. In 2022 IEEE Symposium on VLSI Technology and Circuits (VLSI Technology and Circuits), 266–267 (IEEE, 2022).
  61. Le Gallo, M. et al. A 64-core mixed-signal in-memory compute chip based on phase-change memory for deep neural network inference. \JournalTitleNature Electronics 6, 680–693 (2023).
  62. Li, Y. et al. An adc-less rram-based computing-in-memory macro with binary cnn for efficient edge ai. \JournalTitleIEEE Transactions on Circuits and Systems II: Express Briefs (2023).
  63. Wang, S. et al. In-memory analog solution of compressed sensing recovery in one step. \JournalTitleScience Advances 9, eadj2908, DOI: 10.1126/sciadv.adj2908 (2023).
  64. Brain-inspired computing via memory device physics. \JournalTitleAPL Materials 9 (2021).
  65. Song, M.-K. et al. Recent advances and future prospects for memristive materials, devices, and systems. \JournalTitleACS nano 17, 11994–12039 (2023).
  66. Wang, Z. et al. Reinforcement learning with analogue memristor arrays. \JournalTitleNature electronics 2, 115–124 (2019).
  67. Wang, S. et al. Convolutional echo-state network with random memristors for spatiotemporal signal classification. \JournalTitleAdvanced Intelligent Systems 2200027 (2022).
  68. Activity-difference training of deep neural networks using memristor crossbars. \JournalTitleNature Electronics 6, 45–51 (2023).
  69. One-step regression and classification with cross-point resistive memory arrays. \JournalTitleScience advances 6, eaay2378 (2020).
  70. Zhang, W. et al. Edge learning using a fully integrated neuro-inspired memristor chip. \JournalTitleScience 381, 1205–1211 (2023).
  71. Milo, V. et al. Accurate program/verify schemes of resistive switching memory (rram) for in-memory neural network circuits. \JournalTitleIEEE Transactions on Electron Devices 68, 3832–3837 (2021).
  72. A fully hardware-based memristive multilayer neural network. \JournalTitleScience advances 7, eabj4801 (2021).
  73. Xie, S. et al. 16.2 edram-cim: Compute-in-memory design with reconfigurable embedded-dynamic-memory array realizing adaptive data converters and charge-domain computing. In 2021 IEEE International Solid-State Circuits Conference (ISSCC), vol. 64, 248–250 (IEEE, 2021).
  74. Auto-encoding variational bayes. \JournalTitlearXiv preprint arXiv:1312.6114 (2013).
  75. Low frequency noise in junction field effect transistors. \JournalTitleSolid-State Electronics 21, 1079–1088 (1978).
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (22)
  1. Jichang Yang (5 papers)
  2. Hegan Chen (3 papers)
  3. Jia Chen (85 papers)
  4. Songqi Wang (8 papers)
  5. Shaocong Wang (10 papers)
  6. Yifei Yu (31 papers)
  7. Xi Chen (1036 papers)
  8. Bo Wang (823 papers)
  9. Xinyuan Zhang (60 papers)
  10. Binbin Cui (2 papers)
  11. Yi Li (482 papers)
  12. Ning Lin (25 papers)
  13. Meng Xu (52 papers)
  14. Xiaoxin Xu (9 papers)
  15. Xiaojuan Qi (133 papers)
  16. Zhongrui Wang (32 papers)
  17. Xumeng Zhang (10 papers)
  18. Dashan Shang (16 papers)
  19. Han Wang (420 papers)
  20. Qi Liu (485 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now