Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Resistive memory-based zero-shot liquid state machine for multimodal event data learning (2307.00771v1)

Published 3 Jul 2023 in cs.ET

Abstract: The human brain is a complex spiking neural network (SNN) that learns multimodal signals in a zero-shot manner by generalizing existing knowledge. Remarkably, the brain achieves this with minimal power consumption, using event-based signals that propagate within its structure. However, mimicking the human brain in neuromorphic hardware presents both hardware and software challenges. Hardware limitations, such as the slowdown of Moore's law and the von Neumann bottleneck, hinder the efficiency of digital computers. On the software side, SNNs are known for their difficult training, especially when learning multimodal signals. To overcome these challenges, we propose a hardware-software co-design that combines a fixed and random liquid state machine (LSM) SNN encoder with trainable artificial neural network (ANN) projections. The LSM is physically implemented using analogue resistive memory, leveraging the inherent stochasticity of resistive switching to generate random weights. This highly efficient and nanoscale in-memory computing approach effectively addresses the von Neumann bottleneck and the slowdown of Moore's law. The ANN projections are implemented digitally, allowing for easy optimization using contrastive loss, which helps to overcome the difficulties associated with SNN training. We experimentally implement this co-design on a 40nm 256Kb in-memory computing macro. We first demonstrate LSM-based event encoding through supervised classification and linear probing on the N-MNIST and N-TIDIGITS datasets.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. Liu, K. et al. An optoelectronic synapse based on α𝛼\alphaitalic_α-in2se3 with controllable temporal dynamics for multimode and multiscale reservoir computing. \JournalTitleNature Electronics 5, 761–773 (2022).
  2. Embodied neuromorphic intelligence. \JournalTitleNature communications 13, 1024 (2022).
  3. Christensen, D. V. et al. 2022 roadmap on neuromorphic computing and engineering. \JournalTitleNeuromorphic Computing and Engineering 2, 022501 (2022).
  4. Physics for neuromorphic computing. \JournalTitleNature Reviews Physics 2, 499–510 (2020).
  5. A 128×\times×128 120 db 15μ𝜇\muitalic_μs latency asynchronous temporal contrast vision sensor. \JournalTitleIEEE journal of solid-state circuits 43, 566–576 (2008).
  6. Asynchronous binaural spatial audition sensor with 2×\times×64×\times×4 channel output. \JournalTitleIEEE Transactions on Biomedical Circuits and Systems 8, 453–464, DOI: 10.1109/TBCAS.2013.2281834 (2014).
  7. Jiménez-Fernández, A. et al. A binaural neuromorphic auditory sensor for fpga: a spike signal processing approach. \JournalTitleIEEE transactions on neural networks and learning systems 28, 804–818 (2016).
  8. Choo, K. D. et al. Energy-efficient motion-triggered iot cmos image sensor with capacitor array-assisted charge-injection sar adc. \JournalTitleIEEE Journal of Solid-State Circuits 54, 2921–2931, DOI: 10.1109/JSSC.2019.2939664 (2019).
  9. Finateu, T. et al. 5.10 a 1280×720 back-illuminated stacked temporal contrast event-based vision sensor with 4.86µm pixels, 1.066geps readout, programmable event-rate controller and compressive data-formatting pipeline. In 2020 IEEE International Solid- State Circuits Conference - (ISSCC), 112–114, DOI: 10.1109/ISSCC19947.2020.9063149 (2020).
  10. Hsu, T.-H. et al. A 0.8 v multimode vision sensor for motion and saliency detection with ping-pong pwm pixel. \JournalTitleIEEE Journal of Solid-State Circuits 56, 2516–2524, DOI: 10.1109/JSSC.2021.3075746 (2021).
  11. Gallego, G. et al. Event-based vision: A survey. \JournalTitleIEEE transactions on pattern analysis and machine intelligence 44, 154–180 (2020).
  12. A dynamic vision sensor with 1% temporal contrast sensitivity and in-pixel asynchronous delta modulator for event encoding. \JournalTitleIEEE Journal of Solid-State Circuits 50, 2149–2160 (2015).
  13. Neuromorphic sensory systems. \JournalTitleCurrent opinion in neurobiology 20, 288–295 (2010).
  14. Yao, P. et al. Fully hardware-implemented memristor convolutional neural network. \JournalTitleNature 577, 641–646 (2020).
  15. In-memory computing with resistive switching devices. \JournalTitleNature electronics 1, 333–343 (2018).
  16. Spintronic memristor through spin-torque-induced magnetization motion. \JournalTitleIEEE electron device letters 30, 294–297 (2009).
  17. The future of electronics based on memristive systems. \JournalTitleNature electronics 1, 22–29 (2018).
  18. Karunaratne, G. et al. In-memory hyperdimensional computing. \JournalTitleNature Electronics 3, 327–337 (2020).
  19. A long short-term memory for ai applications in spike-based neuromorphic hardware. \JournalTitleNature Machine Intelligence 4, 467–479 (2022).
  20. Karunaratne, G. et al. Robust high-dimensional memory-augmented neural networks. \JournalTitleNature communications 12, 2468 (2021).
  21. Moon, J. et al. Temporal data classification and forecasting using a memristor-based reservoir computing system. \JournalTitleNature Electronics 2, 480–487 (2019).
  22. Du, C. et al. Reservoir computing using dynamic memristors for temporal information processing. \JournalTitleNature communications 8, 2204 (2017).
  23. Neuromorphic nanoelectronic materials. \JournalTitleNature nanotechnology 15, 517–528 (2020).
  24. Yu, S. Neuro-inspired computing with emerging nonvolatile memorys. \JournalTitleProceedings of the IEEE 106, 260–285 (2018).
  25. Communication lower bound in convolution accelerators. In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA), 529–541 (IEEE, 2020).
  26. Rao, M. et al. Thousands of conductance levels in memristors integrated on cmos. \JournalTitleNature 615, 823–829 (2023).
  27. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. \JournalTitleIEEE Signal Processing Magazine 36, 51–63 (2019).
  28. Conversion of continuous-valued deep networks to efficient event-driven networks for image classification. \JournalTitleFrontiers in neuroscience 11, 682 (2017).
  29. Theory and tools for the conversion of analog to spiking convolutional neural networks. \JournalTitlearXiv preprint arXiv:1612.04052 (2016).
  30. Wu, Y. et al. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI conference on artificial intelligence, vol. 33, 1311–1318 (2019).
  31. Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. \JournalTitleJournal of neuroscience 18, 10464–10472 (1998).
  32. Phenomenological models of synaptic plasticity based on spike timing. \JournalTitleBiological cybernetics 98, 459–478 (2008).
  33. Brown, T. et al. Language models are few-shot learners. \JournalTitleAdvances in neural information processing systems 33, 1877–1901 (2020).
  34. Dosovitskiy, A. et al. An image is worth 16x16 words: Transformers for image recognition at scale. \JournalTitleICLR (2021).
  35. Pattern classification by memristive crossbar circuits using ex situ and in situ training. \JournalTitleNature communications 4, 2072 (2013).
  36. Prezioso, M. et al. Training and operation of an integrated neuromorphic network based on metal-oxide memristors. \JournalTitleNature 521, 61–64 (2015).
  37. Yu, S. et al. Binary neural network with 16 mb rram macro chip for classification and online training. In 2016 IEEE International Electron Devices Meeting (IEDM), 16–2 (IEEE, 2016).
  38. Yao, P. et al. Face classification using electronic synapses. \JournalTitleNature communications 8, 15199 (2017).
  39. Sheridan, P. M. et al. Sparse coding with memristor networks. \JournalTitleNature nanotechnology 12, 784–789 (2017).
  40. Bayat, F. M. et al. Implementation of multilayer perceptron network with highly uniform passive memristive crossbar circuits. \JournalTitleNature communications 9, 2331 (2018).
  41. Hu, M. et al. Memristor-based analog computation and neural network classification with a dot product engine. \JournalTitleAdvanced materials (Deerfield Beach, Fla.) 30 (2018).
  42. Cai, F. et al. A fully integrated reprogrammable memristor–cmos system for efficient multiply–accumulate operations. \JournalTitleNature Electronics 2, 290–299 (2019).
  43. Duan, Q. et al. Spiking neurons with spatiotemporal dynamics and gain modulation for monolithically integrated memristive neural networks. \JournalTitleNature communications 11, 3399 (2020).
  44. Joshi, V. et al. Accurate deep neural network inference using computational phase-change memory. \JournalTitleNature communications 11, 2473 (2020).
  45. Xue, C.-X. et al. A cmos-integrated compute-in-memory macro based on resistive random-access memory for ai edge devices. \JournalTitleNature Electronics 4, 81–90 (2021).
  46. Liu, Z. et al. Neural signal analysis with memristor arrays towards high-efficiency brain–machine interfaces. \JournalTitleNature communications 11, 4234 (2020).
  47. Zhong, Y. et al. Dynamic memristor-based reservoir computing for high-efficiency temporal signal processing. \JournalTitleNature communications 12, 408 (2021).
  48. Milano, G. et al. In materia reservoir computing with a fully memristive architecture based on self-organizing nanowire networks. \JournalTitleNature Materials 21, 195–202 (2022).
  49. Dalgaty, T. et al. In situ learning using intrinsic memristor variability via markov chain monte carlo sampling. \JournalTitleNature Electronics 4, 151–161 (2021).
  50. Real-time computing without stable states: A new framework for neural computation based on perturbations. \JournalTitleNeural computation 14, 2531–2560 (2002).
  51. Radford, A. et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748–8763 (PMLR, 2021).
  52. Converting static image datasets to spiking neuromorphic datasets using saccades. \JournalTitleFrontiers in neuroscience 9, 437 (2015).
  53. Feature representations for neuromorphic audio spike streams. \JournalTitleFrontiers in neuroscience 12, 23 (2018).
  54. Prototypical networks for few-shot learning. \JournalTitleAdvances in neural information processing systems 30 (2017).
  55. Abbott, L. F. Lapicque’s introduction of the integrate-and-fire model neuron (1907). \JournalTitleBrain research bulletin 50, 303–304 (1999).
  56. Jia, C. et al. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, 4904–4916 (PMLR, 2021).
  57. Deep liquid state machines with neural plasticity for video activity recognition. \JournalTitleFrontiers in neuroscience 13, 686 (2019).
  58. A digital liquid state machine with biologically inspired learning and its application to speech recognition. \JournalTitleIEEE transactions on neural networks and learning systems 26, 2635–2649 (2015).
  59. Reinforcement learning with low-complexity liquid state machines. \JournalTitleFrontiers in Neuroscience 13, 883 (2019).
  60. Short-term plasticity in a liquid state machine biomimetic robot arm controller. In 2017 International Joint Conference on Neural Networks (IJCNN), 3399–3408 (IEEE, 2017).
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (19)
  1. Ning Lin (25 papers)
  2. Shaocong Wang (10 papers)
  3. Yi Li (482 papers)
  4. Bo Wang (823 papers)
  5. Shuhui Shi (6 papers)
  6. Yangu He (9 papers)
  7. Woyu Zhang (6 papers)
  8. Yifei Yu (31 papers)
  9. Yue Zhang (620 papers)
  10. Xiaojuan Qi (133 papers)
  11. Xiaoming Chen (140 papers)
  12. Hao Jiang (230 papers)
  13. Xumeng Zhang (10 papers)
  14. Peng Lin (33 papers)
  15. Xiaoxin Xu (9 papers)
  16. Qi Liu (485 papers)
  17. Zhongrui Wang (32 papers)
  18. Dashan Shang (16 papers)
  19. Ming Liu (421 papers)
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now