2000 character limit reached
Active Dendrites Enable Efficient Continual Learning in Time-To-First-Spike Neural Networks
Published 30 Apr 2024 in cs.NE | (2404.19419v2)
Abstract: While the human brain efficiently adapts to new tasks from a continuous stream of information, neural network models struggle to learn from sequential information without catastrophically forgetting previously learned tasks. This limitation presents a significant hurdle in deploying edge devices in real-world scenarios where information is presented in an inherently sequential manner. Active dendrites of pyramidal neurons play an important role in the brain ability to learn new tasks incrementally. By exploiting key properties of time-to-first-spike encoding and leveraging its high sparsity, we present a novel spiking neural network model enhanced with active dendrites. Our model can efficiently mitigate catastrophic forgetting in temporally-encoded SNNs, which we demonstrate with an end-of-training accuracy across tasks of 88.3% on the test set using the Split MNIST dataset. Furthermore, we provide a novel digital hardware architecture that paves the way for real-world deployment in edge devices. Using a Xilinx Zynq-7020 SoC FPGA, we demonstrate a 100-% match with our quantized software model, achieving an average inference time of 37.3 ms and an 80.0% accuracy.
- M. McCloskey et al., “Catastrophic interference in connectionist networks: The sequential learning problem,” in Psychology of learning and motivation. Elsevier, 1989, vol. 24, pp. 109–165.
- D. E. Rumelhart et al., “Learning representations by back-propagating errors,” nature, vol. 323, no. 6088, pp. 533–536, 1986.
- H. Robbins et al., “A stochastic approximation method,” The annals of mathematical statistics, pp. 400–407, 1951.
- O. I. Abiodun et al., “Comprehensive review of artificial neural network applications to pattern recognition,” IEEE access, vol. 7, pp. 158 820–158 846, 2019.
- A. Krizhevsky et al., “Imagenet classification with deep convolutional neural networks,” Advances in neural information processing systems, vol. 25, 2012.
- Z.-Q. Zhao et al., “Object detection with deep learning: A review,” IEEE transactions on neural networks and learning systems, vol. 30, no. 11, pp. 3212–3232, 2019.
- T. Brown et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.
- T. Mikolov et al., “Distributed representations of words and phrases and their compositionality,” Advances in neural information processing systems, vol. 26, 2013.
- S. Schneider et al., “wav2vec: Unsupervised pre-training for speech recognition,” arXiv preprint arXiv:1904.05862, 2019.
- J. Kirkpatrick et al., “Overcoming catastrophic forgetting in neural networks,” Proceedings of the national academy of sciences, vol. 114, no. 13, pp. 3521–3526, 2017.
- F. Zenke et al., “Continual learning through synaptic intelligence,” in Proceedings of the 34th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, D. Precup et al., Eds., vol. 70. PMLR, 06–11 Aug 2017, pp. 3987–3995. [Online]. Available: https://proceedings.mlr.press/v70/zenke17a.html
- N. Y. Masse et al., “Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization,” Proceedings of the National Academy of Sciences, vol. 115, no. 44, pp. E10 467–E10 475, 2018.
- A. Iyer et al., “Avoiding catastrophe: Active dendrites enable multi-task learning in dynamic environments,” Frontiers in neurorobotics, vol. 16, p. 846219, 2022.
- H. Shin et al., “Continual learning with deep generative replay,” Advances in neural information processing systems, vol. 30, 2017.
- G. M. Van de Ven et al., “Brain-inspired replay for continual learning with artificial neural networks,” Nature communications, vol. 11, no. 1, p. 4069, 2020.
- C. Frenkel et al., “Bottom-up and top-down approaches for the design of neuromorphic processing systems: Tradeoffs and synergies between natural and artificial intelligence,” Proceedings of the IEEE, 2023.
- ——, “A 28-nm convolutional neuromorphic processor enabling online learning with spike-based retinas,” in 2020 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, 2020, pp. 1–5.
- M. Zhang et al., “Rectified linear postsynaptic potential function for backpropagation in deep spiking neural networks,” IEEE transactions on neural networks and learning systems, vol. 33, no. 5, pp. 1947–1958, 2021.
- S. M. Bohte et al., “Error-backpropagation in temporally encoded networks of spiking neurons,” Neurocomputing, vol. 48, no. 1-4, pp. 17–37, 2002.
- H. Mostafa, “Supervised learning based on temporal coding in spiking neural networks,” IEEE transactions on neural networks and learning systems, vol. 29, no. 7, pp. 3227–3235, 2017.
- S. R. Kheradpisheh et al., “Temporal backpropagation for spiking neural networks with one spike per neuron,” International Journal of Neural Systems, vol. 30, no. 06, p. 2050027, 2020.
- I.-M. Comşa et al., “Temporal coding in spiking neural networks with alpha synaptic function: learning with backpropagation,” IEEE transactions on neural networks and learning systems, vol. 33, no. 10, pp. 5939–5952, 2021.
- L. Lapicque, “Recherches quantitatives sur l’excitation electrique des nerfs,” J Physiol Paris, vol. 9, pp. 620–635, 1907.
- J. Hawkins et al., “Why neurons have thousands of synapses, a theory of sequence memory in neocortex,” Frontiers in neural circuits, p. 23, 2016.
- Y. Yoshimura et al., “Properties of horizontal and vertical inputs to pyramidal cells in the superficial layers of the cat visual cortex,” Journal of Neuroscience, vol. 20, no. 5, pp. 1931–1940, 2000.
- N. Takahashi et al., “Active dendritic currents gate descending cortical outputs in perception,” Nature Neuroscience, vol. 23, no. 10, pp. 1277–1285, 2020.
- F. Corradi et al., “Gyro: A digital spiking neural network architecture for multi-sensory data analytics,” in Proceedings of the 2021 Drone Systems Engineering and Rapid Simulation and Performance Evaluation: Methods and Tools Proceedings, 2021, pp. 9–15.
- G. M. van de Ven et al., “Three types of incremental learning,” Nature Machine Intelligence, vol. 4, no. 12, pp. 1185–1197, 2022.
Paper Prompts
Sign up for free to create and run prompts on this paper.