Conversion of Artificial Recurrent Neural Networks to Spiking Neural Networks for Low-power Neuromorphic Hardware (1601.04187v1)

Published 16 Jan 2016 in cs.NE

Abstract: In recent years the field of neuromorphic low-power systems that consume orders of magnitude less power gained significant momentum. However, their wider use is still hindered by the lack of algorithms that can harness the strengths of such architectures. While neuromorphic adaptations of representation learning algorithms are now emerging, efficient processing of temporal sequences or variable length-inputs remain difficult. Recurrent neural networks (RNN) are widely used in machine learning to solve a variety of sequence learning tasks. In this work we present a train-and-constrain methodology that enables the mapping of machine learned (Elman) RNNs on a substrate of spiking neurons, while being compatible with the capabilities of current and near-future neuromorphic systems. This "train-and-constrain" method consists of first training RNNs using backpropagation through time, then discretizing the weights and finally converting them to spiking RNNs by matching the responses of artificial neurons with those of the spiking neurons. We demonstrate our approach by mapping a natural language processing task (question classification), where we demonstrate the entire mapping process of the recurrent layer of the network on IBM's Neurosynaptic System "TrueNorth", a spike-based digital neuromorphic hardware architecture. TrueNorth imposes specific constraints on connectivity, neural and synaptic parameters. To satisfy these constraints, it was necessary to discretize the synaptic weights and neural activities to 16 levels, and to limit fan-in to 64 inputs. We find that short synaptic delays are sufficient to implement the dynamical (temporal) aspect of the RNN in the question classification task. The hardware-constrained model achieved 74% accuracy in question classification while using less than 0.025% of the cores on one TrueNorth chip, resulting in an estimated power consumption of ~17 uW.

Authors (5)

Peter U. Diehl (3 papers)
Guido Zarrella (5 papers)
Andrew Cassidy (4 papers)
Bruno U. Pedroni (6 papers)
Emre Neftci (46 papers)

Citations (213)

View on Semantic Scholar

Summary

Conversion of Artificial Recurrent Neural Networks to Spiking Neural Networks for Low-power Neuromorphic Hardware

The paper "Conversion of Artificial Recurrent Neural Networks to Spiking Neural Networks for Low-power Neuromorphic Hardware" addresses a significant challenge in the neural processing domain: the adaptation of recurrent neural networks (RNNs) for use on spiking neural networks (SNNs) within neuromorphic systems. Given the increasing emphasis on energy efficiency in computing, these findings have implications for the deployment of machine learning models on low-power, neuromorphic hardware such as the IBM TrueNorth chip.

Methodology and Conversion Process

The authors introduce a "train-and-constrain" methodology aimed at translating traditional (non-spiking) Elman RNNs into a spiking format compatible with neuromorphic hardware. This involves training RNNs using backpropagation through time, discretizing the synaptic weights, and converting the trained model into a spiking equivalent. The conversion is demonstrated using a NLP task focused on question classification.

One notable aspect of the conversion is the need to account for the intrinsic constraints of the TrueNorth architecture, such as limits on synaptic connectivity and weight precision. Here, synaptic weights are discretized to 16 levels and neural activity is similarly constrained. Through this discrete process, a 4-bit representation of weights and neural states is realized.

Strong Results and Performance Analysis

The methodology achieves notable results in question classification, with the spiking RNN achieving 74% accuracy while only using a fraction of TrueNorth's capabilities, consuming around 17 µW of power. This demonstrates the practical viability of running complex sequence-processing models on energy-efficient neuromorphic hardware.

It is important to note that discretization, inherent in neuromorphic adaptations, led to a decrease in performance when compared to the original machine learning RNN setup (which had an accuracy of 85%). Most of this drop is attributed to the discretization of synaptic weights, although interestingly, the discretization of hidden states to 4-bit resolution did not degrade performance and even showed marginal improvements in some configurations.

Implications and Future Directions

The work offers significant insights for the design and deployment of machine learning models on neuromorphic processors, providing a method to integrate energy-efficient, low-power computing solutions for sequence-based neural models. The use of synaptic delays as a means to encode temporal dynamics in spiking RNNs provides practical paths forward in evolving SNN capabilities for dynamic tasks.

Future research could expand upon the proposed methodology by experimenting with larger network architectures and exploring its feasibility across a wider array of tasks beyond the domain of NLP. There is also room for further optimization techniques to mitigate the losses due to weight discretization, potentially pushing the terrain of SNNs closer to contemporary machine learning models' performance on conventional processors.

Conclusion

Overall, the research presented in this paper advances the field by outlining a direct approach to implementing RNNs on hardware explicitly designed to mimic neural processes. The conversion method detailed provides a structured framework for transitioning traditional machine learning models to the constraints and features offered by neuromorphic hardware platforms. This represents a meaningful step towards the practical application of artificial intelligence on energy-efficient computing substrates, underscoring the harmony between brain-inspired computation models and modern AI's computational demands.

PDF Markdown

Related Papers

Find Related Papers