Long short-term memory networks in memristor crossbars (1805.11801v1)

Abstract: Recent breakthroughs in recurrent deep neural networks with long short-term memory (LSTM) units has led to major advances in artificial intelligence. State-of-the-art LSTM models with significantly increased complexity and a large number of parameters, however, have a bottleneck in computing power resulting from limited memory capacity and data communication bandwidth. Here we demonstrate experimentally that LSTM can be implemented with a memristor crossbar, which has a small circuit footprint to store a large number of parameters and in-memory computing capability that circumvents the 'von Neumann bottleneck'. We illustrate the capability of our system by solving real-world problems in regression and classification, which shows that memristor LSTM is a promising low-power and low-latency hardware platform for edge inference.

• The paper introduces a novel memristor crossbar framework that implements LSTM networks to overcome digital hardware limitations.
• It experimentally validates the approach using a regression task for airline data and a classification task with a 79.1% accuracy in gait recognition.
• The research paves the way for energy-efficient, low-latency edge computing by enabling analog in-situ processing of temporal data.

Long Short-Term Memory Networks Implemented in Memristor Crossbars

The paper presents an experimental exploration of implementing Long Short-Term Memory (LSTM) networks using memristor crossbars, an effort aimed at mitigating the inherent limitations of conventional LSTM implementations in digital hardware. Traditional LSTM networks, pivotal for processing temporal and sequential data, face significant challenges due to their computational complexity and the consequential bottleneck in memory capacity and data communication. The authors propose a novel paradigm by employing memristor-based architectures, which promise enhanced power efficiency and reduced inference latency by coherently integrating data storage and computation.

Key Contributions and Experimental Framework

The authors meticulously demonstrate the integration of LSTM networks in a memristor crossbar framework. Memristors, known for their capability to perform computations at the storage site, circumvent the von Neumann bottleneck by reducing the need for data transfer between discrete memory and processor units. This research attempts to store the substantial parameter set of LSTM networks within memristor crossbars, thereby facilitating analog matrix multiplication directly at the storage site.

The experimental setup involves developing a two-layer Recurrent Neural Network (RNN) composed of an LSTM layer and a fully-connected layer, which is manifested through a 128×64 1T1R memristor crossbar structure. The paper explores two application scenarios: a regression problem predicting monthly airline passenger numbers, and a classification problem recognizing humans based on gait. Both tasks validate the memristor crossbar's efficacy, revealing substantial promise in energy efficiency and prediction accuracy compared to digital hardware implementations.

Results Synopsis

1. Regression Task: The regression experiment involved predicting airline passenger numbers, utilizing a network with 15 LSTM units and 2,040 memristors to solve a regression task across a dataset spanning 144 months. The memristor crossbar demonstrated its capability to train a network to predict future data accurately, affirmatively showcasing the analog in-situ processing's potential.
2. Classification Task: For the classification problem, a two-layer LSTM-RNN was configured to handle gait recognition tasks using down-sampled human silhouette data. The network, consisting of 14 LSTM units, was tasked with classifying sequences into one of eight categories, reaching an impressive maximum accuracy of 79.1% thereby competently adapting to potential hardware imperfections through in-situ training.

Implications and Future Prospects

The results present memristor-based LSTM networks as a compelling candidate for edge computing scenarios, especially those proliferated by the advent of the Internet of Things (IoT). This integration suggests a trajectory towards low-power, high-efficiency computing platforms that can process temporal data in situ—minimizing latency and conserving bandwidth by eliminating excessive data transfer.

The research opens several pathways for future investigations. Enhancing the integration density of memristor arrays and expanding the computational framework to more sophisticated network configurations will be crucial. Further, optimizing the analog implementation of nonlinear functions and gate operations represents a vital step towards realizing a fully memristor-based neural network without the present reliance on software for nonlinear transformations.

The implications extend into scaling the technology for broader applicability in data-rich environments where real-time processing near the data source is paramount. As memristor technology evolves, it could influence a wider range of applications in autonomous systems, real-time surveillance, and more, substantiating its role in progressive AI hardware solutions.

In conclusion, the demonstration of LSTM networks within memristor crossbars proposes a significant stride toward overcoming existing computational and structural challenges faced by traditional digital LSTMs. This innovation aligns well with the growing demand for efficient and adaptive hardware solutions aimed at enhancing AI's practical deployment across various domains.