Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences (1610.09513v1)

Abstract: Recurrent Neural Networks (RNNs) have become the state-of-the-art choice for extracting patterns from temporal sequences. However, current RNN models are ill-suited to process irregularly sampled data triggered by events generated in continuous time by sensors or other neurons. Such data can occur, for example, when the input comes from novel event-driven artificial sensors that generate sparse, asynchronous streams of events or from multiple conventional sensors with different update intervals. In this work, we introduce the Phased LSTM model, which extends the LSTM unit by adding a new time gate. This gate is controlled by a parametrized oscillation with a frequency range that produces updates of the memory cell only during a small percentage of the cycle. Even with the sparse updates imposed by the oscillation, the Phased LSTM network achieves faster convergence than regular LSTMs on tasks which require learning of long sequences. The model naturally integrates inputs from sensors of arbitrary sampling rates, thereby opening new areas of investigation for processing asynchronous sensory events that carry timing information. It also greatly improves the performance of LSTMs in standard RNN applications, and does so with an order-of-magnitude fewer computes at runtime.

• The paper presents Phased LSTM, which utilizes a novel time gate mechanism to efficiently handle irregular, event-based data.
• It applies a sparse update strategy that reduces computational overhead while accelerating convergence compared to traditional LSTMs.
• Experimental results demonstrate superior performance on temporal tasks, making it ideal for neuromorphic and resource-constrained applications.

An Examination of Phased LSTM: An Efficient RNN Architecture for Temporal Data

This paper introduces the Phased Long Short-Term Memory (Phased LSTM), an extension to the traditional LSTM networks, aiming to address recurrent neural networks' (RNNs) challenges in processing event-based or irregularly sampled data. Standard RNN architectures are designed around a constant time step for input data, which can lead to inefficiencies when dealing with asynchronous data streams as seen in applications involving event-based sensors or multi-modal sensors with varied sampling rates.

Model Overview

The Phased LSTM introduces a novel time gate mechanism that uses parametrized rhythmic oscillations to control the update frequency of the memory cell states. These oscillations dictate when the cell values can be updated, effectively segmenting the operational cycle into open phases, where updates occur, and closed phases, where memory persists unaltered. This design allows the Phased LSTM to natively handle data with any form of temporal sampling, thus naturally integrating asynchronous sensory inputs.

Key Advantages and Methodological Insights

1. Sparse Update Strategy: By operating with sparse updates, the Phased LSTM requires significantly fewer updates per time unit, reducing computational overhead by an order of magnitude relative to traditional LSTM models. This characteristic leads to enhanced runtime efficiency without degrading model performance.
2. Rapid Convergence: The paper provides empirical evidence that demonstrates faster convergence rates in Phased LSTMs, suggesting that the structured gating allows the network to learn long sequences more efficiently, bolstering both memory retention and gradient flow during training.
3. Superior Performance on Temporal Tasks: Phased LSTMs outperform conventional LSTMs and batch-normalized LSTMs on a range of standard benchmarks, including tasks that necessitate precise temporal discrimination or involve asynchronous event data such as the addition task and N-MNIST event-based visual recognition.

Experimental Insights

The authors meticulously compare Phased LSTMs against traditional LSTM variants across multiple tasks: frequency discrimination, the adding task, N-MNIST event recognition, and the GRID audiovisual lip-reading task. Notable results include the Phased LSTM's resilience to high-frequency input data, outperforming LSTM models when tasks involve large temporal sequences. It also provided robustness in scenarios missing consistent input samples, demonstrating its applicability for processing real-world sensory data, such as asynchronous event-based data from neuromorphic sensors in the N-MNIST dataset.

Theoretical and Practical Implications

The Phased LSTM's novel architecture has considerable implications for both theoretical research and practical applications. The introduction of oscillatory time gates directly challenges the conventional fixed time-step methodologies predominant in RNN implementations, offering a new perspective that prioritizes the temporal sparsity of inputs. This approach resonates with advances in neuromorphic computing, wherein event-driven processing is paramount, potentially paving the way for more bio-inspired computational models.

From a practical standpoint, the computational efficiency gained from sparse updates presents opportunities for deploying Phased LSTMs in resource-constrained environments, such as embedded systems or low-power devices, without sacrificing performance.

Future Directions

Further research could explore the potential application of Phased LSTM architectures to other forms of RNNs, such as Gated Recurrent Units (GRUs), or investigate architectural simplifications that maintain performance while further reducing computational demands. Moreover, understanding how Phased LSTMs align with neuromorphic principles could enhance the development of spiking neural networks, merging computational neuroscience with machine learning.

In conclusion, the Phased LSTM represents a significant step forward in the efficient processing of time-based data, especially in scenarios that involve asynchronous or event-driven inputs. Its efficient design and remarkable performance across diverse tasks mark it as a compelling model for researchers continuing to push the boundaries of temporal modeling with RNNs.