Traveling Waves Encode the Recent Past and Enhance Sequence Learning (2309.08045v2)

Abstract: Traveling waves of neural activity have been observed throughout the brain at a diversity of regions and scales; however, their precise computational role is still debated. One physically inspired hypothesis suggests that the cortical sheet may act like a wave-propagating system capable of invertibly storing a short-term memory of sequential stimuli through induced waves traveling across the cortical surface, and indeed many experimental results from neuroscience correlate wave activity with memory tasks. To date, however, the computational implications of this idea have remained hypothetical due to the lack of a simple recurrent neural network architecture capable of exhibiting such waves. In this work, we introduce a model to fill this gap, which we denote the Wave-RNN (wRNN), and demonstrate how such an architecture indeed efficiently encodes the recent past through a suite of synthetic memory tasks where wRNNs learn faster and reach significantly lower error than wave-free counterparts. We further explore the implications of this memory storage system on more complex sequence modeling tasks such as sequential image classification and find that wave-based models not only again outperform comparable wave-free RNNs while using significantly fewer parameters, but additionally perform comparably to more complex gated architectures such as LSTMs and GRUs.

• The paper demonstrates that incorporating traveling waves into the Wave-RNN architecture enables efficient memory encoding and faster sequence learning compared to traditional models.
• The authors employ convolutional dynamics and circulant weight initialization, achieving superior results on tasks like sequential MNIST and noisy CIFAR10.
• The research offers a computationally efficient alternative to complex architectures such as LSTMs and GRUs, providing valuable insights for neuromorphic design.

Traveling Waves Encode the Recent Past and Enhance Sequence Learning

The paper "Traveling Waves Encode the Recent Past and Enhance Sequence Learning" investigates the role of traveling waves in neural computation and sequence learning. It introduces the Wave-RNN (wRNN), a novel recurrent neural network architecture designed to exhibit traveling waves in its hidden state. This research provides computational evidence supporting the hypothesis that such waves can enhance memory storage and sequence modeling.

Core Contributions

The authors present the Wave-RNN as an extension of simple recurrent neural networks (sRNNs), integrating traveling wave dynamics via a convolutional architecture. This design is inspired by observations of traveling waves in biological neural systems and their hypothesized role in short-term memory. The paper demonstrates that the Wave-RNN can learn faster and achieve lower error rates compared to traditional wave-free RNNs and even complex architectures like LSTMs and GRUs.

Analytical Methods

Key aspects of the Wave-RNN include:

• Wave Formalism: The model is based on the one-dimensional wave equation, utilizing a circulant matrix multiplication to simulate wave propagation across the hidden state.
• Convolutional Dynamics: A convolutional operation over the hidden state allows localized processing, leading to efficient memory storage and retrieval.
• Initialization Strategy: The architecture benefits from a specific initialization of weights to promote the emergence of wave dynamics, improving training stability and performance.

Experimental Results

The paper provides robust experimental evidence through various synthetic memory tasks and complex sequence modeling benchmarks:

• Copy and Adding Tasks: The Wave-RNN demonstrates superior performance in these synthetic tasks, significantly outperforming iRNNs even with fewer parameters.
• Sequential MNIST and Permuted Sequential MNIST: The wRNN shows competitive results, training faster and maintaining high accuracy, particularly excelling in permuted conditions where the sequence order is scrambled.
• Noisy Sequential CIFAR10: The model surpasses traditional gated architectures like GRUs and LSTMs on this task, confirming its efficacy in handling complex sequence dependencies.

Theoretical and Practical Implications

This work establishes the Wave-RNN as a compelling architecture for tasks requiring efficient short-term memory encoding. Its success in outperforming more parameter-heavy models suggests that integrating wave dynamics provides an advantageous inductive bias. The convolutional approach not only enhances performance but does so with significantly fewer parameters, providing a computationally efficient alternative to existing models.

Future Directions

The insights from this paper open multiple avenues for future exploration:

1. Scaling Studies: Investigating the performance of Wave-RNNs at larger scales and on more diverse datasets could further elucidate their practical utility.
2. Architectural Enhancements: Exploring hybrid models that combine the Wave-RNN with other advanced neural architectures might yield further performance improvements.
3. Neuroscientific Applications: The wave-based encoding technique could inspire new models of neural processing, potentially leading to deeper insights into biological brain function.

In conclusion, this paper contributes significantly to our understanding of the computational advantages of traveling waves in neural networks. The Wave-RNN serves as both a practical tool for sequence learning and a theoretical model reflecting potential mechanisms in biological cognition.