On the Resurgence of Recurrent Models for Long Sequences -- Survey and Research Opportunities in the Transformer Era (2402.08132v2)

Abstract: A longstanding challenge for the Machine Learning community is the one of developing models that are capable of processing and learning from very long sequences of data. The outstanding results of Transformers-based networks (e.g., LLMs) promotes the idea of parallel attention as the key to succeed in such a challenge, obfuscating the role of classic sequential processing of Recurrent Models. However, in the last few years, researchers who were concerned by the quadratic complexity of self-attention have been proposing a novel wave of neural models, which gets the best from the two worlds, i.e., Transformers and Recurrent Nets. Meanwhile, Deep Space-State Models emerged as robust approaches to function approximation over time, thus opening a new perspective in learning from sequential data, followed by many people in the field and exploited to implement a special class of (linear) Recurrent Neural Networks. This survey is aimed at providing an overview of these trends framed under the unifying umbrella of Recurrence. Moreover, it emphasizes novel research opportunities that become prominent when abandoning the idea of processing long sequences whose length is known-in-advance for the more realistic setting of potentially infinite-length sequences, thus intersecting the field of lifelong-online learning from streamed data.

• The paper surveys the emergence of recurrent models as a viable alternative to Transformers for handling long sequences.
• It details efficient methodologies, including Linear Transformers and continuous state-space models, to mitigate computational challenges.
• The research highlights future directions in online, continuous learning that bridge theoretical advances with practical sequence processing.

Recurrent Models and Deep State-space Innovations for Processing Extended Sequences

Introduction to the Resurgence of Recurrent Models

The recent advancements in the processing of extended sequences have stimulated a reevaluation of recurrent models within the landscape dominated by Transformer networks. This reconsideration arises from the imperative to address the computational challenges and to enhance the capacity for handling long-range dependencies in sequence processing tasks. The focus of this summary is on a survey that encapsulates the evolution and resurgence of recurrent models amidst the prevailing dominance of Transformers. It explores the theoretical underpinnings, practical implications, and prospective avenues for research in this field.

The Recent Shift Towards Recurrent Models

Transformers and Their Limitations

Transformers have set a benchmark for processing sequences through parallel attention mechanisms. Despite their proven efficacy, these models encounter scalability issues and computational inefficiencies when dealing with long sequences. This bottleneck primarily concerns the quadratic complexity inherent in the self-attention mechanism of Transformers.

Revival of Recurrent Models

In response to the constraints of Transformers, the recent period has witnessed a revival of interest in Recurrent Neural Networks (RNNs) and novel recurrent units designed to amalgamate the advantages of both Transformers and traditional RNNs. These developments have been paralleled by innovations in Deep Space-State Models (SSMs) as robust frameworks for temporal function approximation.

Key Advances in Recurrent Modeling

Linear Transformers and Efficiency Gains

A pivotal advancement has been the introduction of Linear Transformers, which achieve linear computational complexity with respect to sequence length. This has been realized through kernelization techniques that facilitate efficient self-attention computations.

State-space Models and Continuous Approximations

The exploration of Deep State-space Models (SSMs) represents another significant forward leap. These models, interpreted as extensions of linear RNN structures, introduce a formalism capable of efficiently capturing long-term dependencies. The SSMs offer a promising avenue by leveraging mathematical concepts of continuity and differential equations, thereby facilitating the approximation of functions over time while preserving computational tractability.

Emerging Trends and Methodologies

The survey identifies several emerging trends in the recurrent modeling domain:

• Integration of Recurrent Mechanisms with Transformers: This trend involves devising strategies to imbue Transformer architectures with recurrent properties, aiming to mitigate the computational overhead associated with long sequences.
• Structuring Recurrence for Effective Learning: Novel architectural designs are being explored to embed recurrent dynamics within models effectively. This includes intra-sequence modeling, segment-level recurrence, and chunk-level computational strategies.
• Advancements in State-space Models: There's a growing focus on deep learning paradigms that integrate SSMs with neural networks. These approaches offer a promising framework for dealing with sequences by conceptualizing them as continuous-time signals.

Future Directions and Challenges

Processing Potentially Infinite Sequences

A frontier area of research highlighted concerns the transition from handling long sequences to managing potentially infinite sequences. This paradigm shift mandates innovative learning algorithms capable of online, continuous learning from data streams, mirroring human cognitive processes more closely.

Bridging Theory and Practical Implementation

While the resurgence of recurrence underscores the theoretical viability of recurrent models for handling long sequences efficiently, a gap remains in practical implementation. The development of learning algorithms that can operate effectively in an online setting and adapt over time without forgetfulness poses a substantial challenge.

Conclusion

The survey articulates the significant strides made in rejuvenating recurrent models for sequence processing. It underscores the convergence of ideas from recurrent neural networks, Transformers, and state-space models as a testament to the dynamism in this research area. The outlined advancements not only address the computational challenges posed by long sequences but also pave the way for models capable of continuous learning from infinite data streams, bringing us closer to realizing more human-like learning systems.