Efficient Transformers: A Survey (2009.06732v3)

Abstract: Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision and reinforcement learning. In the field of natural language processing for example, Transformers have become an indispensable staple in the modern deep learning stack. Recently, a dizzying number of "X-former" models have been proposed - Reformer, Linformer, Performer, Longformer, to name a few - which improve upon the original Transformer architecture, many of which make improvements around computational and memory efficiency. With the aim of helping the avid researcher navigate this flurry, this paper characterizes a large and thoughtful selection of recent efficiency-flavored "X-former" models, providing an organized and comprehensive overview of existing work and models across multiple domains.

An Expert Overview of "Efficient Transformers: A Survey"

The rapid evolution of Transformer architectures has been driven by their remarkable versatility and effectiveness across various domains, including NLP, computer vision, and reinforcement learning. With the proliferation of numerous Transformer variants, often termed "X-formers," there has been a pressing need to survey and categorize these innovations, particularly those aimed at enhancing efficiency in terms of computational and memory demands. This paper, "Efficient Transformers: A Survey," comprehensively addresses this necessity by characterizing a broad selection of recent efficient Transformer models and consolidating the literature across multiple domains.

Background and Motivation

Transformers, notably introduced by Vaswani et al. (2017), have fundamentally altered the landscape of deep learning. They employ a self-attention mechanism facilitating the modeling of complex dependencies across input sequences. While their success has been notable, the quadratic complexity of the self-attention mechanism poses substantial challenges for scalability, particularly when handling long sequences pertinent to text, images, and videos. This quadratic cost arises from the necessity to compute attention weights between all possible pairs of sequence tokens, leading to quadratic time and memory complexity.

The paper identifies two primary dimensions of efficiency: memory efficiency and computational cost, and it aims to highlight models addressing these concerns. Efficient Transformer models either reduce the memory footprint, computational complexity, or both, making them applicable to resource-constrained scenarios like on-device applications.

Taxonomy of Efficient Transformers

The paper proposes a comprehensive taxonomy of efficient Transformer models characterized by their primary technique and use case. This taxonomy includes:

1. Fixed Patterns (FP): These models restrict the self-attention mechanism to local neighborhoods or predefined patterns, significantly reducing computation. Examples include Image Transformer and Sparse Transformer, which leverage local and strided attention patterns, respectively.
2. Combination of Patterns (CP): By amalgamating different attention patterns, these models improve coverage and efficiency. Sparse Transformer combines local and strided patterns to enhance overall performance.
3. Learnable Patterns (LP): These models dynamically learn the sparsity patterns based on input data. Notable examples are the Reformer and Routing Transformer, which utilize hash-based and $k$-means clustering mechanisms, respectively, to determine attention patterns.
4. Neural Memory: Utilizing global memory tokens or neural memory modules, these models maintain a global view of the sequence without incurring quadratic costs. Longformer and ETC integrate global tokens to extend the model's receptive field efficiently.
5. Low-Rank Methods: By assuming low-rank structures in the attention matrix, these models reduce dimensionality, lowering both memory and computational complexity. Linformer exemplifies this approach by projecting keys and values to lower-dimensional spaces.
6. Kernels: These models approximate the self-attention mechanism using kernel methods, avoiding explicit computation of the full attention matrix. Performers and Linear Transformers are prominent implementations that achieve linear complexity by leveraging kernel-based approximations.
7. Recurrence: Extending the framework of local attention, these models incorporate recurrent mechanisms to maintain context across chunks. Transformer-XL and Compressive Transformer utilize recurrent connections to handle long-range dependencies.
8. Downsampling: These methods reduce the input sequence length via pooling or other downsampling techniques, followed by processing the compressed sequence. Perceiver and Funnel Transformer are recent examples employing this strategy to manage long sequences efficiently.
9. Sparse Models: Leveraging sparse activation of parameters, these models optimize the parameter-to-FLOPs ratio. Switch Transformer and GShard exemplify the use of Mixture-of-Experts (MoE) architectures to achieve this balance.

Detailed Walk-Through of Efficient Transformer Models

The survey explores the specifics of various models, discussing their methodologies and contributions to improving efficiency. For instance, the Memory Compressed Transformer employs local and memory-compressed attention to manage long sequences, while the Routing Transformer uses online $k$-means clustering for dynamic token grouping. Additionally, models like Longformer and ETC incorporate global attention tokens to enhance the model's capability to capture long-range dependencies without quadratic complexity.

Implications and Future Directions

The implications of these efficient Transformer models are significant for both theoretical advancements and practical applications. Theoretically, these models push the boundaries of scalability, enabling the application of Transformers to broader and more complex tasks. Practically, enhanced efficiency makes Transformers viable for deployment in resource-constrained environments, promoting their adoption in real-world applications such as mobile and edge computing.

Looking ahead, the paper identifies several directions for future research, including the convergence of different efficiency techniques, improved evaluation benchmarks, and the balance between speed and memory efficiency. The emergence of hybrid models combining the strengths of various approaches, along with rigorous evaluation protocols, will likely shape the next generation of Transformer architectures.

Conclusion

"Efficient Transformers: A Survey" provides a thorough and insightful overview of the state of research in efficient Transformer models. By categorizing and analyzing a multitude of approaches, the paper offers valuable guidance for researchers and practitioners aiming to navigate this rapidly evolving field. As efficiency continues to be a critical factor in deep learning, such surveys will be indispensable in driving the development of more scalable and effective Transformer models.