Augmenting Self-attention with Persistent Memory (1907.01470v1)

Abstract: Transformer networks have lead to important progress in LLMing and machine translation. These models include two consecutive modules, a feed-forward layer and a self-attention layer. The latter allows the network to capture long term dependencies and are often regarded as the key ingredient in the success of Transformers. Building upon this intuition, we propose a new model that solely consists of attention layers. More precisely, we augment the self-attention layers with persistent memory vectors that play a similar role as the feed-forward layer. Thanks to these vectors, we can remove the feed-forward layer without degrading the performance of a transformer. Our evaluation shows the benefits brought by our model on standard character and word level LLMing benchmarks.

• The paper introduces an innovative transformer model that replaces feed-forward layers with persistent memory vectors to streamline the self-attention mechanism.
• The approach integrates key-value persistent memory into self-attention layers, effectively combining contextual information with general knowledge.
• The model achieves competitive performance on benchmarks like enwik8 and WikiText-103, reducing complexity and parameter count while maintaining efficiency.

Simplifying Transformers with an All-Attention Network

Introduction to All-Attention Network Architecture

The dominance of Transformer architectures in NLP tasks is well documented, with their ability to capture long-term dependencies attributing largely to their success. Standard Transformer modules comprise two main components: self-attention layers and feed-forward layers. This paper introduces an innovative approach that eschews the traditional feed-forward layers in favor of an all-attention mechanism. By augmenting self-attention layers with persistent memory vectors, the authors propose a model architecture that maintains competitive performance metrics while simplifying structural complexity.

Revising the Transformer Layer

The conventional Transformer layer employs a sequence of self-attention followed by feed-forward sub-layers, each contributing to the model's ability to process sequential data and generate rich representations. However, the introduction of an all-attention network questions the indispensable nature of feed-forward layers. In the proposed architecture, the self-attention sub-layers are augmented with persistent memory vectors acting as key-value pairs, directly engaging in the information aggregation process without necessitating a feed-forward transformation. This proposal not only simplifies the network architecture by eliminating feed-forward layers but also introduces a novel method to integrate general knowledge with contextual information seamlessly.

Evaluation and Results

The model's efficacy is evaluated across standard LLMing benchmarks, including character and word-level datasets like enwik8, text8, and WikiText-103. The experiments showcase that this novel architecture attains performance on par with traditional Transformer models, thereby validating the hypothesis that feed-forward layers can be replaced without degrading model performance. For instance, on the enwik8 dataset, the large all-attention model achieves a bit per character (bpc) score competitive with state-of-the-art models, while maintaining a reduced parameter count. Similarly, on the WikiText-103 dataset, the model outperforms comparable Transformer models in perplexity, illustrating its efficiency in word-level LLMing.

Theoretical and Practical Implications

This research contributes to the ongoing dialogue regarding the necessity and functionality of different components within Transformer networks. By demonstrating that a Transformer can maintain its performance metrics without feed-forward layers, the authors encourage a reevaluation of current architectural norms. The introduction of persistent memory vectors as a mechanism to include general knowledge and contextual information within the same framework presents a plausible pathway for future models to become more parameter-efficient. The findings suggest a potential shift in designing sequence models, emphasizing simplification without compromising effectiveness.

Exploring Future Directions

The exploration into all-attention networks opens several avenues for future research, particularly in extending this architecture to a broader range of applications beyond LLMing. Investigating the interplay between persistent vectors and self-attention in different contexts, such as machine translation and text summarization, could yield valuable insights into the generalizability of this architecture. Additionally, diving deeper into the characteristics and optimal size of persistent vectors could further enhance our understanding of how these models store and utilize information.

Conclusion

The proposed all-attention network marks a significant step towards understanding and optimizing the architectural components of Transformer models. By successfully eliminating the need for feed-forward layers without sacrificing performance, this work challenges existing paradigms and sets the stage for future innovations in the field of generative AI and NLP. Through continued exploration and adaptation, the all-attention network provides a compelling blueprint for building more efficient and streamlined models capable of handling the complexities of natural language.