Simple Local Attentions Remain Competitive for Long-Context Tasks (2112.07210v2)

Abstract: Many NLP tasks require processing long contexts beyond the length limit of pretrained models. In order to scale these models to longer text sequences, many efficient long-range attention variants have been proposed. Despite the abundance of research along this direction, it is still difficult to gauge the relative effectiveness of these models in practical use cases, e.g., if we apply these models following the pretrain-and-finetune paradigm. In this work, we aim to conduct a thorough analysis of these emerging models with large-scale and controlled experiments. For each attention variant, we pretrain large-size models using the same long-doc corpus and then finetune these models for real-world long-context tasks. Our findings reveal pitfalls of an existing widely-used long-range benchmark and show none of the tested efficient attentions can beat a simple local window attention under standard pretraining paradigms. Further analysis on local attention variants suggests that even the commonly used attention-window overlap is not necessary to achieve good downstream results -- using disjoint local attentions, we are able to build a simpler and more efficient long-doc QA model that matches the performance of Longformer~\citep{longformer} with half of its pretraining compute. The code to replicate our experiments can be found at https://github.com/pytorch/fairseq/tree/main/examples/xformers

Analysis of "Simple Local Attentions Remain Competitive for Long-Context Tasks"

The paper "Simple Local Attentions Remain Competitive for Long-Context Tasks" by Xiong et al. provides a detailed examination of efficient attention mechanisms within the transformer architecture, specifically for tasks that require processing long text sequences beyond typical model limits. This research contributes significantly to the comparative evaluation of various attention model architectures under a practical, pretrain-and-finetune framework, offering insights into the applicability and efficiency of these approaches in real-world scenarios.

Methodological Approach

The paper focuses on three categories of efficient attention mechanisms: fixed local patterns, learnable sparse attention patterns, and kernel-based/low-rank methods. Notably, each class is represented by models like Local Window Attention, Reformer, and Linformer, respectively. The authors implement these attention variants within a consistent framework, facilitating a controlled comparison.

To provide a robust evaluation, the paper utilizes large-scale pretraining on a corpus comprising long documents, ensuring fairness by initiating training from scratch rather than leveraging pre-existing model checkpoints such as RoBERTa. This approach is pivotal in ensuring that different architectural parametrizations do not unfairly influence the results, offering a truly comparative analysis of the attention mechanisms themselves.

Key Findings

The results of this paper indicate several noteworthy conclusions:

1. Performance of Local Attentions: Simple local attentions, characterized by their restriction to a token's immediate neighbors, demonstrated competitive performance in long-context downstream tasks. The paper shows that after standard pretraining, other complex attention mechanisms failed to substantially outperform these laid-back approaches.
2. Inadequacies of Long-Range Arena (LRA) Benchmark: The authors reveal discrepancies between results obtained from the LRA benchmark and those from large-scale pretraining followed by downstream evaluation. The LRA does not adequately predict the performance on practical tasks due to its synthetic nature.
3. Efficiency and Simplification: By analyzing attention-window overlaps, the authors demonstrate that these models can be made more efficient. Specifically, a disjoint local attention variant performed comparably to Longformer, achieving similar downstream results with significantly reduced computational demands.

Implications and Future Directions

The implications of these findings highlight the enduring efficacy of simpler local attention mechanisms, challenging the necessity for more complex architectures in certain contexts. This reaffirms the importance of locality biases inherent in natural language processing tasks.

Future research could explore the integration of these insights into generative models or investigate further optimizations in pretraining strategies, such as dynamic or staged context window scaling to enhance efficiency without sacrificing model performance.

Moreover, this work calls for the reconsideration of benchmark design to better reflect the nuances of real-world task performance, thereby guiding the development of more effective and practical long-range attention models. The potential divergence between synthetic benchmarks and actual downstream utility underscores the need for benchmarks that incorporate more realistic, contextually varied datasets.

In conclusion, this paper presents a compelling argument for re-evaluating the complexities introduced into transformer-based models and reaffirms the enduring value of simple, computationally efficient attention mechanisms in processing long-context textual data.