Do Efficient Transformers Really Save Computation? (2402.13934v2)

Abstract: As transformer-based LLMs are trained on increasingly large datasets and with vast numbers of parameters, finding more efficient alternatives to the standard Transformer has become very valuable. While many efficient Transformers and Transformer alternatives have been proposed, none provide theoretical guarantees that they are a suitable replacement for the standard Transformer. This makes it challenging to identify when to use a specific model and what directions to prioritize for further investigation. In this paper, we aim to understand the capabilities and limitations of efficient Transformers, specifically the Sparse Transformer and the Linear Transformer. We focus on their reasoning capability as exhibited by Chain-of-Thought (CoT) prompts and follow previous works to model them as Dynamic Programming (DP) problems. Our results show that while these models are expressive enough to solve general DP tasks, contrary to expectations, they require a model size that scales with the problem size. Nonetheless, we identify a class of DP problems for which these models can be more efficient than the standard Transformer. We confirm our theoretical results through experiments on representative DP tasks, adding to the understanding of efficient Transformers' practical strengths and weaknesses.

• The paper demonstrates that while Sparse and Linear Transformers can address general dynamic programming tasks, they require model scaling with problem size.
• It shows that increased hidden dimensions lead efficient Transformers to incur computational costs akin to standard Transformers for general-purpose reasoning.
• It finds that localized reasoning tasks, such as arithmetic evaluation, enable efficient Transformers like Sparse variants to achieve genuine computational efficiency.

Overview of "Do Efficient Transformers Really Save Computation?"

The paper "Do Efficient Transformers Really Save Computation?" scrutinizes the effectiveness of efficient Transformer models, particularly the Sparse Transformer and the Linear Transformer, in reducing computational complexity compared to the standard Transformer architecture. The investigation concentrates on the reasoning capabilities of these models as exhibited by Chain-of-Thought (CoT) prompts, modeled as Dynamic Programming (DP) problems.

Context and Motivation

As Transformer-based models evolve with larger datasets and increasing parameters, efficient alternatives to the standard Transformer have garnered considerable interest. The primary objective is to manage the quadratic computational complexity induced by the self-attention mechanism, especially in tasks requiring lengthy sequence generation. Despite the proliferation of efficient Transformer variants, theoretical guarantees proving these models as suitable replacements remain elusive. This paper steps into this gap, aiming to assess the practical benefits and limitations of efficient Transformer architectures.

Theoretical Analysis and Results

The research rigorously analyzes the capability of Sparse and Linear Transformers in solving DP problems, a class representing reasoning tasks. The key results from the analysis are:

1. Expressiveness of Efficient Transformers: The paper establishes that Sparse and Linear Transformers possess the necessary expressiveness to tackle general DP tasks. Nevertheless, despite this capacity, both models require a scaling of model size with the problem size - a stark contrast to the constant size sufficiency of standard Transformers.
2. Complexity Analysis: It is demonstrated that the complexity of solving general DP problems with efficient Transformers aligns with the standard Transformer's complexity, $\Theta(L^2)$, due to the needed increase in hidden dimension ($\tilde\Omega(\sqrt{L})$).
3. Efficiency Paradox: The results highlight a paradox where the supposed computational efficiency of these models dissipates for general DP tasks, rendering them as computationally demanding as their standard counterparts.

Exploring Conditions for Efficiency

In pursuit of conditions under which efficient Transformers indeed offer computational gains, the paper explores tasks with inherent structural properties, such as arithmetic evaluation. A central finding is the efficiency lies in leveraging the locality of reasoning steps. The mathematical constructs reveal that when each reasoning step is dependent only on a limited number of preceding steps (locality assumption), efficient Transformers achieve reduced complexity. Sparse Transformers, for instance, exhibit lowered complexity under such conditions, aligning with $\tilde{\Theta}(L\sqrt L)$.

Empirical Validation

Complementing the theoretical insights, the paper presents empirical evaluations across various DP tasks - Arithmetic, Longest Increasing Subsequence (LIS), and Edit Distance (ED). Results empirically affirm the theoretical claims, showing a dependency of model performance on hidden dimension and task locality, and further validating the standard Transformer's adaptability with constant dimensionality across problem scales.

Implications and Future Directions

The findings emphasize the nuanced understanding required in deploying efficient Transformers. For general-purpose reasoning tasks, these models may not inherently lead to reduced computational costs. However, in applications where reasoning can be decomposed into localized steps, these architectures can offer meaningful efficiency. Future research might explore architectural innovations to further enhance the efficiency gains or seek to identify additional conditions that make these models viable alternatives to the standard Transformer framework.