How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpad (2406.06467v3)

Abstract: Can Transformers predict new syllogisms by composing established ones? More generally, what type of targets can be learned by such models from scratch? Recent works show that Transformers can be Turing-complete in terms of expressivity, but this does not address the learnability objective. This paper puts forward the notion of 'globality degree' of a target distribution to capture when weak learning is efficiently achievable by regular Transformers. This measure shows a contrast with the expressivity results of Transformers captured by $TC^0/TC^1$ classes (further studied here), since the globality relates to correlations with the more limited $NC^0$ class. We show here experimentally and theoretically under additional assumptions that distributions with high globality cannot be learned efficiently. In particular, syllogisms cannot be composed on long chains. Further, we develop scratchpad techniques and show that: (i) agnostic scratchpads cannot break the globality barrier, (ii) educated scratchpads can break the globality with intermediate steps, although not all such scratchpads can generalize out-of-distribution (OOD), (iii) a notion of 'inductive scratchpad', that composes the prior information more efficiently, can both break the globality barrier and improve the OOD generalization. In particular, some of our inductive scratchpads can achieve length generalizations of up to $6\times$ for some arithmetic tasks depending on the input formatting.

• The paper introduces distribution locality and the locality barrier, showing that high locality demands exponentially more tokens for effective reasoning.
• The study proposes educated and inductive scratchpads to break down complex tasks, significantly improving out-of-distribution generalization.
• Empirical results demonstrate that using tailored scratchpads enables transformers to overcome reasoning limitations in tasks like cycle, parity, and arithmetic.

How Far Can Transformers Reason? The Locality Barrier and Inductive Scratchpad

The paper "How Far Can Transformers Reason? The Locality Barrier and Inductive Scratchpad" investigates the limitations of Transformers in reasoning tasks and proposes novel methodologies to overcome these limitations.

Transformers have demonstrated considerable learning capabilities across various domains, including text, image, and audio data. However, when faced with more complex reasoning tasks, current Transformer models often exhibit significant shortcomings. This paper provides a comprehensive analysis of these limitations and introduces the concepts of distribution locality and inductive scratchpads to address them.

Key Contributions

1. Distribution Locality: The authors introduce the notion of distribution locality, which measures the minimal number of input tokens required to correlate non-trivially with the target given the token histogram. This concept is mathematically formalized and empirically validated through the cycle task. The cycle task, a binary classification problem, requires determining the connectivity of certain nodes in a graph composed of cycles—a task that inherently involves high locality and thus poses a significant challenge for Transformers. The paper proposes that weak learning is achievable by regular Transformers if and only if the target distribution has constant locality.
2. Locality Barrier: The authors theorize and empirically demonstrate that tasks with high distribution locality cannot be efficiently learned by regular Transformers. Specifically, they show that in the cycle task, the complexity of learning increases exponentially with the cycle size. This finding highlights the locality barrier of Transformers, wherein a high number of tokens must be attended to for successful learning.
3. Scratchpad Methodologies:

The paper explores different types of scratchpads, namely agnostic, educated, and inductive scratchpads: - Agnostic Scratchpad: The authors show that even with additional memory space, if the scratchpad is unsupervised, it does not effectively break the locality barrier. This requires a more informed approach to designing scratchpads. - Educated Scratchpad: This approach involves breaking down the target task into sub-tasks with lower locality, ensuring that each step can be learned efficiently. Empirical results validate that such an approach significantly improves learning on tasks like parity and cycle tasks. - Inductive Scratchpad: Leveraging the concept of induction, the inductive scratchpad facilitates better OOD generalization by focusing on learning induction steps. This method shows marked improvements in generalizing to larger input sizes for arithmetic tasks.

Experimental Results

Empirical analysis shows that Transformers struggle with tasks requiring long compositions, such as the cycle task with increasing cycle sizes. For instance, a Transformer with 10M parameters fails to learn cycles of size 7 within 100K iterations. However, using appropriately designed educated scratchpads significantly mitigates this issue. For example, the cycle task becomes learnable even at larger sizes when a depth-first search (DFS) scratchpad is used. Furthermore, the inductive scratchpad shows substantial improvement in OOD generalization, such as extending the length generalization for parity and addition tasks.

Implications and Future Directions

The findings of this paper have several practical and theoretical implications. Distribution locality provides a quantifiable measure to understand the learning limitations of Transformers. This measure can guide the design of model architectures and training methodologies to improve learning efficiency on complex reasoning tasks. The proposed inductive scratchpad methodology paves the way for enhanced OOD generalization, which is critical for real-world applications where input distributions often differ from training data distributions.

Future research can explore automated generation and optimization of scratchpads, leveraging the insights from distribution locality. Additionally, integrating the inductive scratchpad with pre-trained models presents an exciting avenue, potentially enabling LLMs to apply learned induction rules to novel tasks.

By addressing the locality barrier uncovered in this paper, the research community can develop more robust models capable of tackling a broader range of complex, real-world reasoning tasks.