A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers (2503.03961v2)

Abstract: Recent theoretical results show transformers cannot express sequential reasoning problems over long inputs, intuitively because their computational depth is bounded. However, prior work treats the depth as a constant, leaving it unclear to what degree bounded depth may suffice for solving problems over short inputs, or how increasing the transformer's depth affects its expressive power. We address these questions by analyzing transformers whose depth can grow minimally with context length $n$. We show even highly uniform transformers with depth $\Theta(\log n)$ can express two important problems: recognizing regular languages, which captures state tracking abilities and was known to be expressible only by an unconventional, non-uniform model of transformers, and graph connectivity, which underlies multi-step reasoning. Notably, both of these problems cannot be expressed by fixed-depth transformers under standard complexity conjectures, demonstrating the expressivity benefit of growing depth. Moreover, our theory quantitatively predicts how depth must grow with input length to express these problems, showing that depth scaling is more efficient than scaling width or chain-of-thought steps. Empirically, our detailed experiments designed to bridge the expressivity vs. learnability gap reveal that our theoretical depth requirements for regular language recognition closely match the practical depth requirements for successfully training transformers. Thus, our results clarify how depth affects a transformer's reasoning capabilities, and provide practical guidance for effective depth selection for sequential reasoning.

• The paper demonstrates that transformers with depths scaling logarithmically with input size (\[\Theta(\log n)\]) can solve complex tasks like recognizing regular languages and determining graph connectivity, which are inexpressible by fixed-depth transformers.
• The study finds that scaling depth is significantly more efficient for increasing expressive power compared to increasing transformer width or using chain-of-thought steps.
• Quantitative predictions suggest practical implications for transformer design, indicating that model depth can be adjusted to balance computational efficiency and complex reasoning capacity.

The Expressive Power of Log-Depth Transformers

The paper "A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers" offers a thorough examination of the capabilities of transformers, specifically focusing on the impact of incrementally increasing the computational depth relative to the context length. Historically, transformers have been recognized for their inability to handle sequential reasoning tasks over extended input sequences, primarily attributed to their bounded depth. This paper advances our understanding by analyzing the expressive capacity of universal transformers with depths that scale logarithmically with the input size.

Key Contributions

1. Depth's Impact on Expressive Power: The authors challenge the notion of constant-depth transformers by demonstrating that transformers with depths increasing as $\Theta(\log n)$ can solve problems deemed inexpressible by fixed-depth transformers. Importantly, they prove that such log-depth transformers can achieve two significant tasks: recognizing regular languages and determining graph connectivity—both tasks essential for complex reasoning.
2. Theoretical and Empirical Insights: The theory indicates that log-depth transformers can efficiently simulate certain computations traditionally requiring more resources. The empirical analysis aligns with such predictions, suggesting that the depth requirement scales logarithmically with the input length, corroborating the theoretical underpinnings.
3. Comparison with Other Scaling Strategies: The paper compares depth scaling with alternative approaches like increasing the transformer width or employing chain-of-thought (CoT) steps. It concludes that scaling depth is significantly more efficient than width, which needs superpolynomial increases, and CoT steps, requiring growth beyond logarithmic scales.
4. Practical Implications: Quantitative predictions about depth scaling offer tangible insights that can influence transformer design. The findings propose that model depth can be adjusted to balance computational efficiency with reasoning capacity, directing future work towards deploying transformers in resource-constrained settings.

Discussion and Implications

The paper posits several implications for theoretical computer science and practical AI systems. By positioning log-depth transformers as a bridge between practical constraints and computational power, it challenges current limitations imposed by fixed-depth architectures. Practically, this insight is poised to guide the development of transformer models that balance between robustness, depth, and efficiency, potentially impacting applications that require complex reasoning over substantial context, such as natural language processing and sequential data analysis.

The research raises pertinent questions about the outer limits of transformer capabilities and opens avenues for exploring how minimal depth increases can unlock broader problem-solving potential. The methodological advances presented also suggest potential shifts in model design principles, emphasizing adaptive depth as a flexible tool for tackling logical reasoning tasks.

In conclusion, this paper illustrates the latent prowess of log-scaled depth transformers, advocating for models that adapt by growing dynamically with the task at hand. Through theoretical rigor and empirical validation, it sets the stage for future exploration in optimizing the architecture of AI models to better simulate complex cognitive processes.