Physics of Language Models: Part 1, Learning Hierarchical Language Structures (2305.13673v3)

Abstract: Transformer-based LLMs are effective but complex, and understanding their inner workings is a significant challenge. Previous research has primarily explored how these models handle simple tasks like name copying or selection, and we extend this by investigating how these models grasp complex, recursive language structures defined by context-free grammars (CFGs). We introduce a family of synthetic CFGs that produce hierarchical rules, capable of generating lengthy sentences (e.g., hundreds of tokens) that are locally ambiguous and require dynamic programming to parse. Despite this complexity, we demonstrate that generative models like GPT can accurately learn this CFG language and generate sentences based on it. We explore the model's internals, revealing that its hidden states precisely capture the structure of CFGs, and its attention patterns resemble the information passing in a dynamic programming algorithm. This paper also presents several corollaries, including showing why positional embedding is inferior to relative attention or rotary embedding; demonstrating that encoder-based models (e.g., BERT, deBERTa) cannot learn very deeply nested CFGs as effectively as generative models (e.g., GPT); and highlighting the necessity of adding structural and syntactic errors to the pretraining data to make the model more robust to corrupted language prefixes.

• The paper demonstrates that transformer models, particularly those with advanced positional embeddings like GPT-RELPOS and GPT-ROPE, achieve near-perfect accuracy in learning CFG structures.
• It reveals through entropy and KL-divergence analyses that hidden states encode non-terminal ancestors and boundaries, mirroring dynamic programming strategies in CFG parsing.
• Analysis of attention patterns uncovers distinct position-based and boundary-based mechanisms that enhance long-range dependency parsing and improve robustness against errors.

Understanding the Physics of LLMs: Insights from Context-Free Grammar (CFG) Learning

Introduction

The paper explores the sophisticated mechanisms by which generative LLMs, particularly transformers, learn and represent context-free grammars (CFGs). Through a range of carefully designed experiments, the authors explore how models like GPT-2 can generate and comprehend complex and ambiguous CFGs, which underpin much of the structure of natural languages, programming languages, and logical systems. This paper provides both empirical and theoretical insights into the internal working of transformers, highlighting their capacity to encapsulate and utilize CFG structures efficiently.

Key Contributions

The major contributions of this paper fall into several categories: empirical evaluations of transformer capabilities in learning CFGs, insights into the internal representations of CFGs by transformers, and analyses of attention patterns that govern transformers' processing of CFG structures.

Empirical Findings

The experiments reveal that transformers, specifically modern variants with relative or rotary positional embeddings, can achieve near-perfect accuracy in generating sentences that conform to CFG rules. For instance, in the cfg3 dataset, models such as GPT-RELPOS and GPT-ROPE outperformed the vanilla GPT-2, showcasing excellent generation and completion accuracies. This outcome emphasizes the importance of positional embeddings in learning complex structures.

Further analysis via entropy and KL-divergence measurements confirmed that the output distributions of these models not only maintained high diversity but also closely aligned with the ground truth CFG distributions. This indicates that the learned models did not merely memorize a few patterns but grasped the CFG rules' fundamental principles.

Internal Representations of CFGs

A pivotal finding of the paper is that the hidden states of transformers implicitly encode the CFG structure at various levels. Linear probing revealed that post-training, the models' hidden states could predict non-terminal (NT) ancestors and boundaries almost perfectly. This suggests that these internal states contain comprehensive information about the CFG's hierarchical structure.

More specifically, the NT ancestor and boundary information were encoded hierarchically across layers and gradually during the training process. This hierarchical encoding aligns with dynamic programming (DP) principles used in CFG parsing, demonstrating that transformers implicitly adopt DP-like strategies in learning CFGs.

Analysis of Attention Patterns

The research also highlights the role of attention patterns in transformers, showing that these patterns mirror the CFG's syntactic structure. There are two major types of attention patterns discovered:

1. Position-Based Attention: This attention depends primarily on the relative distances between tokens, suggesting that transformers leverage position-based cues to understand regularity and periodicity in sequences.
2. Boundary-Based Attention: Tokens on NT-end boundaries typically attend to the most adjacent NT-ends, supporting efficient hierarchical processing akin to DP. This attention pattern ensures that the model captures long-range dependencies necessary for parsing CFGs.

Extensions and Robustness

Beyond the primary CFG experiments, the authors investigated implicit CFGs and the robustness of transformers in handling errors. The model's performance on implicit CFGs demonstrated that transformers could encode the distribution of terminal symbols effectively within their token embeddings.

Robustness tests involving corrupted prefixes showed that while models pre-trained on clean data were less resilient against errors, introducing perturbed data during training improved their robustness significantly. This indicates that transformers can adapt to low-quality data, suggesting practical pre-training strategies for real-world applications.

Implications and Future Directions

The paper's findings have significant implications for understanding and improving generative LLMs. By revealing how transformers encode and process CFG structures, this research provides a foundation for future explorations into more complex grammars, including context-sensitive grammars. Additionally, the insights into attention patterns and robustness strategies could inform the development of more efficient and resilient models.

Future research may focus on transferring the CFG learning capabilities to different domains, exploring low-rank updates for task-specific adaptations, and extending the interpretability techniques to other language aspects like semantics, pragmatics, and style.

Conclusion

This paper presents a thorough investigation into how transformers learn and represent CFG structures. By leveraging synthetic datasets and various probing techniques, the researchers have shed light on the internal mechanisms that enable transformers to generate and comprehend complex languages. These insights not only enhance our understanding of current models but also pave the way for future advancements in the field of artificial intelligence and machine learning.