Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases (2502.19249v2)

Abstract: Pretraining LLMs on formal language can improve their acquisition of natural language. Which features of the formal language impart an inductive bias that leads to effective transfer? Drawing on insights from linguistics and complexity theory, we hypothesize that effective transfer occurs when two conditions are met: the formal language should capture the dependency structures present in natural language, and it should remain within the computational limitations of the model architecture. We experiment with pre-pretraining (training on formal language before natural languages) on transformers and find that formal languages capturing hierarchical dependencies indeed enable LLMs to achieve lower loss on natural language and better linguistic generalization compared to other formal languages. We also find modest support for the hypothesis that the formal language should fall within the computational limitations of the architecture. Strikingly, pre-pretraining reduces loss more efficiently than training on a matched amount of natural language. For a 1B-parameter LLM trained on roughly 1.6B tokens of natural language, pre-pretraining achieves the same loss and better linguistic generalization with a 33% smaller token budget. Finally, we also give mechanistic evidence of transfer from formal to natural language: attention heads acquired during pre-pretraining remain crucial for the model's performance on syntactic evaluations.

Pre-pretraining on Formal Languages: Inducing Linguistic Biases in LLMs

The paper "Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases" examines the impact of pre-pretraining LLMs on formal languages before they are exposed to natural language data. This approach aims to enhance the models' ability to acquire and generalize natural language patterns by introducing specific inductive biases. The authors, Hu et al., propose a conceptual framework based on insights from linguistics and complexity theory, hypothesizing that formal languages can effectively impart these biases when they capture dependency structures similar to natural languages and remain within a model's computational scope.

Key Findings:

1. Efficiency in Training: Pre-pretraining on formal languages, particularly those characterized by hierarchical dependencies, enhances the efficiency of subsequent natural language training. The study shows that a 1 billion-parameter LLM pre-pretrained on formal languages reached the same loss as baseline models with 33% fewer tokens during training. This efficiency gain highlights the potential of formal languages to reduce computational resources typically required for large-scale model training.
2. Choice of Formal Languages: The selection of formal languages affects the outcome of pre-pretraining. Pre-pretraining on context-sensitive languages, especially those expressible in transformer architectures like $k$-Shuffle Dyck, led to better generalization to natural languages. This suggests that the expressiveness of formal languages should align with both linguistic structure and model capabilities.
3. Mechanistic Insights: The study indicates that attention mechanisms in transformers are particularly influenced by pre-pretraining. Attention heads developed during formal language exposure remain critical for syntactic evaluation tasks even after transitioning to natural language data. This points to the transfer of useful features learned during pre-pretraining, which modify the model's inductive biases.
4. Influence on Linguistic Generalization: Pre-pretraining on formal languages facilitates better performance on linguistic tasks requiring grammaticality judgment and verbatim retrieval—tasks heavily reliant on understanding structural language properties. The results indicate that certain linguistic biases are engrained during the initial exposure to artificial formal structures.

Theoretical Implications:

The paper suggests an intersection between the Chomsky hierarchy of formal languages and circuit complexity, as it pertains to a model's ability to realize these languages, guides the formulation of effective pre-pretraining strategies. This dual consideration not only informs the selection of formal languages for inducing hierarchical biases but also can influence how we think about the broader expressivity limits of machine learning architectures in handling complex data structures.

Practical Implications and Future Directions:

The impressive token efficiency gains point to significant practical benefits, especially in scenarios where computational resources or data are limited. Working with formal languages allows researchers to construct efficient pretraining pipelines, thereby democratizing AI training methodologies by enabling smaller institutions to train large models with fewer resources. Future research might explore adaptive methodologies where pre-pretraining is adjusted dynamically based on the learning progression, potentially incorporating curriculum learning strategies to further optimize computational and data resource utilization.

Moreover, understanding how attention mechanisms are shaped by initial pre-pretraining lends insight into robustness against syntactic variations and model interpretability—a promising direction for improving model interaction with complex, hierarchically-structured languages beyond the scope of current models. This approach, therefore, extends beyond performance improvements, contributing to foundational knowledge on model design and language interaction.