Linear Correlation in LM's Compositional Generalization and Hallucination (2502.04520v1)

Abstract: The generalization of LLMs (LMs) is undergoing active debates, contrasting their potential for general intelligence with their struggles with basic knowledge composition (e.g., reverse/transition curse). This paper uncovers the phenomenon of linear correlations in LMs during knowledge composition. For explanation, there exists a linear transformation between certain related knowledge that maps the next token prediction logits from one prompt to another, e.g., "X lives in the city of" $\rightarrow$ "X lives in the country of" for every given X. This mirrors the linearity in human knowledge composition, such as Paris $\rightarrow$ France. Our findings indicate that the linear transformation is resilient to large-scale fine-tuning, generalizing updated knowledge when aligned with real-world relationships, but causing hallucinations when it deviates. Empirical results suggest that linear correlation can serve as a potential identifier of LM's generalization. Finally, we show such linear correlations can be learned with a single feedforward network and pre-trained vocabulary representations, indicating LM generalization heavily relies on the latter.

• The paper demonstrates a stable linear transformation between logits that maps relationships like city to country, maintaining consistency even after fine-tuning.
• The paper reveals that precise linear mappings enable successful generalization, while imprecise ones lead to hallucinations and incorrect updates.
• The paper highlights the pivotal role of vocabulary embeddings, showing that simplified architectures can still preserve key generalization patterns.

Analysis of Linear Correlation in LLM Compositional Generalization and Hallucination

The paper “Linear Correlation in LM’s Compositional Generalization and Hallucination” provides a meticulous examination of LLMs (LMs), particularly focusing on their ability to generalize and hallucinate based on linear correlations in next token prediction (NTP). The authors present insightful analyses on the existence and implications of linear transformations between logits of related knowledge pairs in LMs, such as the mapping from city to country, and the factors that contribute to these transformations.

Key Findings

The research introduces a method to identify linear correlations in LMs, focusing on the transformation of knowledge representations. By analyzing transformations between logits of related NTPs (e.g., City -> Country), the authors uncover linear correlation patterns. The core findings can be summarized as follows:

1. Existence of Resilient Linearity: The paper demonstrates that there is a linear transformation, characterized by a matrix $W$ and bias $b$, that maps the logits corresponding to one prompt to another, such as "X lives in the city of" to "X lives in the country of". This transformation remains stable even after large-scale fine-tuning, indicating that such linear correlations are ingrained in the structural parameters of LMs.
2. Generalization and Hallucination: High correlation between source and target knowledge leads to successful generalization only if $W$ is precise. Otherwise, imprecise $W$ may cause hallucinations, exemplified by instances where changes to one piece of knowledge unintentionally induce incorrect updates to another, such as misconceiving the country for a city.
3. Dependency on Vocabulary Representations: The research highlights the role of vocabulary representations in forming and maintaining these linear correlations. By substituting complex LM architectures with simplified structures, such as a mean-pooling layer and a feedforward network, the paper observes that generalization patterns persist, implicating vocabulary embeddings as a fundamental component.

Implications

The exploration of linear correlational behaviors in LMs extends our understanding of how LMs organize and utilize knowledge:

• Practical Implications: Recognizing linear correlations allows for improved diagnostic tools to identify and address the hallucinations in LMs that arise from imprecise knowledge correlations. This can lead to more robust model design and training practices that prevent incorrect generalization.
• Theoretical Implications: The findings contribute to the ongoing discourse on how LMs generalize information. The paper provides evidence that LMs might employ linearity as a mechanism to connect related semantic domains, mimicking rudimentary aspects of human knowledge organization.
• Future Obstacles and Directions: While the paper elaborates on the observation of linear correlations, understanding the foundational causes of these correlations remains an open challenge. Future investigations could probe into which specific data properties or architectural characteristics are pivotal in nurturing such linearities. Similarly, identifying which pairs of knowledge might naturally exhibit these properties could streamline fine-tuning strategies for enhanced knowledge updates in LMs.

Conclusion

This paper presents a compelling evaluation of the linear dependencies formed within LMs during compositional generalization and demonstrates significant implications for improving the reliability and interpretability of LMs. The research encourages further exploration into model architectures and training data compositions to harness the potential of linear transformation for robust model performance while mitigating unintended consequences of hallucinations. The work exemplifies a continuation in understanding the nuanced capabilities and limitations of LMs, offering valuable avenues to optimize their deployment in diverse applications.