Monotonic Representation of Numeric Properties in Language Models (2403.10381v1)

Abstract: LLMs (LMs) can express factual knowledge involving numeric properties such as Karl Popper was born in 1902. However, how this information is encoded in the model's internal representations is not understood well. Here, we introduce a simple method for finding and editing representations of numeric properties such as an entity's birth year. Empirically, we find low-dimensional subspaces that encode numeric properties monotonically, in an interpretable and editable fashion. When editing representations along directions in these subspaces, LM output changes accordingly. For example, by patching activations along a "birthyear" direction we can make the LM express an increasingly late birthyear: Karl Popper was born in 1929, Karl Popper was born in 1957, Karl Popper was born in 1968. Property-encoding directions exist across several numeric properties in all models under consideration, suggesting the possibility that monotonic representation of numeric properties consistently emerges during LM pretraining. Code: https://github.com/bheinzerling/numeric-property-repr

• The paper demonstrates that language models encode numeric information in low-dimensional, monotonic subspaces using PLS regression.
• It details an intervention method via activation patching that causally manipulates numerical outputs in models like Llama-2 7B and 13B.
• The findings offer practical insights for improving model interpretability, debugging, and knowledge updating in language models.

Monotonic Representation of Numeric Properties in LLMs

Introduction to Monotonic Representations

Recent advancements in LMs have showcased their capability to store and express factual knowledge involving numeric properties, such as dates of birth or population sizes. However, the internal mechanism through which these models encode numeric information remains only partially understood. This paper introduces an innovative approach to identify and manipulate the internal representations of numeric properties within LMs, revealing a low-dimensional, interpretable, and editable subspace where these properties are encoded. Our findings suggest a consistent emergence of monotonic representation of numeric properties across various LMs.

Identifying Property-Encoding Directions

The foundation of our analysis lies in the hypothesis that numeric properties are encoded within low-dimensional linear subspaces of a model’s activation space. We employ partial least squares regression (PLS) to probe the relationship between entity representations in the LM and the corresponding numeric attribute expressions. By correlating activation spaces with numeric outputs across several models and properties, we identify subspaces predictive of numeric attributes. Our empirical findings, using models like Llama-2 7B and Llama-2 13B, indicate that a large fraction of the numeric attribute information is contained within two to six dimensions of these subspaces. Projections onto the top components of PLS regressions reveal clear monotonic structures, reinforcing the hypothesis of low-dimensional encoding of numeric properties.

Causal Effect of Property-Encoding Directions

To transition from correlative evidence to establishing a causal relationship between activation space directions and numeric property expression, we perform interventions through activation patching. By manipulating model activations in targeted directions and observing changes in output, we demonstrate a monotonic relationship between the strength of intervention and the numerical value expressed by the LM. This causal analysis not only confirms the existence of monotonic representations but also highlights the nuanced ways LMs interpret numeric attributes based on their internal representations.

Theoretical and Practical Implications

The discovery of monotonic representation of numeric properties in LMs has both theoretical and practical implications. Theoretically, it enhances our understanding of how abstract numeric information is structured within high-dimensional model internals, suggesting an inherent ability of LMs to organize and utilize this information efficiently. Practically, the ability to edit representation subspaces and predictably alter LM outputs can open new avenues in model interpretability, debugging, and perhaps even in data augmentation and knowledge base updating without retraining.

Future Perspectives in AI

Looking ahead, this work poses several intriguing questions for future research in generative AI and LLMs. The complexity of the discovered subspaces relative to the intrinsic dimensions of activation space, the potential for finding more specific or orthogonal encoding directions, and the impact of these representations on model performance on tasks requiring numeric understanding are areas ripe for exploration. This research lays the groundwork for probing deeper into the representations of not only numeric properties but also more abstract and complex knowledge structures within LLMs.

Conclusion

In summary, our analysis reveals that LLMs encode numeric information in low-dimensional, monotonic subspaces, enabling predictable manipulation of this information. The findings underscore the sophistication with which LMs organize knowledge, providing a new lens through which to view and understand their internal workings. As AI continues to advance, uncovering these underlying structures will be crucial for leveraging the full potential of LLMs in knowledge representation and manipulation.