Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models (2301.04213v2)

Abstract: LLMs learn a great quantity of factual information during pretraining, and recent work localizes this information to specific model weights like mid-layer MLP weights. In this paper, we find that we can change how a fact is stored in a model by editing weights that are in a different location than where existing methods suggest that the fact is stored. This is surprising because we would expect that localizing facts to specific model parameters would tell us where to manipulate knowledge in models, and this assumption has motivated past work on model editing methods. Specifically, we show that localization conclusions from representation denoising (also known as Causal Tracing) do not provide any insight into which model MLP layer would be best to edit in order to override an existing stored fact with a new one. This finding raises questions about how past work relies on Causal Tracing to select which model layers to edit. Next, we consider several variants of the editing problem, including erasing and amplifying facts. For one of our editing problems, editing performance does relate to localization results from representation denoising, but we find that which layer we edit is a far better predictor of performance. Our results suggest, counterintuitively, that better mechanistic understanding of how pretrained LLMs work may not always translate to insights about how to best change their behavior. Our code is available at https://github.com/google/belief-localization

Analyzing the Relationship Between Causal Localization and Model Editing in LLMs

The paper "Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in LLMs" explores the intricate relationship between the localization of factual information in LLMs and the efficacy of model editing techniques. The authors aim to discern whether causality-based localization insights can accurately inform model editing operations, a question that addresses the broader challenge of understanding and manipulating the behavior of pretrained LLMs (PLMs).

Core Findings

The paper reveals that the presumed connection between factual information localization, ascertained through Causal Tracing, and the success of model editing, particularly in replacing stored facts, is unexpectedly tenuous. The authors rigorously demonstrate several key findings:

1. Disconnect Between Localization and Editing: The authors show that there is a negligible correlation between the localization results from techniques such as Causal Tracing and the success of model editing in injecting new information into PLMs. This stands in stark contrast to the prior assumption that knowing where the information is stored in a model would naturally guide effective modifications.
2. Evaluation of Editing Methods: The paper systematically evaluates multiple model editing approaches, including ROME and MEMIT, across various layers of a LLM. It finds that success with these methods is largely uncorrelated with where information is localized in the model, challenging the rationale behind their design.
3. Variants of the Editing Problem: By exploring different editing problem variants—such as Tracing Reversal, Fact Erasure, Fact Amplification, and Fact Forcing—the authors aim to bridge the gap between localization insights and editing success. Surprisingly, while Fact Forcing shows a somewhat stronger correlation, tracing results continue to provide limited predictive value.

Numerical Insights

The numerical evidence demonstrates that for the most part, edit success explains nearly none of the variance that can be accounted for by locating factual information via Causal Tracing. For instance, tracing effects contribute only marginally to the variance in success metrics, even when optimizing editing methods using models such as GPT-J and GPT2-XL.

Implications of the Research

Theoretically, this exposes a critical gap in our understanding of PLMs' internal mechanisms. It suggests that factors other than the precise storage location of information are influencing the model's capacity for successful adaptation through editing. It also signals that the interventions made in pretrained transformer networks should consider factors beyond mere layer-wise localization for the effective modification of stored knowledge.

The paper urges a re-evaluation of how we conceptualize the internal workings of LLMs. The paper proposes that while localization methods like Causal Tracing yield valuable insight into model internals, they do not directly inform optimal editing strategies. Consequently, it challenges researchers to rethink the methodological frameworks guiding model manipulations.

Future Directions

In terms of future work, this paper sets the stage for exploring more nuanced connections between neural representations and model editing success. It calls for a deeper investigation into why specific layers contribute to successful editing beyond trace-based localization, potentially focusing on the broader, systemic interactions across model layers.

Additionally, the insights gleaned can drive the development of more sophisticated editing techniques that do not solely rely on localization-based guidance. Such advancements might leverage other model introspection methods or machine learning approaches that capture model dynamics not currently addressed by tracing or zeroing methods.

In conclusion, this paper provides a pivotal reconsideration of the efficacy of localization as a predictive tool for model editing and emphasizes the need for a richer understanding of LLMs' internal processes to better guide future model manipulation endeavors.