An Interpretability Illusion for BERT (2104.07143v1)

Abstract: We describe an "interpretability illusion" that arises when analyzing the BERT model. Activations of individual neurons in the network may spuriously appear to encode a single, simple concept, when in fact they are encoding something far more complex. The same effect holds for linear combinations of activations. We trace the source of this illusion to geometric properties of BERT's embedding space as well as the fact that common text corpora represent only narrow slices of possible English sentences. We provide a taxonomy of model-learned concepts and discuss methodological implications for interpretability research, especially the importance of testing hypotheses on multiple data sets.

• The paper demonstrates that neuron activations in BERT yield deceptively simple interpretations that vary notably across different datasets.
• The paper presents experiments on datasets like Quora, Wikipedia, and Toronto BookCorpus to expose the role of local semantic coherence and dataset idiosyncrasies.
• The paper calls for cross-dataset validation and standardized methods to ensure robust and consistent interpretability in NLP models.

An Interpretability Illusion for BERT

The paper entitled "An Interpretability Illusion for BERT" discusses a phenomenon encountered when interpreting the BERT LLM. The authors identify what they refer to as an "interpretability illusion," wherein interpretations derived from individual neuron activations or linear combinations of neuron activations appear deceptively simple and meaningful on specific datasets but fail to maintain consistent meanings across different text corpora.

Overview of the Paper

The paper highlights a critical issue in interpretability research within NLP: the false impression that specific neurons in models like BERT encode straightforward, human-interpretable concepts. The authors suggest that such interpretations can be misleading, as different datasets yield disparate concepts that neurons seem to encode. This variability poses significant challenges for understanding and analyzing neural network representations.

The researchers illustrate the illusion through a series of experiments using multiple datasets. They focus on interpreting maximally activating sentences for individual neurons and random directions on datasets such as Quora Question Pairs, Question-answering Natural Language Inference, Wikipedia, and Toronto BookCorpus. Their findings reveal that neurons can show distinct activation patterns linked to different concepts depending on the dataset, highlighting an inherent dataset-induced illusion of interpretability.

Experimental Findings

The experiments conducted involved sorting sentence embeddings according to their activation levels and annotating the emerging patterns. The analysis demonstrated that while certain neurons exhibited consistent activation patterns within a dataset, these patterns were not universally applicable to all datasets. Notably, 80% of annotated neuron activations contained identifiable patterns, but consistent patterns across datasets were rare.

Further analysis revealed three primary sources contributing to the interpretability illusion:

1. Dataset Idiosyncrasy: Each dataset occupies distinct regions within BERT's embedding space, causing top activating sentences to differ across datasets.
2. Local Semantic Coherence: Although some global concept directions exist within BERT's embedding space, local semantic coherence frequently emerged. This coherence results in clusters of semantically similar sentences which appear meaningful but lack a consistent directional encoding of concepts.
3. Annotator Variability: Disparities in pattern recognition among annotators suggest an element of subjectivity in identifying meaningful patterns, contributing further to the illusion.

Implications and Future Research

The implications of this research extend to both practical and theoretical domains. Practically, it emphasizes the necessity of caution in interpretability analyses, advocating for the validation of neuron interpretations across multiple datasets. It also illustrates the need for standardized approaches to interpretability that account for dataset biases and embedding space geometry.

Theoretically, the work raises significant questions regarding the underlying mechanisms of neural representation. It challenges existing methodologies for understanding and dissecting LLMs, advocating for more robust and dataset-neutral paradigms. The authors call for further exploration of BERT’s internal structure, specifically examining earlier layers and other types of neural models.

Conclusion

Through the identification of an interpretability illusion in BERT, the paper underscores the complexity of decoding neural representations in NLP models. The findings advocate for a more critical approach to interpretability research, highlighting the necessity of considering dataset-specific idiosyncrasies alongside local geometries within embedding spaces. Future research might focus on exploring diverse neural architectures, other types of data, and refining interpretability techniques to mitigate illusory interpretations across applications.