Rigorously Assessing Natural Language Explanations of Neurons

Abstract: Natural language is an appealing medium for explaining how LLMs process and store information, but evaluating the faithfulness of such explanations is challenging. To help address this, we develop two modes of evaluation for natural language explanations that claim individual neurons represent a concept in a text input. In the observational mode, we evaluate claims that a neuron $a$ activates on all and only input strings that refer to a concept picked out by the proposed explanation $E$. In the intervention mode, we construe $E$ as a claim that the neuron $a$ is a causal mediator of the concept denoted by $E$. We apply our framework to the GPT-4-generated explanations of GPT-2 XL neurons of Bills et al. (2023) and show that even the most confident explanations have high error rates and little to no causal efficacy. We close the paper by critically assessing whether natural language is a good choice for explanations and whether neurons are the best level of analysis.

• The paper introduces a rigorous evaluation framework that integrates observational and intervention-based tests to assess natural language explanations for neurons.
• Observational evaluation assesses how closely neuron activations align with conceptual strings, quantifying precision, recall, and identifying an F1 score of 0.56.
• Intervention-based analysis reveals limited causal efficacy of neurons, challenging the reliability of language-based explanations for actionable model insights.

A Rigorous Evaluation Framework for Natural Language Explanations of Neurons

The paper "Rigorously Assessing Natural Language Explanations of Neurons" addresses a critical challenge in the field of interpretability of LLMs—the evaluation of natural language explanations purportedly detailing the role of individual neurons in these models. The authors establish a clear framework for assessing these explanations through two evaluation modes: observational and intervention-based, both rigorously assessing the explanations' fidelity.

Overview of Evaluation Framework

The framework proposed in the paper delineates two distinct approaches to evaluate explanations that claim certain neurons represent specific concepts:

1. Observational Evaluation: This mode examines whether a neuron's activations accord with the explanations given. It involves defining an explanation as a set of strings related to the concept it is supposed to represent. Observational evaluation tests whether neuron activations align accurately across these strings. The authors underscore the necessity to quantify precision and recall in this mode, thereby identifying both Type I and Type II errors in the explanations.
2. Intervention-Based Evaluation: This mode uses causal relations to verify explanations. It assesses if a neuron functions as a causal mediator for the concept proposed by the explanation. By intervening on neuron's activations, they determine the extent to which neuron manipulation affects model behaviors associated with the concept in question.

Findings from Applying the Framework

The framework was applied to evaluate explanations generated by an automated method involving GPT-4, focusing on neurons in GPT-2~XL. Despite high confidence assigned by GPT-4 to these explanations, observational tests revealed deficiencies—the F1 score was 0.56, far from satisfactory, and decisively pointing to discrepancies between neuron activation predictions and actual activations.

Moreover, intervention-based evaluation revealed limited causal efficacy. Neurons, even when considered collectively, failed to mediate the concepts effectively as per their explanations, often exhibiting effects equivalent to those obtained by random neuron selection.

Implications for Future Research

The implications of these findings are significant for both theoretical understanding and practical applications. The paper propounds that while neurons may show encoding of relevant features, explanations often lack causal grounding. This poses a challenge for downstream tasks like model editing or bias mitigation, which rely on precise neuron-to-concept mappings.

From a theoretical standpoint, it suggests reevaluating natural language as the preferred medium for explanation. The inherent ambiguity and context dependence in language might lead to explanations that are not directly actionable for technical decision-making. Additionally, considering explanations beyond individual neurons might be beneficial, as model reasoning often involves distributed representations transcending single neurons.

Conclusion

This paper advocates for an empirical and rigorous approach to validate neuron-level interpretations within LLMs. By critiquing the effectiveness of explanations derived from heuristic LLMs, it casts a cautious glance toward emerging methods of interpretability. Addressing these findings, future research might focus on formalizing languages for explanations or exploring other hierarchical structures in models that might offer a more interpretable level of analysis beyond the confines of individual neurons.