N2G: A Scalable Approach for Quantifying Interpretable Neuron Representations in Large Language Models (2304.12918v1)

Abstract: Understanding the function of individual neurons within LLMs is essential for mechanistic interpretability research. We propose $\textbf{Neuron to Graph (N2G)}$, a tool which takes a neuron and its dataset examples, and automatically distills the neuron's behaviour on those examples to an interpretable graph. This presents a less labour intensive approach to interpreting neurons than current manual methods, that will better scale these methods to LLMs. We use truncation and saliency methods to only present the important tokens, and augment the dataset examples with more diverse samples to better capture the extent of neuron behaviour. These graphs can be visualised to aid manual interpretation by researchers, but can also output token activations on text to compare to the neuron's ground truth activations for automatic validation. N2G represents a step towards scalable interpretability methods by allowing us to convert neurons in an LLM to interpretable representations of measurable quality.

• The paper introduces the N2G tool, which automatically converts neuron behaviors into interpretable graph representations.
• It employs truncation and saliency methods to extract key tokens that reduce noise and highlight significant neuron activations.
• The method augments diverse sample data and includes automatic validation, enabling scalable and robust LLM interpretation.

"N2G: A Scalable Approach for Quantifying Interpretable Neuron Representations in LLMs" addresses a key challenge in the interpretability of LLMs: understanding the behavior of individual neurons. Mechanistic interpretability involves comprehending how specific components within these models contribute to their overall function. This paper proposes an innovative tool called Neuron to Graph (N2G), which aims to automate and scale the interpretation of neuron behaviors, making it less labor-intensive compared to current manual methods.

Main Contributions

1. Neuron to Graph (N2G) Tool:
   • N2G automatically converts the behavior of neurons into interpretable graphs. It takes as input a neuron and a set of dataset examples where the neuron is active.
   • The method focuses on truncation and saliency techniques to highlight critical tokens, ensuring that only the most relevant parts of the data are presented.
2. Truncation and Saliency Methods:
   • These methods identify and retain only the significant tokens impacting neuron activation.
   • This step is crucial for filtering out noise and redundant information, thereby enhancing interpretability.
3. Augmentation with Diverse Samples:
   • To capture the full range of a neuron's behavior, the dataset examples are augmented with additional, diverse samples.
   • This ensures that the graph representation of the neuron is comprehensive and reflective of its true behavior across varied contexts.
4. Visualization and Interpretation:
   • The resulting graphs can be visualized, aiding researchers in manual interpretation.
   • This visual aid simplifies the process of understanding complex neuron behaviors and relationships.
5. Automatic Validation:
   • N2G is not only a visualization tool; it can also output token activations on new text inputs to compare with ground truth neuron activations.
   • This feature allows for automatic validation of the neuron's interpreted behavior, adding a layer of robustness to the interpretation results.

Impact and Scalability

• Reduction of Labour Intensity: By automating significant portions of the interpretability process, N2G reduces the manual effort required, which is particularly beneficial when dealing with the vast number of neurons in LLMs.
• Scalability: The approach is geared towards scaling interpretability methods, making it feasible to apply to large-scale models. This scalability is critical as models continue to grow in size and complexity.

Conclusion

The N2G tool represents a significant advancement towards scalable interpretability methods for LLMs. By converting neuron behaviors into interpretable and measurable graph representations, it opens up new possibilities for understanding and analyzing the inner workings of complex neural networks. This tool not only aids manual interpretability but also incorporates automated validation, potentially leading to more trusted and transparent AI systems.