Understanding polysemanticity in neural networks through coding theory (2401.17975v1)

Abstract: Despite substantial efforts, neural network interpretability remains an elusive goal, with previous research failing to provide succinct explanations of most single neurons' impact on the network output. This limitation is due to the polysemantic nature of most neurons, whereby a given neuron is involved in multiple unrelated network states, complicating the interpretation of that neuron. In this paper, we apply tools developed in neuroscience and information theory to propose both a novel practical approach to network interpretability and theoretical insights into polysemanticity and the density of codes. We infer levels of redundancy in the network's code by inspecting the eigenspectrum of the activation's covariance matrix. Furthermore, we show how random projections can reveal whether a network exhibits a smooth or non-differentiable code and hence how interpretable the code is. This same framework explains the advantages of polysemantic neurons to learning performance and explains trends found in recent results by Elhage et al.~(2022). Our approach advances the pursuit of interpretability in neural networks, providing insights into their underlying structure and suggesting new avenues for circuit-level interpretability.

• The paper demonstrates that neural networks develop polysemantic codes that balance redundancy and channel capacity, as shown by power-law eigenspectrum decay.
• The paper employs neuroscience and information theory methods to link network architecture with interpretability and error correction capabilities.
• The paper finds that training techniques like dropout significantly impact code redundancy, enhancing network robustness to adversarial perturbations.

Understanding Polysemanticity in Neural Networks Through Coding Theory

The paper, "Understanding Polysemanticity in Neural Networks Through Coding Theory," presents an innovative examination of neural network interpretability by leveraging concepts from neuroscience and information theory. By dissecting the polysemantic nature of neurons—where neurons respond to disparate features—the authors advance theoretical and practical understandings of neural network interpretability, density of codes, and effective channel coding.

Conceptual Framework and Methodology

The authors employ tools from neuroscience and information theory to address the polysemantic nature of neurons. They explore the eigenspectrum of the activation's covariance matrix to infer redundancy levels and assess how effective these codes are in facilitating interpretability.

A key line of inquiry is the investigation of whether neural networks can adopt superposition codes, where single neurons participate in multiple unrelated contexts, thus bypassing traditional monosemantic interpretations. By studying the decay rate of the eigenspectrum, they illuminate how network architecture and coding dynamics influence the robustness and interpretability of neural networks. This approach aligns with the insights proposed by Elhage et al. (2022) concerning the benefits of polysemantic neurons, serving both learning performance and interpretability.

Results and Numerical Insights

The paper reveals that neural networks tend to form intricate codes that strategically balance redundancy and channel capacity. Through their analyses, the authors demonstrate that:

• Redundancy Detection: The eigenspectrum of neural networks often exhibits a power-law decay, indicating varying degrees of redundancy. Steeper decays correspond to more redundant codes, which the paper posits may be employed for improved error correction.
• Network Robustness: There is an observed correlation between network robustness to adversarial attacks and interpretability. This robustness is enhanced by optimizing redundancy within the network structure.
• Effect of Dropout: The inclusion of dropout during training impacts code redundancy significantly. Higher dropout levels correspond with a rapid decay in the eigenspectrum, leading to codes that are more robust to perturbations.

The authors also discuss the implications of these findings on the understanding of LLMs, such as GPT-2, noting how polysemanticity and superposition could contribute to a model's ability to encode more information efficiently.

Implications and Future Directions

The analysis of polysemanticity through the lens of coding theory presents implications for both interpretability and neural network architecture design. The paper elucidates theoretical connections between eigenspectrum decay and biologically-inspired coding mechanisms, hinting at parallels with neural processing in large animals.

Speculation on the application of these insights in large-scale LLMs suggests potential in harmonizing single neuron interpretability methods with random projection-based approaches, possibly enhancing current methodologies in understanding model outputs comprehensively.

As a promising avenue for future exploration, the paper suggests investigating the nonlinearity within network codes and their influence on both coding efficiency and interpretability. The articulation of how linear and nonlinear codes are optimally employed under different architecture and training circumstances could offer further depth to both foundational and applied machine learning research.

Conclusively, this paper provides a multifaceted view of neural network interpretability by intersecting the domains of information theory and neuroscience, fostering a deeper comprehension of both network efficiency and how polysemantic codes might be designed for practical robustness and theoretical interpretability.