Generating Interpretable Networks using Hypernetworks (2312.03051v1)

Abstract: An essential goal in mechanistic interpretability to decode a network, i.e., to convert a neural network's raw weights to an interpretable algorithm. Given the difficulty of the decoding problem, progress has been made to understand the easier encoding problem, i.e., to convert an interpretable algorithm into network weights. Previous works focus on encoding existing algorithms into networks, which are interpretable by definition. However, focusing on encoding limits the possibility of discovering new algorithms that humans have never stumbled upon, but that are nevertheless interpretable. In this work, we explore the possibility of using hypernetworks to generate interpretable networks whose underlying algorithms are not yet known. The hypernetwork is carefully designed such that it can control network complexity, leading to a diverse family of interpretable algorithms ranked by their complexity. All of them are interpretable in hindsight, although some of them are less intuitive to humans, hence providing new insights regarding how to "think" like a neural network. For the task of computing L1 norms, hypernetworks find three algorithms: (a) the double-sided algorithm, (b) the convexity algorithm, (c) the pudding algorithm, although only the first algorithm was expected by the authors before experiments. We automatically classify these algorithms and analyze how these algorithmic phases develop during training, as well as how they are affected by complexity control. Furthermore, we show that a trained hypernetwork can correctly construct models for input dimensions not seen in training, demonstrating systematic generalization.

• The paper demonstrates that hypernetworks can generate interpretable network weights that reveal multiple unseen algorithms for computing the L1 norm.
• Experiments on a simple L1 norm task resulted in three distinct algorithms, highlighting the method's capacity for systematic generalization.
• The study shows that controlling model complexity with hypernetworks allows for auto-classification and a balance between interpretability and performance.

Introduction

In the field of AI, understanding how neural networks make decisions is a significant challenge. While earlier efforts have concentrated on encoding human-understandable algorithms into network weights, which are inherently interpretable, such an approach has inherent restrictions. It disregards the potential for discovering new interpretable algorithms that are currently unknown. This paper proposes addressing this limitation by employing hypernetworks to generate interpretable networks.

Mechanistic Interpretability and Hypernetworks

The research focuses on the question of whether we can strike a balance between interpretability and flexibility in neural networks. By using hypernetworks, the paper aims to reveal undiscovered algorithms that are still understandable to humans. Hypernetworks are particularly suited for this purpose because they generate regular patterns of weight, akin to the notion of interpretability. They also allow control over model complexity by generating a family of networks with varying complexity levels. Through this method, the paper demonstrates the hypernetwork's ability to systematically generalize to new dimensions not encountered during training.

L1 Norm Computation Experiments

The paper utilized a simple task of computing the L1 norm of a vector to illustrate the potential of hypernetworks in generating novel algorithms. Intriguingly, despite the task's apparent simplicity, the hypernetwork generated three unique algorithms – the double-sided algorithm, the pudding algorithm, and the convexity algorithm. Each algorithm sheds light on different computation methods and information-processing strategies that may differ from typical human approaches. The paper also defines order parameters that help in the auto-classification of these algorithms and observes phase transitions in their occurrence during training and with changing model complexity.

Discussion and Conclusion

The findings highlight the complexity of mechanistic descriptions in models designed for simple mathematical tasks and encourage using hypernetworks to explore algorithmic spaces in a systematic way. Through adjustments in complexity control, the research demonstrates that these networks can generalize to compute L1 norms across various input and hidden dimensions, indicating the hypernetworks’ potential in learning general algorithms for any vector size. Nonetheless, the standard algorithm trained by traditional methods, while achieving lower loss, appeared significantly less interpretable, pointing toward the hypernetworks' advantage in creating simpler and more understandable solutions.