From Neurons to Neutrons: A Case Study in Interpretability (2405.17425v1)

Abstract: Mechanistic Interpretability (MI) promises a path toward fully understanding how neural networks make their predictions. Prior work demonstrates that even when trained to perform simple arithmetic, models can implement a variety of algorithms (sometimes concurrently) depending on initialization and hyperparameters. Does this mean neuron-level interpretability techniques have limited applicability? We argue that high-dimensional neural networks can learn low-dimensional representations of their training data that are useful beyond simply making good predictions. Such representations can be understood through the mechanistic interpretability lens and provide insights that are surprisingly faithful to human-derived domain knowledge. This indicates that such approaches to interpretability can be useful for deriving a new understanding of a problem from models trained to solve it. As a case study, we extract nuclear physics concepts by studying models trained to reproduce nuclear data.

• The paper demonstrates that neural networks can recover known nuclear physics laws, such as shell effects and pairing phenomena, from binding energy data.
• The study employs embedding analysis and symbolic matching, using principal component projections and cosine similarity to link latent features with physics models.
• The research implies that mechanistic interpretability in AI not only enhances prediction accuracy but also deepens scientific understanding in interdisciplinary fields.

From Neurons to Neutrons: A Case Study in Interpretability

The paper "From Neurons to Neutrons: A Case Study in Interpretability" explores the capacity of mechanistic interpretability (MI) to derive meaningful scientific insights from machine-learned models trained on nuclear physics data. This paper hinges on the hypothesis that neural networks, when trained on high-dimensional data, can learn low-dimensional representations that are not only useful for accurate predictions but also interpretable through a mechanistic lens, providing scientifically meaningful insights.

Core Contributions and Key Findings

Mechanistic Interpretability in Nuclear Physics

The authors use nuclear binding energy predictions as a case paper to test whether neural networks can encapsulate and reveal human-derived scientific concepts. Models trained merely to predict binding energies and other nuclear properties, such as neutron and proton separation energies, showed significant potential to rediscover known physical laws and structures within the data.

Embedding Analysis

A central discovery in this paper is the formation of a helical structure in the embeddings of proton (Z) and neutron (N) numbers. The identified helix aligns with known physical phenomena, such as the volume term of the Semi-Empirical Mass Formula (SEMF), which scales with the total number of nucleons $A = N + Z$. The periodicity and ordering observed in the principal components (PCs) of embeddings are indicative of underlying physical laws, like the pairing effect and the trend towards higher binding energy with an increasing number of nucleons.

Hidden Layer Feature Analysis

The paper explores the penultimate layer activations to uncover symbolic representations that align with physical terms in nuclear theory. For instance, the primary components of latent features correspond to the volume term, pairing term, and more intricate shell effects as predicted by the nuclear shell model. The authors employ cosine similarity to correlate these AI-extracted features with physics-derived formula components, showing how neural networks can inherently discover and utilize domain-relevant knowledge.

Implications and Future Outlook

Enhanced Scientific Discovery

This work demonstrates the practical potential for neural networks to not only predict outcomes but also help identify and understand the scientific principles governing the data. This capacity can profoundly impact fields where data is abundant, but theoretical understanding lags, or where existing theories are known to be approximations, such as astrophysics, materials science, and genomics.

Symbolic Regression and Physics Modeling

A noteworthy step in utilizing learned representations is their application in symbolic regression to recover physics models. The paper's symbolic regression efforts yield expressions that approximate the SEMF and hint at more accurate corrections, albeit less interpretable ones. Future work could refine these techniques, enhancing the interpretability of derived models.

Methodological Advances

The authors outline a rigorous methodology that combines neural network training with systematic representation analysis. By projecting embeddings and activations into principal component spaces and examining their structures, the paper advocates for a comprehensive interpretability approach applied to scientific data-driven models. This approach includes:

1. Latent Space Topography: Using projections onto principal components to visualize how changes in latent features affect predictions.
2. Helix Parameterization: Fitting and perturbing helix parameters to understand their implications on model outputs.
3. Symbolic Matching: Employing cosine similarity for feature comparison between AI-derived components and known physical terms.

Conclusion

In summary, the paper posits that mechanistic interpretability, when applied to models trained on scientific data, can lead to the rediscovery of known principles and identification of novel insights. This approach is shown to be particularly effective in domains such as nuclear physics, where both well-understood areas and unresolved questions coexist. By revealing how machine-learned representations compare with human-derived theories, this work opens new avenues for integrating AI into scientific discovery, providing both practical tools for better model understanding and theoretical opportunities for advancing domain knowledge. As computational power and modeling techniques progress, the potential for such interdisciplinary applications of AI will only grow, promising further breakthroughs at the nexus of data science and fundamental research.