The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysis

Abstract: Interpretability provides a toolset for understanding how and why neural networks behave in certain ways. However, there is little unity in the field: most studies employ ad-hoc evaluations and do not share theoretical foundations, making it difficult to measure progress and compare the pros and cons of different techniques. Furthermore, while mechanistic understanding is frequently discussed, the basic causal units underlying these mechanisms are often not explicitly defined. In this article, we propose a perspective on interpretability research grounded in causal mediation analysis. Specifically, we describe the history and current state of interpretability taxonomized according to the types of causal units (mediators) employed, as well as methods used to search over mediators. We discuss the pros and cons of each mediator, providing insights as to when particular kinds of mediators and search methods are most appropriate. We argue that this framing yields a more cohesive narrative of the field and helps researchers select appropriate methods based on their research objective. Our analysis yields actionable recommendations for future work, including the discovery of new mediators and the development of standardized evaluations tailored to these goals.

• The paper demonstrates how causal mediation analysis structures mechanistic interpretability by categorizing neural mediators based on granularity and alignment.
• It contrasts discrete mediators like neurons and attention heads with coarse units such as layers, highlighting trade-offs like polysemanticity versus generality.
• The study shows that optimization-based search methods can enhance model understanding, driving improvements in AI transparency and safety.

The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysis

Introduction

Mechanistic interpretability endeavors to comprehend the functional roles of neural network components, aiming for a deeper algorithmic understanding of neural network behaviors. Traditionally fragmented, this field lacks unified evaluation frameworks, making comparison and progress measurement challenging. The discussed paper proposes structuring interpretability research through causal mediation analysis, harmonizing diverse methods by categorizing them according to the types of causal units employed and their respective pros and cons. This methodology intends to provide a coherent narrative of the field and assists researchers in selecting apt methods for different research objectives.

Figure 1: Outline of survey. Necessary causal terminology is defined, setting the stage for differentiating and contextualizing various mechanistic interpretability methods.

Causal Mediation Analysis Framework

Causal mediation analysis constitutes the core analytical lens proposed for mechanistic interpretability. This statistical framework allows researchers to explore how intermediate nodes in neural networks influence model outputs, effectively enabling a more structured examination of neural components' functionality. The framework integrates direct and indirect impacts, solidifying the basis for understanding functional roles and interaction mechanisms within neural computation graphs. By mapping neural components to causal graphs, the framework supports nuanced insights into how models establish complex dependencies among variables.

Figure 2: Illustration of causal mediation analysis applied to neural networks, emphasizing how changes in mediator states can alter predicted outcomes.

Mediator Types and Trade-Offs

The paper categorizes mediators by their granularity and alignment. Neurons and Attention Heads serve as naturally discrete, fine-grained mediators, offering high resolution in pinpointing functional roles but often suffer from polysemanticity—where multiple concepts are encoded by a single unit. Submodules and Layers are coarser mediators providing generality across tasks and simpler enumeration for analyses. Conversely, Non-basis-aligned Spaces allow the discovery of human-interpretable features via linear combinations of neuron activations, which often yield monosemantic and sparse representations, conducive to detailed functional examinations.

Figure 3: Visualization of common mediator types in neural networks. It highlights the shift towards exploring non-basis-aligned spaces for extracting monosemantic features.

Search Methods for Mediator Discovery

An array of search methodologies are delineated, catering to different mediator types. Exhaustive Search and Gradient-Based Approximations find application in scenarios involving countable mediators. For continuous or vast mediator spaces, Optimization-based Searches—comprising both supervised and unsupervised methods—are employed. These include methods like supervised probing, which aligns representations with human-specified concepts, and sparse autoencoders for interpretable direction discovery in latent space. Importantly, optimization techniques are susceptible to biases inherent in training processes, prompting the need for rigorous validation.

Figure 4: Neurons are not guaranteed to encode interpretable features. The challenge of non-orthogonal feature representation necessitates methods for identifying non-basis-aligned mediators.

Discussion on Practical and Theoretical Implications

The discourse extends to implications of these insights for future developments in AI:

1. Model Understanding: Improved interpretability aids in demystifying behaviors of black-box models, promoting safer and more reliable AI deployments.
2. Algorithmic Advancements: Structured methodologies can guide the development of inherently interpretable models or architectures, enhancing controllability and comprehensibility.
3. Causality Theory Integration: Deepening the theoretical integration of causality concepts across AI systems could further refine these approaches, leading to more transparent models.

Conclusion

The quest for the right causal mediator is essential for gaining profound insights into a neural network's inner workings. The outlined approach via causal mediation analysis offers a pragmatic pathway to unify disparate methodologies, encouraging consistency and comparison in mechanistic interpretability research. As AI systems grow increasingly complex, such structured frameworks are vital for ensuring interpretability alongside performance, security, and ethical alignment.

Figure 5: Example of alignment search, demonstrating the process of isolating specific causal variables within the computation graph, based on the proposed causal mediation analysis framework.