The Curse of Multiple Mediators: Hidden Interaction Effects in Activation Patching

Abstract: Activation patching is the primary tool in mechanistic interpretability. It attributes causal responsibility for a model behavior to each of its individual components by estimating its natural indirect effect (NIE). Re-deriving the activation patching estimand from causal mediation analysis, we find that the NIE does not solely capture the causal effect through the specific component. It also contains interaction effects (INT) that measure how much the component's causal effect itself depends on the state of other components in the model. A natural response may be to try to eliminate INT by adjusting the estimator or unit of analysis, but each of these potential remedies has predictable failure modes. We demonstrate these failure modes in the GPT-2 IOI circuit; components whose causal importance is conditional on the state of other components are either invisible or artificially inflated, and INT variance explains the previously documented instability of faithfulness scores. We prove that INT scales with the distance between clean and patched component activations, is negligible when the model is locally affine, and decomposes combinatorially into pairwise and higher-order group interactions. Despite its inevitability, INT is not a nuisance to be eliminated, but rather a diagnostic for interpretability studies. Its individual and group-level magnitude and sign signal when causal conclusions are prompt-dependent, and when greedy NIE-based component ranking will miss mechanisms only discoverable through combinatorial search.

• The paper reveals that activation patching in transformers is confounded by hidden interaction effects, as the natural indirect effect (NIE) splits into a pure effect (PIE) and an interaction term (INT).
• The paper demonstrates quantitatively that INT significantly alters head importance rankings, with rank correlations dropping to around 0.51 in GPT-2’s IOI task when INT is substantial.
• The paper highlights that multi-component patching leads to combinatorial explosions and dominant cross-interactions, urging the development of new heuristics for reliable causal attribution.

The Curse of Multiple Mediators: Hidden Interaction Effects in Activation Patching

Causal Attribution in Mechanistic Interpretability

Mechanistic interpretability, particularly in transformer-based LLMs, hinges on the ability to localize and quantify the causal role of internal components—predominantly attention heads and MLP layers—via activation patching. Activation patching is underpinned by causal mediation analysis, operationalizing the natural indirect effect (NIE) of each mediator. The NIE is implicitly assumed to reflect the path-specific causal contribution of a component by measuring the change in model output when substituting its activation with a counterfactual value. However, the theoretical underpinnings from causal inference reveal structural pitfalls due to residual skip connections, which guarantee the existence of bypass pathways for every mediator. This intrinsic architectural property, combined with the nonlinearity of transformers, introduces interaction effects (INT) that distort NIE estimates, rendering standard causal attributions unreliable in many contexts.

Formalizing Interaction Effects in Transformers

The paper rigorously develops the mediation decomposition for transformer circuits. The NIE contains not only the pure indirect effect (PIE), which captures the genuine path-specific contribution through a component, but also an INT term that quantifies how the component's causal influence depends on the activations of other components on the bypass path. The core finding is the decomposition $\text{NIE} = \text{PIE} + \text{INT}$, where INT is a mixed second-order derivative (Hessian bilinear form) of the downstream function with respect to both the component and the bypass perturbations. This leads to strong numerical divergences between NIE and PIE-based rankings of head importance: rank correlation as low as $\rho = 0.51$ is observed for the GPT-2 IOI task when INT is substantial (Figure 1).

Figure 1: Spearman rank correlation between mean NIE and mean PIE per head, demonstrating strong ranking disagreement when INT is large; INT scales linearly with patch distance across circuit heads.

The paper demonstrates that INT is negligible when the model's downstream mapping is locally affine or perturbations are small—that is, when clean and counterfactual prompts are semantically close or in synthetic benchmark models like Tracr.

Multi-Component INT and Faithfulness Decomposition

While single-component patching is the predominant approach for causal localization, practical circuit evaluations consistently involve ablation or selection of multiple nodes. The multi-component mediation analysis reveals a combinatorial explosion: INT decomposes into individual, pairwise, and higher-order group interactions among patched components. Crucially, cross-interactions (xINT) are invisible to per-head patching and significantly dominate in circuit faithfulness metrics, explaining much of the prompt-level variability and disagreement documented in prior work.

Figure 2: Faithfulness decomposition $$F = \sum\text{PIE} + \sum\text{sINT} + \sum\text{xINT}$$ on the IOI task; cross-interaction terms dwarf the sum of individual PIE effects and account for most faithfulness gap and variance.

The decomposition quantitatively shows that group-level INTs markedly reduce the net circuit causal effect: under pABC corruption, summed PIE contributions reach $+5.78$ logit-diff, but circuit faithfulness is only $+1.52$, with negative xINT of $-4.05$ absorbing most of the discrepancy. INT components also exhibit larger per-prompt standard deviations compared to PIE.

INT Patterns in GPT-2 IOI Circuit

The IOI circuit in GPT-2 small provides a unique testbed due to its manually characterized head functional groups and prior combinatorial ablation. The analysis reveals three qualitatively distinct INT signatures across circuit heads:

• Backup Compensation (INT < 0, NIE < PIE): Head contributions are suppressed when the rest of the circuit is intact, particularly among Name Mover and Backup Name Mover heads.
• Context-Specific Mechanisms (INT > 0, PIE ≈ 0): Heads contribute only when specific downstream conditions hold, such as Duplicate Token and Induction heads.
• Harmful Mechanisms (PIE < 0, INT > 0): Heads that negatively influence task performance, mitigated by interactions with other mechanisms.

  Figure 3: Scatter plot of mean NIE vs. mean PIE per head on IOI, colored by functional group; backup and context-specific heads diverge sharply in INT.

Strong numerical evidence supports systematic ranking failures: NIE-based rankings underrank backup mechanisms (many backup heads relegated to ranks 11–116), while PIE-based rankings exclude context-specific heads (many fall below rank 30 despite functional importance), as shown in Figure 4.

Figure 4: All 144 GPT-2 small heads ranked by signed mean; backup heads are misranked under NIE, context-specific mechanisms are missed under PIE.

Pairwise and group-level INT terms also indicate candidate competing mechanisms, e.g., mutual opposition between S-Inhibition and Negative Name Mover heads, diagnosed via negative pairwise cross-INT.

Implications for Interpretability, Practice, and Future Directions

This work establishes that interaction effects are a fundamental, inseparable artifact of causal mediation in neural networks with redundant and nonlinear computational pathways. As transformer width and task complexity increase, INT will grow increasingly dominant. Consequently, causal attribution via single-component patching is not only insufficient for modular explanation, but systematically misrepresents functional importance. The only theoretically sound remedy—combinatorial search across mediators—is intractable for modern models. However, the quantitative diagnostic value of INT, as advocated in epidemiology, can inform prompt-dependence, functional overlap, and mechanism intersection.

In practical terms, future research should prioritize heuristics and clustering methods to guide combinatorial search, leverage concept geometry for selective steerability, and develop standardized evaluations sensitive to interaction-driven effects. INT analysis, if integrated into interpretability agents and benchmark taxonomies, could facilitate tractable approximations of mechanism discovery and permit selective trust in attribution-based conclusions.

Conclusion

The paper rigorously demonstrates that interaction effects pervade activation patching, distorting causal attribution and circuit evaluations in transformer models. INT is not a nuisance to be eliminated; it is an indispensable diagnostic, quantifying functional dependencies and revealing hidden mechanism interactions. Its measurement enables researchers to limit the scope of combinatorial search and ground mechanistic claims in empirically validated model behavior. The findings demand a paradigm shift: modular explanations in contemporary neural networks must explicitly account for interaction effects to avoid pathological misinterpretation of internal representations (2606.27510).