Causal Faithfulness & Activation Patching

• The paper establishes that causal faithfulness, validated by activation patching, rigorously tests the sufficiency and necessity of neural network circuits with quantitative recovery metrics.
• The methodology replaces corrupted activations with clean ones to isolate circuit contributions and diagnose their impact on output behavior.
• Scalable approximations like EAP, EAP-IG, and Relevance Patching enable efficient evaluation of activation patching in large-scale neural models.

Causal faithfulness via activation patching is a central criterion for mechanistic interpretability, denoting when a subset of components (a "circuit") in a neural network is causally sufficient and necessary to explain specific behaviors. Activation patching, and its methodological variants, have become the principal means to operationally define and empirically test this property in large-scale models and across interpretability tasks.

1. Circuit Faithfulness: Formal Definitions

Circuit faithfulness refers to the property that when all model edges outside a designated circuit $C$ are ablated (i.e., their activations are replaced with counterfactual values from a corrupted input), the model’s performance on the clean task is unchanged. Given a transformer LLM $f$ and a task metric $M(x)$, with activations $Z_u(x)$ and $Z'_u(x')$ at component $u$ under clean and corrupted inputs respectively, a circuit $C \subseteq E$ (edges in the computational graph $G=(V,E)$) is said to be $\epsilon$-faithful if:

$\forall x: |M(f(x)) - M(f_C(x, x'))| \leq \epsilon$

where $f$0 denotes outputs under the intervention that corrupts all edges outside $f$1 (Hanna et al., 2024). Faithfulness requires both necessity (removal of part of the circuit diminishes the behavior) and sufficiency (retaining only the circuit suffices for the behavior).

This definition extends to layerwise interventions as in Causal Layer Attribution via Activation Patching (CLAP), which quantifies the recovery percentage when specific layer activations are repaired (Bahador, 3 Apr 2025).

2. Activation Patching Methodology

Activation patching, also known as causal tracing, causal mediation analysis, or interchange intervention, is a mechanistic intervention technique. It consists of:

1. Running the model on a clean input to cache activations.
2. Running on a corrupted input (in-distribution corruption preferred).
3. Overwriting the activation of a specific component (e.g., a neuron, head, or layer) in the corrupted forward pass with the corresponding clean activation.
4. Evaluating whether the output recovers the original target behavior via metrics such as probability, logit difference, or KL divergence (Zhang et al., 2023, Hanna et al., 2024).

Componentwise patching can also be extended to subspace interventions, where only a specified subspace of a component’s activation is replaced (Makelov et al., 2023).

Faithfulness is diagnosed by measuring, for example, the normalized recovery:

$f$2

with $f$3 being the metric after patching (Hanna et al., 2024).

3. Scalable and Approximate Methods

Naive activation patching is computationally infeasible for modern models due to the $f$4 cost. This has motivated several scalable approximations:

• Edge Attribution Patching (EAP): Approximates the effect of an edge by a first-order Taylor expansion. For edge $f$5, the effect is

$f$6

Scores $f$7 rank edges; top edges construct candidate circuits.

• Integrated Gradients for Patching (EAP-IG): Addresses EAP’s abrupt-gradient failures by averaging gradients along linear interpolations between clean and corrupted activations:

$f$8

EAP-IG circuits consistently achieve higher faithfulness than EAP at the same circuit size, despite similar edge/node overlap with ground-truth circuits (Hanna et al., 2024).

• Relevance Patching (RelP): Replaces local gradients with Layer-wise Relevance Propagation (LRP) coefficients, leading to a substantial increase in faithfulness as measured by Pearson correlation to ground-truth patching (e.g., from 0.006 to 0.956 on GPT-2 Large MLP outputs) and at parity with Integrated Gradients but at lower computational cost (Jafari et al., 28 Aug 2025).
• AtP*: Introduces QK-fix for attention-related saturation and GradDrop for direct-indirect effect cancellation. AtP* achieves rank-faithful approximations and provides a statistical guarantee (via subsampling) on missed contributors (Kramár et al., 2024).

The effectiveness and diagnostic reliability of these approximations are always anchored to their agreement with ground-truth causal patching.

4. Empirical Faithfulness Assessment and Best Practices

Causal faithfulness demands rigorous empirical validation protocols. Best practices established by recent surveys include (Zhang et al., 2023, Bahador, 3 Apr 2025):

• Use in-distribution corruption (symmetric token replacement) to avoid spurious OOD effects.
• Report both sufficiency and comprehensiveness: sufficiency quantifies how much of a behavior is retained using only the identified components; comprehensiveness measures the reduction when those components are ablated.
• Prefer logit-difference metrics over raw probabilities.
• Conduct per-layer and per-component patching, interpret 100% recovery as evidence for localization and partial recovery for distributed computation.
• Validate the causal role of subspaces via nullspace ablation, circuit tracing, and cross-validation.
• Confirm findings using multiple task instances and statistical significance testing.
• For complex behaviors, employ window patching to capture synergy and necessity of sustained intervention (Akarlar, 16 Apr 2026).

5. Faithfulness in Subspace and Combined-Model Abstractions

Faithfulness is subtle in subspace interventions. A subspace patch can spuriously achieve the intended output change by activating dormant or disconnected pathways. To certify that a subspace is truly causal:

• Demonstrate that patching only the rowspace (as determined by output weights) retains the effect.
• Show strong in-distribution correlation between subspace projections and the target feature.
• Verify that ablating the disconnected part does not diminish the effect, and full-component patches are consistent (Makelov et al., 2023).

When no single mechanism is faithful across all inputs, input-dependent combinations of simple models—aligned and verified using activation patching—can yield more accurate abstractions. The resulting trade-off curve between faithfulness (interchange intervention accuracy) and model strength (input coverage) provides a systematic basis for selecting between candidate explanations (Pîslar et al., 14 Mar 2025).

6. Interpretability, Explanations, and Future Directions

Causal faithfulness underpins the reliability of both mechanistic and natural-language explanations extracted from network components. Faithful explanations must cite components empirically verified by activation patching, as sufficiency alone may not guarantee comprehensiveness due to distributed backup mechanisms (Mahale, 13 Feb 2026, Yeo et al., 2024). In natural-language settings, Causal Faithfulness metrics (e.g., CaF, measuring the alignment of attribution matrices under patching) provide a principled alternative to correlational heuristics, distinguishing plausible but unfaithful explanations (Yeo et al., 2024, Zaman et al., 26 Feb 2025).

Open research challenges include reducing the cost of faithful patching for large models, reliably approximating interventional effects without faithfulness loss, extending faithfulness analysis to black-box and data-limited contexts, and synthesizing causally faithful interface layers for end-to-end debugging and alignment.

---

In summary, causal faithfulness via activation patching is both a formal criterion and an operational standard for validating mechanistic accounts of network computation. The combination of rigorous intervention protocols and scalable approximations enables quantitative, falsifiable claims about the internal causal organization of modern neural models, establishing faithfulness—not mere overlap—as the baseline for credible interpretability (Hanna et al., 2024, Jafari et al., 28 Aug 2025, Kramár et al., 2024, Zhang et al., 2023, Makelov et al., 2023, Bahador, 3 Apr 2025, Akarlar, 16 Apr 2026, Mahale, 13 Feb 2026, Yeo et al., 2024, Pîslar et al., 14 Mar 2025, Zaman et al., 26 Feb 2025).