Two-Stage Path Patching in Neural Networks

• The paper introduces two-stage path patching, a method that evaluates path-specific contributions in a neural network via counterfactual interventions.
• It operationalizes model localization by defining and testing hypotheses about which paths mediate input-output behavior using metrics like average unexplained effect.
• The approach refines explanations iteratively, revealing complex behaviors such as induction heads and parroting in transformer models like GPT-2.

Searching arXiv for the specified paper and closely related context. Two-stage path patching is an intervention-based methodology for localizing neural-network behavior to a subset of interactions in a computational graph. In "Localizing Model Behavior with Path Patching" (Goldowsky-Dill et al., 2023), the central claim is that many mechanistic-interpretability hypotheses are better expressed as statements about paths than about isolated nodes: a hypothesis asserts that a subset of paths mediates the relation between input and output on a given dataset, while all other paths are unimportant. The "two-stage" characterization refers to a two-step localization procedure: first specifying a path-based hypothesis, and then testing that hypothesis by patching in counterfactual activations along the paths deemed unimportant. The method is framed as a quantitative criterion for localization, rather than a full computational explanation, and is used to refine an account of induction heads, characterize a behavior of GPT-2, and support iterative mechanistic analysis (Goldowsky-Dill et al., 2023).

1. Localization as a Path-Based Hypothesis

The motivating problem is localizing model behavior: identifying which parts of a network matter for a behavior, or more specifically which interactions between parts matter. The paper emphasizes that many mechanistic claims take forms such as "these neurons/heads/layers are important" or "this circuit explains behavior $X$," but are often qualitative, hard to compare, and not clearly tied to a formal criterion (Goldowsky-Dill et al., 2023).

The key formal move is to define a hypothesis not merely as a set of important nodes, but as a set of important paths in a directed acyclic computational graph $\mathcal{G}$. This is more expressive because many behaviors depend on interactions between components rather than on isolated components. The paper explicitly frames the relevant hypothesis class as a claim that a subset of paths mediate the relation between input and output on a given dataset; all other paths are unimportant (Goldowsky-Dill et al., 2023).

This notion is deliberately narrower than a complete mechanistic explanation. The question is not what each component computes, but which components or interactions are necessary to preserve the behavior. The paper describes this as a notion of approximate abstraction or sufficiency, not completeness (Goldowsky-Dill et al., 2023). A plausible implication is that two-stage path patching is best understood as an evaluative framework for mechanistic hypotheses rather than a theory of representation.

2. Formal Framework and the Two-Stage Procedure

The model is written as a function $G : X \to Y$ with input $x \in X$ and output $y \in Y$, and its computation is represented as a DAG $\mathcal{G}$ (Goldowsky-Dill et al., 2023). The paper first introduces a node-based counterfactual intervention and then generalizes to paths.

A hypothesis is formalized as

$H = (\mathcal{G}, \delta, S, D)$

in the node-based setting, where $\mathcal{G}$ is the computational graph, $\delta : Y^2 \to \mathbb{R}$ is a dissimilarity measure between outputs, $S$ is a set of important nodes, and $\mathcal{G}$0 is a joint distribution over $\mathcal{G}$1 pairs (Goldowsky-Dill et al., 2023). The patched function $\mathcal{G}$2 is evaluated on a reference input $\mathcal{G}$3 while each unimportant node is replaced with the value that node would have on a counterfactual input $\mathcal{G}$4.

The main quantitative criterion is the average unexplained effect:

$\mathcal{G}$5

A good hypothesis has low AUE; $\mathcal{G}$6 means the hypothesis perfectly preserves behavior on the tested distribution, while larger AUE indicates that the hypothesis misses more of the behavior (Goldowsky-Dill et al., 2023).

Path patching generalizes this setup by replacing the set of important nodes with a set of important paths:

$\mathcal{G}$7

The two-stage methodology is then explicit:

1. Specify a path hypothesis: which paths are important.
2. Patch the graph accordingly by substituting $\mathcal{G}$8 versus $\mathcal{G}$9 along those paths and measuring the resulting discrepancy.

This two-step structure is the defining operational form of two-stage path patching (Goldowsky-Dill et al., 2023).

3. Treeification and Path-Specific Intervention

To patch paths independently, the paper introduces $G : X \to Y$0. The intuition is that if a node’s output feeds multiple consumers, then one must copy that subtree so each consumer gets its own independent copy; otherwise, patching one path would affect others (Goldowsky-Dill et al., 2023). The treeified graph $G : X \to Y$1 satisfies

$G : X \to Y$2

The paper illustrates this by fully expanding the computation and relabeling each occurrence of $G : X \to Y$3 with a distinct subscript:

$G : X \to Y$4

In this representation, each path corresponds to one of the inputs $G : X \to Y$5 (Goldowsky-Dill et al., 2023).

Once treeified, the path-patched counterfactual is defined by

$G : X \to Y$6

with

$G : X \to Y$7

where $G : X \to Y$8 are the paths in the treeified graph (Goldowsky-Dill et al., 2023). Important paths are therefore fed the reference input, while unimportant paths are fed the counterfactual input.

The paper notes that this construction is a deterministic analogue of Pearl’s path-specific effect in causal mediation: the reference input corresponds to the reference setting, the counterfactual input to the intervention setting, selected paths remain at the reference value, and unselected paths are intervened on (Goldowsky-Dill et al., 2023). This suggests that two-stage path patching occupies a middle ground between qualitative circuit descriptions and fully general causal mediation analyses.

4. Evaluation Metrics and Attribution

Beyond AUE, the paper defines the average total effect:

$G : X \to Y$9

It then reports proportion explained:

$x \in X$0

The stated interpretation is that $x \in X$1 means the hypothesis exactly recovers the original behavior, $x \in X$2 means it performs no better than the trivial hypothesis where nothing is explained, and the quantity can be negative if $x \in X$3 (Goldowsky-Dill et al., 2023).

The paper also discusses a loss-based formulation in which the loss is included in the computational graph. If labels $x \in X$4 are available and $x \in X$5 is a loss, then the intervention can be evaluated by a loss difference, with the label always kept important and $x \in X$6 taken as absolute value. This focuses the evaluation on whether the explanation preserves predictive performance rather than the exact output distribution. The paper warns, however, that this can be misleading if the model is already very confident, because the denominator $x \in X$7 can be large even for weak hypotheses (Goldowsky-Dill et al., 2023).

A further alternative is the difference in expected loss:

$x \in X$8

This can be useful when one wants the explanation to ignore factors that cancel out in expectation across the dataset, but the paper warns that it can be misleading due to cancellation: some examples get better, others get worse, and the average may look small even if the hypothesis misses important structure (Goldowsky-Dill et al., 2023).

For single examples, the paper defines path patching attribution as

$x \in X$9

Here the absolute value is dropped so that one can see whether patching increases or decreases loss. This is presented as useful for identifying which paths are responsible for a particular failure mode or example (Goldowsky-Dill et al., 2023).

5. Iterative Refinement and Efficiency

The practical workflow is presented as an iterative refinement loop:

1. Start with a sparse path hypothesis based on inspection.
2. Test it with path patching using a reference/counterfactual dataset.
3. If AUE is high, use path patching attribution to find what was missed.
4. Refine the hypothesis by adding additional paths.
5. Repeat until the explanation is sufficiently faithful.

This iterative structure is described as central to the paper and especially visible in the induction-head study (Goldowsky-Dill et al., 2023).

A major implementation detail is that the same counterfactual input $y \in Y$0 is reused across all unimportant paths. The paper gives four reasons for this design: it preserves unimportance under rewrites, matches ensemble-like residual behavior, allows cancellation, and improves computational efficiency because shared subexpressions can be cached (Goldowsky-Dill et al., 2023). The efficiency point matters because, in residual networks, the number of paths can grow exponentially with depth.

The paper also notes that path patching is more efficient than causal scrubbing because it typically needs only two samples per intervention, $y \in Y$1 and $y \in Y$2 (Goldowsky-Dill et al., 2023). This suggests that the two-stage formulation is not only conceptually precise but also designed to keep the intervention budget tractable in large residual architectures.

6. Empirical Studies: Induction Heads and GPT-2 Number Completion

The main empirical case study concerns the induction-head phenomenon in a 2-layer attention-only transformer trained on OpenWebText, with 8 attention heads per layer, layer norm before attention heads, shortformer positional embeddings, and separate unembedding (Goldowsky-Dill et al., 2023). The paper identifies head 0.0 as a previous-token head and heads 1.5 and 1.6 as induction heads. The dataset $y \in Y$3 is the first 300 tokens of 100K held-out OpenWebText examples, and the graph includes cross-entropy loss at each token, so the target behavior is the probability of the true next token (Goldowsky-Dill et al., 2023).

The paper evaluates several initial path hypotheses:

Hypothesis	Proportion explained
Direct-V	64.1%
Direct-Q	48.7%
PTH-K	53.3%
All-Initial	28.2%

These results show that the naive minimal circuit is far from sufficient (Goldowsky-Dill et al., 2023). The authors then add positional pathways through layer-0 heads into the induction heads:

Hypothesis	Proportion explained
Positional-V	86.2%
Positional-Q	72.6%
Positional-K	55.8%
All-Positional	48.6%

This demonstrates that much of the missing behavior is mediated by early heads processing position-dependent information (Goldowsky-Dill et al., 2023).

A second refinement adds longer-context query and key paths from a window of recent tokens, with $y \in Y$4 found to be the shortest effective window. This yields Long-QK and All-Long, with results of 59.5% and 55.2% respectively (Goldowsky-Dill et al., 2023). The remaining unexplained effect is then analyzed with path patching attribution, which reveals that head 1.5 exhibits a distinct parroting heuristic in addition to induction: tokens that have appeared before are treated as evidence they should repeat, even outside strict prefix-matching induction cases. After adding the extra paths needed to explain this behavior, the final hypothesis All-Final reaches 73.5% explained (Goldowsky-Dill et al., 2023).

The second case study concerns GPT-2 small and GPT-2 XL on prompts of the form "The organization estimates that [N]-" with $y \in Y$5 ranging from 0 to 100. The path patching experiments test each attention head by making all paths through that head unimportant and measuring mean KL divergence. The reported finding is that path patching identifies fewer heads than causal tracing or input corruption, because its counterfactual is more on-distribution: the counterfactual number is still a number, so activations remain closer to normal model states (Goldowsky-Dill et al., 2023). The paper also compares path patching to causal tracing as a special case with Gaussian-noised counterfactual input and to zero ablation as setting components to zero. A concrete subanalysis suggests that heads 5.6 and 6.1 help recognize that the token before the hyphen is a number: corrupting these heads lowers the probability of numeric completion, while patching them does not (Goldowsky-Dill et al., 2023).

The paper briefly explores greedy hypothesis building by adding heads in descending order of mean KL divergence. For GPT-2 small, the hypothesis that all heads are unimportant except 9.1 and 8.8 recovers 72.8%, and including only 8 of 144 heads as important recovers 98.0% (Goldowsky-Dill et al., 2023). This suggests that path patching can be paired with automated search, although the paper does not claim that the greedy baseline is optimal.

7. Relation to Prior Methods, Scope, and Limitations

The paper argues that path patching has several advantages over older mechanistic-interpretability techniques. Compared with causal tracing or input corruption, it is described as more precise, able to isolate specific interactions rather than whole nodes, more on-distribution, and often capable of yielding a sparser explanation. Compared with zero ablation, it can ask which paths through a head matter, rather than whether the head matters at all. Compared with causal scrubbing, it is presented as a simpler special case that generally needs only two samples per intervention and avoids claims about node equivalence classes (Goldowsky-Dill et al., 2023).

Relative to prior induction-head explanations, the method is used to quantify how much of the behavior is explained by the minimal circuit, identify missing pathways, and discover that head 1.5 also implements parroting (Goldowsky-Dill et al., 2023). This is one of the paper’s main scientific outcomes: what initially appears to be a single induction circuit can combine induction with a separate repeating-copying heuristic.

The paper is also careful about limitations. A low AUE means that the hypothesis is sufficient to mediate the behavior on the tested distribution; it does not prove that the hypothesis is the full mechanism. Claims are distribution-dependent, so a behavior that is unimportant on one dataset may matter on another. Some metrics, especially difference-in-expected-loss, can hide errors because positive and negative deviations cancel. Finite-sample estimation may miss rare but important behaviors. Path patching provides evidence, not a mathematical proof that a hypothesis is correct, and the demonstrations are explicitly limited, with more work needed on larger and more diverse models (Goldowsky-Dill et al., 2023).

Taken together, these points place two-stage path patching within mechanistic interpretability as a formal intervention-based method for testing localization claims about neural networks. Its defining contribution is the conversion of an informal circuit hypothesis into a path-based sufficiency claim that can be quantitatively evaluated, refined, and compared across candidate explanations (Goldowsky-Dill et al., 2023).