Dual Path Attribution in SwiGLU Transformers

• Dual Path Attribution (DPA) is a transformer interpretability framework that efficiently attributes components via backward target propagation.
• It decomposes transformer computations into content and control pathways, accurately capturing both direct and indirect contributions.
• DPA achieves O(1) efficiency in component attribution by avoiding expensive counterfactual forward passes, making it suitable for dense attribution tasks.

Searching arXiv for the named method and closely related path-attribution work to ground the article in current papers. Dual Path Attribution (DPA) is a mechanistic-interpretability attribution framework for transformer-based LLMs that was introduced for dense attribution in SwiGLU transformers through layer-wise target propagation. Rather than estimating each component’s effect by repeatedly forwarding counterfactual activations, DPA propagates a targeted unembedding vector backward through a frozen transformer, computes an effective target at each residual position, and then scores tokens, attention heads, and MLP neurons by dot products with cached activations or local component outputs. In the formulation of "Dual Path Attribution: Efficient Attribution for SwiGLU-Transformers through Layer-Wise Target Propagation," the method is designed to trace information flow in one forward and one backward-style pass, without requiring counterfactual examples, while remaining $O(1)$ with respect to the number of attributed components $M$ (Jantsch et al., 20 Mar 2026).

1. Terminological scope and problem setting

In the attribution literature, the term Dual Path Attribution refers specifically to the transformer-attribution method above, not to other unrelated uses of the acronym DPA. The same acronym is used for Dual Power Assignment in network design and Dual Probabilistic Alignment in universal domain adaptive object detection; those topics are unrelated to transformer attribution (Grimmer, 2015, Zheng et al., 2024).

DPA was proposed against a familiar tradeoff in large-model interpretability. Direct methods such as Direct Logit Attribution capture only direct contribution to the final residual stream and miss indirect downstream effects, especially for early-layer components. Intervention methods such as activation patching are described as faithful but prohibitively expensive for dense component attribution because each component or subset typically requires a fresh counterfactual forward pass. Gradient methods and integrated gradients are efficient but may be noisy, saturated, or insufficiently structured for tracing transformer-specific routing dynamics. Prior decomposition methods such as DePass move toward structured decomposition but still scale linearly in the number of attributed components $M$ because they carry many separate contribution vectors. DPA addresses this by reversing the attribution perspective: instead of forwarding each candidate component through the rest of the model, it propagates the target-logit direction backward and converts attribution into local dot products (Jantsch et al., 20 Mar 2026).

2. Transformer model and target-centric reformulation

DPA is developed for a pre-norm decoder-only transformer, essentially LLaMA-style, with $L$ layers, sequence length $T$, and residual width $d$. The residual stream is written as

$$\mathbf{X}^{(l)}_{\text{mid}} = \mathbf{X}^{(l-1)} + \mathrm{MHSA}\!\left(\mathrm{LN}(\mathbf{X}^{(l-1)})\right),$$

$$\mathbf{X}^{(l)} = \mathbf{X}^{(l)}_{\text{mid}} + \mathrm{GLU}\!\left(\mathrm{LN}(\mathbf{X}^{(l)}_{\text{mid}})\right).$$

The architecture uses RMSNorm, RoPE in attention, and a SwiGLU MLP. A “frozen transformer” means that for a given prompt the forward activations are fixed and no counterfactual recomputation is done during attribution; the linearization is taken around those fixed activations (Jantsch et al., 20 Mar 2026).

The target of explanation is a token logit. If $t^{(j)}\in\mathcal V$ is the target token and $\mathbf W_{UE}$ is the unembedding matrix, then $M$0 is the latent direction that increases that token’s logit. For an embedding vector $M$1 at position $M$2, direct logit attribution is

$M$3

but that omits effects mediated by downstream layers. DPA re-expresses the full attribution by inverting the viewpoint: $M$4 where $M$5 is the effective target propagated back to layer $M$6. A plausible implication is that DPA turns a global causal question—how much a component mattered after all downstream computation—into a local alignment question at each residual insertion point.

3. Dual-path decomposition of SwiGLU and attention

The term “dual path” in DPA refers to an analytical decomposition of transformer computation into content and control pathways. In MLP blocks, content flows through the up/down projections and control through the gate; in attention, content flows through values and control through queries and keys. DPA exploits the fact that both MHSA and SwiGLU are bilinear or nearly bilinear if one pathway is frozen, and then propagates the target through those decomposed pathways (Jantsch et al., 20 Mar 2026).

For the feed-forward block, the paper writes

$M$7

For neuron $M$8,

$M$9

DPA splits this into an up/content path and a gate/control path,

$M$0

Freezing the gate activation $M$1 gives an up-path target

$M$2

while the gate path freezes the content term and locally linearizes the nonlinearity, yielding the corresponding control-path target. The paper notes that the appendix retains the content term $M$3 explicitly in the gate-path formula (Jantsch et al., 20 Mar 2026).

For attention, each head $M$4 at query position $M$5 outputs

$M$6

with

$M$7

DPA decomposes inverse propagation through a head into value/content and query/key control paths: $M$8 If attention weights are frozen, the value path is linear. Query and key paths are treated with an AttnLRP-style Taylor decomposition rather than naive local linearization because softmax outputs remain nonzero even for zero inputs and naive linearization is unstable. The weighting parameters $M$9 regularize inverse propagation and prevent target norm explosion. The same section states that although DPA generally supports completeness, the choice of softmax linearization breaks completeness “for the sake of a robust gradient” (Jantsch et al., 20 Mar 2026).

4. Layer-wise target propagation and attribution rule

DPA operates in three stages. First, it runs a normal forward pass on the prompt and caches normalized residual inputs before each MHSA and GLU block, attention weights, query/key/value-related states as needed, GLU preactivations, gate activations, and local component outputs when direct scoring is desired. Second, it initializes the final target vector as the unembedding direction of the target token and propagates that target backward layer by layer: through the GLU residual branch using the up and gate targets, and then through each attention head using value, query, and key targets. Third, it scores components by dotting local outputs with the propagated target at the residual point where those outputs enter the stream (Jantsch et al., 20 Mar 2026).

The resulting attribution rules are simple. For token attribution,

$L$0

For attention heads and GLU neurons, the appendix algorithm gives

$L$1

$L$2

Positive scores indicate that a component pushes the residual stream in the direction that increases the target logit; negative scores indicate that it opposes or suppresses the target logit. Magnitude reflects strength of alignment. Because targets exist per residual position, DPA naturally yields token-level and position-specific scores, and because modules are decomposed into heads and neurons it also yields dense component-level attributions (Jantsch et al., 20 Mar 2026).

5. Computational complexity and empirical performance

The efficiency claim of DPA is explicitly with respect to the number of attributed components. After activations are cached, DPA requires one forward pass and one backward-style propagation pass, with no additional forward passes for each component. The paper states the complexity as $L$3 with respect to the number of attributed components $L$4, in contrast to activation patching and DePass, which both scale as $L$5. This is not $L$6 overall runtime independent of model size or sequence length; the method still incurs the ordinary cost of one forward pass plus one backward-style propagation through all layers and cached tensors, and attention retains its usual dependence on sequence length (Jantsch et al., 20 Mar 2026).

The paper evaluates token attribution on Known 1000, SQuAD v2.0, and IMDb, and component attribution on Known 1000 and IOI, using Llama-3.1-8B-Instruct, Llama-2-7B-Chat, Qwen3-4B-Instruct, Mistral-7B-Instruct-v0.3, and Qwen2.5-32B-Instruct. On Llama-3.1-8B-Instruct token attribution, DPA reports the best total score on all three datasets in the main table: Known 1000 total $L$7, versus DePass $L$8 and Integrated Gradients $L$9; SQuAD v2.0 total $T$0, versus Mean Attention $T$1 and Integrated Gradients $T$2; IMDb total $T$3, versus the next best around $T$4 or $T$5. On component attribution for Llama-3.1-8B-Instruct, DPA reports total $T$6 on Known 1000, above AtP $T$7, Attn-only $T$8, and Norm-only $T$9, while on IOI it reports total $d$0, slightly below Attn-only $d$1 in aggregate AUC but with stronger early circuit isolation in the paper’s discussion (Jantsch et al., 20 Mar 2026).

Runtime measurements emphasize long-context efficiency. On Llama-3.1-8B-Instruct, Known 1000 requires $d$2 s for activation patching, $d$3 s for DePass, and $d$4 s for DPA; on SQuAD v2.0 the corresponding times are $d$5 s, $d$6 s, and $d$7 s. The paper characterizes the SQuAD result as roughly $d$8 faster. Peak memory is described as comparable and sometimes slightly higher for DPA, reflecting a time–memory tradeoff due to dense activation caching (Jantsch et al., 20 Mar 2026).

6. Relation to neighboring attribution frameworks and limitations

DPA occupies a distinct position among attribution methods. Compared with plain gradients, it is more structured and model-aware. Compared with integrated gradients, it avoids baseline-choice issues and path-integral cost while still using local linearization ideas. Compared with perturbation or ablation methods such as activation patching, it is dramatically more efficient but only approximate. Compared with DePass, its target-centric backward propagation avoids the $d$9 growth that comes from carrying one contribution vector per component. Compared with Direct Logit Attribution, it captures indirect downstream effects because effective targets are propagated through the remaining computation rather than read only at the output (Jantsch et al., 20 Mar 2026).

The term “path” should be distinguished carefully. In vision-oriented path attribution, a path usually means a trajectory from a baseline to an input, as in Integrated Gradients-style line integrals and later work on baseline validity, path ambiguity, or distributional reveal paths (Akhtar et al., 2023, Zhang et al., 2024, Murphy et al., 2 Jun 2026). DPA uses “path” differently: it decomposes transformer blocks into content and control pathways and propagates a target vector through a frozen network. This suggests that DPA and input-space path attribution are neighboring attribution traditions rather than instances of the same formalism.

The main limitations stated for DPA are architectural dependence, approximation error, target choice, and memory cost. The derivation is explicitly tailored to dense SwiGLU transformers; adapting it to MoE, SSMs, or other non-SwiGLU architectures requires new inverse rules. The method is a first-order approximation that “drops second-order cross-derivatives and truncates higher-order dynamics,” so it works best in locally linear regimes with stable activations and weak synergistic coupling. It attributes the target logit rather than the post-softmax target probability, which can miss mechanisms that help chiefly by suppressing rival logits. Finally, although the method is $$\mathbf{X}^{(l)}_{\text{mid}} = \mathbf{X}^{(l-1)} + \mathrm{MHSA}\!\left(\mathrm{LN}(\mathbf{X}^{(l-1)})\right),$$0 in the number of attributed components, the memory footprint remains substantial because it caches dense intermediate activations (Jantsch et al., 20 Mar 2026).