Attention Knockout in Sequence Models

• Attention Knockout is a causal interpretability method that tests whether removing specific token-to-token connections in sequence models affects prediction accuracy.
• It intervenes by zeroing selected entries in the attention matrix and extends this framework to state-space models like Mamba, highlighting its versatility across architectures.
• Empirical findings reveal both universal subject-to-target pathways and architecture-specific behaviors, informing practical applications such as model debugging and feature pruning.

Searching arXiv for the original Attention Knockout paper and related work. arXiv search query: all:"Attention Knockout" Attention Knockout is a causal interpretability technique for sequence models that tests whether information about a source token flows to a target token through specific internal token-to-token connections in a way that matters for a prediction. In its original Transformer formulation, the method zeroes selected entries of an attention matrix and recomputes the forward pass; the effect on the probability of the correct next token is then treated as an importance measure for that edge in the internal computation graph. More recent work extends the same intervention logic to selective state-space models such as Mamba by identifying attention-like kernels or linear-attention matrices inside the model and performing the same fine-grained knockout there (Endy et al., 30 May 2025). In a broader, analogical sense, related work has used “knockout” interventions on gene-presence representations and on orbital components in nuclear reactions to probe which inputs or components are functionally decisive, although those settings do not implement Transformer attention itself (He et al., 3 Oct 2025, Edagawa et al., 2022).

1. Definition and conceptual scope

Attention Knockout, as described for Transformers, asks a specific causal question: whether information about token $r$ actually flows to token $c$ at layer $\ell$ in a way that matters for the prediction. The canonical intervention is to take a factual prompt such as “Beats Music is owned by __”, remove selected attention links from the subject token to the final input token, and measure whether the probability of the correct continuation decreases. If severing that connection substantially lowers the probability of the correct next token, the link is treated as a critical information pathway (Endy et al., 30 May 2025).

The method differs from descriptive analyses of raw attention weights. Its central object is not the magnitude of an attention coefficient alone, but the consequence of deleting that coefficient from the computation and rerunning the model. This makes the technique causal in the interventionist sense adopted by the cited work: the edge is important if prediction quality changes when that edge is removed (Endy et al., 30 May 2025).

A common misconception is that Attention Knockout explains what information is represented in a model’s states. The cited work is explicit that the method identifies where information flows, not what information is represented in the states. It is therefore best understood as a circuit-localization tool rather than a semantic decoder (Endy et al., 30 May 2025).

2. Formal intervention in Transformers and Mamba

In a standard attention layer with attention matrix $A^{(\ell)} \in \mathbb{R}^{N \times N}$, Attention Knockout zeroes out specific entries $A^{(\ell)}_{c,r}$, possibly across a range of layers, and recomputes the forward pass. The baseline pass produces the probability $p_{\text{base}}$ of the correct next token, while the intervened pass produces $p_{\text{ko}}$. The effect is reported as the relative change

$$\Delta = \frac{p_{\text{ko}} - p_{\text{base}}}{p_{\text{base}}}.$$

Negative values indicate that removing the connection harms the model’s factual recall, whereas positive values indicate that the intervention improves it (Endy et al., 30 May 2025).

The same paper adapts this procedure to two Mamba variants by exploiting theoretical results that selective SSMs can be written in an attention-like form. In Mamba-1, the kernel connecting positions $i$ and $j$ is written as

$c$0

where $c$1, $c$2, and $c$3 encodes state propagation. Knockout is then defined by setting $c$4 for the selected source-target pair in a chosen layer window (Endy et al., 30 May 2025).

In Mamba-2, the SSM layer is expressed as

$c$5

with $c$6 serving as the attention matrix. The intervention forms this matrix explicitly, sets the selected entry $c$7, and recomputes the layer output (Endy et al., 30 May 2025).

Architecture	Knockout object	Intervention
Transformer	$c$8	Set selected attention entries to $c$9
Mamba-1	$\ell$0	Set selected kernel entries to $\ell$1
Mamba-2	$\ell$2 from $\ell$3	Set selected matrix entries to $\ell$4

This formalization preserves the core idea across architectures: token-to-token connectivity is treated as a causal edge, and its importance is estimated by cutting the edge and measuring the prediction change (Endy et al., 30 May 2025).

3. Experimental protocol and measurement

The cited Mamba study operationalizes Attention Knockout on factual recall using COUNTERFACT prompts built from subject-relation-attribute triples. It restricts analysis to a subset of 672 triplets for which all examined models produce the correct next token in the baseline condition, so that performance changes can be attributed to the intervention rather than to baseline model failure (Endy et al., 30 May 2025).

The procedure analyzes connections from four token classes to the last token: subject token(s), relation token(s), the first token in the sequence, and the last token itself. Rather than intervening in a single layer only, the experiments define a contiguous window of layers with size $\ell$5 and slide that window through depth. Reported window sizes include $\ell$6, and plots index the relative depth of the first layer within the window (Endy et al., 30 May 2025).

The primary metric is the mean relative change in correct-token probability under the knockout intervention. The work also reports scatterplots of $\ell$7 versus $\ell$8 and example-level heatmaps in which rows correspond to source tokens and columns correspond to window start layers. These visualizations support statements about which token-to-token routes matter and at what depths they matter (Endy et al., 30 May 2025).

An important methodological limitation is that knockout is applied in contiguous layer windows. The authors note that this may miss more distributed, non-contiguous circuits of information flow. They also note that the intervention is artificial, so any change might in principle reflect general disruption rather than targeted removal, although the localized patterns they observe are taken to suggest specific information removal (Endy et al., 30 May 2025).

4. Empirical patterns in factual information flow

Across Mamba-1, Mamba-2, Falcon-Mamba, GPT-2, LLaMA, and Mistral, knocking out subject-to-last-token connections in late-intermediate layers reliably reduces correct-token probability. The paper presents this as a cross-architecture regularity and interprets it as evidence that factual recall depends on routing subject information to the prediction position in late-intermediate depth (Endy et al., 30 May 2025).

The same experiments also identify architecture-specific differences. Transformer models such as GPT-2, LLaMA, and Mistral show strong first-token influence in later layers, described as a first-position bias or attention sink. Mamba-1 and Mamba-2 do not show the same strong first-position bias, which the paper links to the absence of softmax-normalization mechanisms in SSMs (Endy et al., 30 May 2025).

Relation-token effects also differ by architecture. GPT-2 shows strong dependence on relation tokens in early layers, decreasing later. Mamba-1 and Falcon-Mamba instead show relation-token importance primarily in later layers, often with a pattern in which knockout first decreases correctness and then sharply increases it. Mamba-2 likewise places primary relation-token importance in later layers, but without the strong last-token phenomenon characteristic of Mamba-1 (Endy et al., 30 May 2025).

The most striking architecture-specific result is Mamba-1’s last-token self-connection behavior. Removing last-token-to-itself connections in the final layers causes a dramatic increase in correct-token probability, often up to nearly 1. The paper interprets this as evidence that Mamba-1’s final-layer self-connection may contribute noise or conflicting information, such that disabling it forces reliance on earlier tokens and improves factual recall (Endy et al., 30 May 2025).

These findings motivate a distinction between candidate universal behaviors and architecture-specific behaviors. Subject-token importance and its late-intermediate routing to the last token are presented as universal candidates, whereas first-token bias, relation-token dynamics, and the Mamba-1 self-connection effect are presented as architecture-specific (Endy et al., 30 May 2025).

5. Feature knockout and structured factorization in Mamba

The Mamba analysis introduces a second level of intervention beyond token-to-token edges: feature-level knockout. This relies on the shared parameterization

$\ell$9

which implies that over a token span,

$A^{(\ell)} \in \mathbb{R}^{N \times N}$0

Because $A^{(\ell)} \in \mathbb{R}^{N \times N}$1 is diagonal, each SSM feature behaves as an independent one-dimensional state with its own decay profile (Endy et al., 30 May 2025).

The paper classifies features by the $A^{(\ell)} \in \mathbb{R}^{N \times N}$2-norm of $A^{(\ell)} \in \mathbb{R}^{N \times N}$3. Context-dependent features are defined as the top third by $A^{(\ell)} \in \mathbb{R}^{N \times N}$4, while context-independent features are defined as the bottom third. The intuition given is that features with $A^{(\ell)} \in \mathbb{R}^{N \times N}$5 retain history and carry long-range context, whereas features with $A^{(\ell)} \in \mathbb{R}^{N \times N}$6 decay rapidly and therefore primarily process local token information (Endy et al., 30 May 2025).

Feature knockout is implemented by zeroing the outputs of selected features, thereby removing their contribution to the SSM layer’s output. When the authors restrict subject-to-last-token knockout to context-dependent features, the effect is almost the same as knocking out all features: correct-token probability drops strongly in late-intermediate layers. By contrast, knocking out context-independent features has minimal effect and yields curves near zero (Endy et al., 30 May 2025).

This leads to a specific functional interpretation: token-to-token factual information exchange is carried almost entirely by context-dependent features, while context-independent features mostly enrich individual tokens and contribute much less to cross-token routing. The paper presents this decay-based factorization as a unified lens analogous to the functional specialization of attention heads in Transformers (Endy et al., 30 May 2025).

6. Extensions, analogies, and broader interpretive use

Attention Knockout is explicitly defined for attention or attention-like token-to-token connectivity, but related work uses analogous knockout logic in other domains. In the deep-learning framework for single-cell gene knockout prediction, a model learns a mapping from a cell’s binary gene assemblage to its full expression profile and then performs virtual knockout by setting a gene’s assemblage entry to zero and recomputing the prediction. That paper states that, although it does not implement an explicit attention layer, the encoding of gene assemblages and propagation of their influence to all other genes plays an attention-like role, effectively focusing on each gene’s presence or absence to infer its cell-specific impact (He et al., 3 Oct 2025).

In that setting, the per-gene knockout impact score functions as a functional importance measure. The paper states that one can treat per-gene KO impact scores and gene-wise ranks as analogous to attention weights highlighting which genes matter most for the transcriptional state. This is not Attention Knockout in the Transformer sense, but it is an intervention-based importance analysis grounded in deletion of an input component and measurement of downstream effect (He et al., 3 Oct 2025).

A second analogical extension appears in proton-induced $A^{(\ell)} \in \mathbb{R}^{N \times N}$7-knockout reactions. There, the phrase “pay attention” is used figuratively to describe reaction-position selectivity: the reaction selectively samples a particular orbital component of an $A^{(\ell)} \in \mathbb{R}^{N \times N}$8 cluster, producing effective polarization of the residual nucleus. The mechanism is not based on computational attention or token-to-token matrices; rather, it is a physically distinct selection effect driven by geometry and strong absorption of the emitted $A^{(\ell)} \in \mathbb{R}^{N \times N}$9 particle (Edagawa et al., 2022).

These extensions suggest a broader editorial usage of the term. In strict machine-learning usage, Attention Knockout refers to causal deletion of attention or attention-like edges in sequence models. In a looser analogical usage, it may denote intervention-based removal or selective suppression of a component whose effect is then quantified. A plausible implication is that the term’s conceptual center is not “attention” in the narrow architectural sense alone, but causal testing of routed influence under targeted ablation (Endy et al., 30 May 2025, He et al., 3 Oct 2025, Edagawa et al., 2022).

7. Limitations, applications, and future directions

The cited Mamba work frames Attention Knockout as a practical interpretability toolkit with token-level, layer-level, and feature-level diagnostics. At the token level, it identifies which source tokens are critical for factual prediction. At the layer level, it localizes the depth range in which source-to-target communication becomes indispensable. At the feature level, it distinguishes subsets of SSM features that carry cross-token context from those associated with local processing (Endy et al., 30 May 2025).

The same work outlines several practical uses. Because context-independent features contribute little to cross-token information flow, they are proposed as candidates for pruning, analogously to redundant attention heads. The authors also suggest fine-tuning only context-dependent features to alter factual behavior or adapt the model, and they describe knockout-guided model editing and debugging as a way to locate token interactions that drive a factual prediction or a hallucination (Endy et al., 30 May 2025).

The limitations are equally clear. The method focuses on between-token pathways rather than token-independent components such as gating or convolution. It does not explain why some subject-token patterns appear universal across architectures. It also does not, by itself, decode the content of the critical features or states. Future work is described in terms of more fine-grained or learning-based interventions, decoding the content of critical features, and understanding the inductive biases behind both universal and architecture-specific behaviors (Endy et al., 30 May 2025).

Within the broader intervention-based landscape, Attention Knockout therefore occupies a specific methodological niche. It is neither a mechanistic proof of full causal structure nor a semantic explanation of internal representations. Its distinctive contribution is to transform internal connectivity into experimentally testable edges and to estimate, by deletion, which of those edges are indispensable for a given prediction (Endy et al., 30 May 2025).