Attention Rollout in Transformers

• Attention Rollout is a post hoc method that recursively multiplies and aggregates attention matrices to estimate the contribution of each input token in Transformer models.
• It addresses interpretability issues by incorporating residual connections, ensuring that token importance is more reliably measured than by raw attention weights.
• The method improves alignment with ablation and gradient-based evaluations, making it a valuable tool for debugging and analyzing Transformer behavior.

Searching arXiv for the primary paper and closely related work on Attention Rollout. Attention Rollout (AR) is a post hoc method for approximating attention to input tokens in Transformer models from attention weights. Introduced by Abnar and Zuidema in “Quantifying Attention Flow in Transformers,” it addresses a central interpretability problem: self-attention mixes information across layers, so raw attention weights become unreliable as explanation probes for input token importance. AR quantifies the flow of information through self-attention by recursively multiplying and aggregating attention matrices across layers while incorporating residual connections, thereby estimating how final-layer representations depend on original input tokens (Abnar et al., 2020).

1. Motivation and interpretive setting

In a Transformer, self-attention combines information from attended embeddings into the representation of the focal embedding in the next layer. Across layers, information originating from different tokens gets increasingly mixed. This mixing makes raw attention weights unreliable as explanations probes, particularly in higher layers, where visualizations and analyses reveal that raw attention becomes nearly uniform and therefore obscures meaningful interpretability (Abnar et al., 2020).

The objective of AR is to track how information from each input token propagates through layers via attention. In this framing, the target is not the immediate attention distribution of a single layer, but the accumulated contribution of input tokens to a representation after repeated attention-mediated mixing. This makes AR a method for estimating input token importance under the assumption that attention weights encode relative relevance.

A key implication is that AR is concerned with token identity recovery after deep mixing. Rather than interpreting a final-layer attention map in isolation, it reconstructs how earlier tokens continue to influence later states through the entire stack. This is why the method is presented as a way to obtain a sounder measure of input token importance in model predictions.

2. Mathematical construction

The defining operation in AR is recursive aggregation of attention across layers, with residual connections explicitly included. The paper describes the layer update as

$$V_{l+1} = V_l + W_{\text{att}} V_l = (W_{\text{att}} + I) V_l,$$

so the effective attention matrix, after accounting for residual connections and renormalization, becomes

$A = 0.5 W_{\text{att}} + 0.5 I.$

This residual-aware formulation is central. Residual connections are fundamental in Transformers, and ignoring them leads to misattribution of importance. In AR, the identity contribution preserves the possibility that information remains at the same token position while attention redistributes part of it elsewhere.

To trace attention from an upper layer $l_i$ back to a lower layer $l_j$ with $j < i$, AR recursively multiplies the augmented attention matrices:

$$\tilde{A}(l_i) = \begin{cases} A(l_i)\cdot \tilde{A}(l_{i-1}) & \text{if } i > j \ A(l_i) & \text{if } i = j \end{cases}$$

When the goal is input token importance, one sets $j = 0$, so the overall rollout across $L$ layers is

$\tilde{A} = A(L)\cdot A(L-1)\cdots A(1).$

Operationally, AR multiplies and aggregates attention matrices across all layers to estimate the contribution of input tokens to final-layer representations. Its reported complexity is $O(d n^2)$, where $A = 0.5 W_{\text{att}} + 0.5 I.$0 is depth and $A = 0.5 W_{\text{att}} + 0.5 I.$1 is the number of tokens. The method is described as applicable for any layer, any token, and for single-head or multi-head analyses.

3. Relation to raw attention and attention flow

AR was introduced alongside a second method, Attention Flow. Both are approximations to attention to input tokens given attention weights, and the two methods are described as giving complementary views on the flow of information (Abnar et al., 2020).

Raw attention only considers attention weights from the immediate previous layer. It therefore does not track the cumulated propagation of information through earlier layers, and it fails to capture true input token contributions at higher layers. By contrast, AR tracks information flow recursively through the whole stack.

The distinction between AR and Attention Flow is methodological. AR multiplies proportions weighted by all possible paths, whereas Attention Flow treats the attention network as a flow network and measures maximal non-overlapping flows as capacities. In the reported qualitative characterization, AR yields more focused attention patterns and provides a sharper, more discriminative attribution than raw attention.

These differences matter interpretively. AR recovers a path-integrated notion of attribution, while Attention Flow emphasizes capacity constraints in the network of attentional interactions. The paper’s framing is not that one subsumes the other, but that they expose different aspects of information propagation through layered self-attention.

4. Empirical behavior and validation

The quantitative evaluation compares raw attention, AR, and Attention Flow using Spearman’s rank correlation between attention-based scores and two reference metrics for token importance: blank-out or ablation, and input gradients. The central empirical result is that AR and Attention Flow both yield higher and more stable correlations than raw attention, especially at higher layers (Abnar et al., 2020).

Method	Correlation (Layer 6)	Reference
Raw Attention	0.29 ± 0.39	Blank-out
Rollout	0.71 ± 0.24	Blank-out
Attention Flow	0.70 ± 0.22	Blank-out

The same evaluation framework reports that raw attention correlations are near zero or negative in upper layers, whereas AR and Attention Flow remain substantially better aligned with ablation- and gradient-based importance estimates. This supports the claim that accumulated information flow is a more faithful interpretive object than layer-local attention alone.

Qualitatively, the paper reports that AR produces attention heatmaps that more plausibly focus on important tokens, including the subject noun in subject-verb agreement and pronoun antecedents in BERT. The qualitative error analysis further states that AR can indicate sources of model errors by highlighting distractor tokens in misclassified cases. This positions AR not only as an attribution method but also as a debugging aid for Transformer behavior.

5. Extensions to multi-head and vision-transformer interpretability

Subsequent work has adapted AR to settings where the equal treatment of attention heads is considered a limitation. “GMAR: Gradient-Driven Multi-Head Attention Rollout for Vision Transformer Interpretability” argues that not all attention heads are equally meaningful and introduces Gradient-Driven Multi-Head Attention Rollout (GMAR), which quantifies head importance using class-specific gradients and uses the resulting normalized weights to form a weighted aggregate attention score (Jo et al., 28 Apr 2025).

In GMAR, a gradient-based score $A = 0.5 W_{\text{att}} + 0.5 I.$2 for each head is computed using either an $A = 0.5 W_{\text{att}} + 0.5 I.$3 or $A = 0.5 W_{\text{att}} + 0.5 I.$4 norm,

$A = 0.5 W_{\text{att}} + 0.5 I.$5

and then normalized as

$A = 0.5 W_{\text{att}} + 0.5 I.$6

The rollout is then performed with head-weighted attention and a residual term:

$A = 0.5 W_{\text{att}} + 0.5 I.$7

The reported empirical comparison in ViTs shows GMAR outperforming traditional Attention Rollout on Average Drop, Average Increase, Insertion, and Deletion. For example, Attention Rollout is reported at 25.78 for Average Drop, 46.2 for Average Increase, 11.97 for Insertion, and 12.17 for Deletion, while GMAR with $A = 0.5 W_{\text{att}} + 0.5 I.$8 normalization is reported at 22.13, 55.9, 12.16, and 10.64, respectively (Jo et al., 28 Apr 2025). This suggests that AR became a baseline for later, head-sensitive variants rather than a fixed endpoint in Transformer interpretability.

6. Limits, assumptions, and terminological ambiguity

AR and Attention Flow are both described as making simplifying assumptions about how attention encodes information flow. They are therefore not presented as perfect accounts of causal contribution, even though they improve substantially over raw attention (Abnar et al., 2020). The paper’s own framing is comparative and post hoc: AR is an approximation that better tracks input token relevance under repeated mixing, not a proof that attention exactly equals explanation.

A second limitation concerns architectural dependence. Residual connections are crucial in the method, and naive attention analyses that ignore them lead to misattribution of importance. This is not a secondary implementation detail; it is part of the mathematical basis of the rollout calculation.

The abbreviation “AR” is also ambiguous across recent literature. In HorizonDrive, “AR rollout” refers to autoregressive rollout in long-horizon driving simulation, where a world model generates short video chunks step-wise and feeds its own predictions back as context (Zhang et al., 12 May 2026). In RolloutPipe, “rollout” refers to rollout generation in disaggregated on-policy LLM reinforcement learning, not interpretability (Chen et al., 25 Jun 2026). These usages are unrelated to Attention Rollout, even though they share the same abbreviation.

Within Transformer interpretability, however, Attention Rollout remains a specific method: residual-aware recursive multiplication of attention matrices to estimate how information from input tokens contributes to later representations. Its significance lies in making the notion of “attention to the input” explicit in deep self-attentive stacks, where raw attention alone is insufficient.