Decision-Aware Attention Propagation for Vision Transformer Explainability

Abstract: Vision Transformers (ViTs) have become a dominant architecture in computer vision, yet their prediction process remains difficult to interpret because information is propagated through complex interactions across layers and attention heads. Existing attention based explanation methods provide an intuitive way to trace information flow. However, they rely mainly on raw attention weights, which do not explicitly reflect the final decision and often lead to explanations with limited class discriminability. In contrast, gradient based localization methods are more effective at highlighting class specific evidence, but they do not fully exploit the hierarchical attention propagation mechanism of transformers. To address this limitation, we propose Decision-Aware Attention Propagation (DAP), an attribution method that injects decision-relevant priors into transformer attention propagation. By estimating token importance through gradient based localization and integrating it into layer wise attention rollout, the method captures both the structural flow of attention and the evidence most relevant to the final prediction. Consequently, DAP produces attribution maps that are more class sensitive, compact, and faithful than those generated by conventional attention based methods. Extensive experiments across Vision Transformer variants of different model scales show that DAP consistently outperforms existing baselines in both quantitative metrics and qualitative visualizations, indicating that decision aware propagation is an effective direction for improving ViT interpretability.

• The paper proposes the Decision-Aware Attention Propagation (DAP) method, integrating gradient-based cues into Vision Transformer attention for enhanced explainability.
• The DAP method improves class sensitivity, token contribution consistency, and attention map coherence, verified through comprehensive quantitative and qualitative analyses.
• Experimentation reveals DAP's superior performance over traditional attention rollout and gradient-only methods, particularly in class discriminative tasks across multiple ViT scales.

Decision-Aware Attention Propagation for Vision Transformer Explainability

The paper presents a method for enhancing interpretability in Vision Transformers (ViTs) by integrating gradient-derived decision cues directly into the attention propagation mechanism. The proposed Decision-Aware Attention Propagation (DAP) method modifies the conventional attention rollout by injecting class-discriminative priors into the layer-wise token transitions. This integration yields explanations that preserve the intrinsic transformer attention flow while improving class sensitivity and attribution compactness.

Figure 1: Overall pipeline of Decision-Aware Attention Propagation (DAP).

Methodological Framework

The foundation of the DAP method lies in coupling gradient-based localization with traditional self-attention propagation. Unlike standard attention-based explanation methods that rely solely on raw attention weights, DAP decomposes the propagation process into two key components. First, token importance is estimated using a gradient-based method (specifically leveraging Grad-CAM). Second, these gradients are normalized to form a decision prior that modulates the residual-aware attention transition matrices at every transformer layer. The formulation centers on a multiplicative pairwise modulation—in which each token-to-token interaction is weighted by the product of corresponding decision cues—thereby ensuring that transitions are biased towards tokens that are semantically and decision-relevant.

The modulation is incorporated directly into the propagation operator before a row-normalization step preserves the distribution semantics. This refined propagation ensures that the final attribution map, extracted from the class-token row of the cumulative relevance matrix, is both propagation-consistent and class-discriminative.

Experimental Evaluation

The experimental section includes a comprehensive quantitative and qualitative analysis across multiple ViT backbones (ViT-T, ViT-S, ViT-B, and ViT-L). Under both balanced and non-balanced sampling settings, extensive evaluation was performed using metrics such as deletion (Del), insertion (Ins), Class Sensitivity (CS), Token Contribution Consistency (TCC), Attention Flow Sparsity (AFS), and Layer-wise Decision Alignment (LDA).

Notably, when comparing directly against attention-based methods (e.g., Attention Rollout, AttR) and hybrid methods (e.g., GMAR) as well as pure gradient-based methods (e.g., Grad-CAM, CDAM), DAP consistently achieves higher CS scores (up to 0.355 on ViT-L) and demonstrates significant improvements in TCC and AFS metrics. The ablation studies further underscore the importance of injecting the gradient-derived decision prior during propagation rather than solely at the final attribution stage. Quantitative results confirm that as the ViT backbone scale increases, the benefits of integrating decision cues become more pronounced, suggesting that richer representations facilitate more reliable and interpretable attention maps.

Figure 2: Layer-wise Attention Map Comparison Across Methods.

A comparison of layer-wise attention elucidates that DAP maintains a more coherent evolution of attention maps across transformer depth. Additionally, perturbation experiments employing deletion and insertion curves verify that the regions highlighted by DAP are tightly coupled with the network’s prediction.

Figure 3: Evaluation of Explanation Quality via Deletion, Mass, and Alignment Curves.

In a qualitative analysis, visualizations show that DAP yields attribution maps that gradually transition to concentrate on semantically relevant regions while suppressing less informative context. The coherent progression across layers contrasts with the broader, diffuse responses observed in traditional attention rollout and gradient-only methods.

Figure 4: Successful Case.

Theoretical and Practical Implications

By directly integrating decision relevance into token-level propagation, DAP bridges the gap between gradient-based localization and attention-based propagation. This method not only preserves the internal hierarchical information flow of the transformer but also provides a mechanism to align class discriminative evidence with the final model prediction. In practice, such an approach offers enhanced interpretability, making it easier to diagnose and trust model behavior in high-stakes applications. The results suggest that future work on transformer explainability should further explore mechanisms that combine internal attention structures with external decision cues, potentially extending the framework to other transformer-based architectures and diverse datasets.

Conclusion

The paper systematically develops Decision-Aware Attention Propagation (DAP), which augments ViT explainability by infusing gradient-derived decision cues into transformer attention propagation. Through rigorous experiments and comprehensive ablations, DAP is shown to improve class sensitivity, token contribution consistency, and layer-wise consistency while preserving the inherent structure of transformer attention. These findings lay a promising foundation for future research directed toward achieving a more reliable and theoretically grounded interpretability framework in transformer-based vision models.