Token Transformation Matters: Towards Faithful Post-hoc Explanation for Vision Transformer (2403.14552v1)
Abstract: While Transformers have rapidly gained popularity in various computer vision applications, post-hoc explanations of their internal mechanisms remain largely unexplored. Vision Transformers extract visual information by representing image regions as transformed tokens and integrating them via attention weights. However, existing post-hoc explanation methods merely consider these attention weights, neglecting crucial information from the transformed tokens, which fails to accurately illustrate the rationales behind the models' predictions. To incorporate the influence of token transformation into interpretation, we propose TokenTM, a novel post-hoc explanation method that utilizes our introduced measurement of token transformation effects. Specifically, we quantify token transformation effects by measuring changes in token lengths and correlations in their directions pre- and post-transformation. Moreover, we develop initialization and aggregation rules to integrate both attention weights and token transformation effects across all layers, capturing holistic token contributions throughout the model. Experimental results on segmentation and perturbation tests demonstrate the superiority of our proposed TokenTM compared to state-of-the-art Vision Transformer explanation methods.
- Quantifying attention flow in transformers. In ACL, 2020.
- Openxai: Towards a transparent evaluation of model explanations. In NeurIPS, 2022.
- Xai for transformers: Better explanations through conservative propagation. In ICML, 2022.
- A diagnostic study of explainability techniques for text classification. In EMNLP, 2020.
- On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. In PloS one, 10(7):e0130140, 2015.
- Grad-sam: Explaining transformers via gradient self-attention maps. In CIKM, 2021.
- Visual explanations via iterated integrated attributions. In ICCV, 2023a.
- Deep integrated explanations. In CIKM, 2023b.
- Layer-wise relevance propagation for neural networks with local renormalization layers. In ICANN, 2016.
- On identifiability in transformers. In ICLR, 2020.
- End-to-end object detection with transformers. In ECCV, 2020.
- Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In ICCV, 2021a.
- Transformer interpretability beyond attention visualization. In CVPR, 2021b.
- Generating hierarchical explanations on text classification via feature interaction detection. In ACL, 2020.
- Real time image saliency for black box classifiers. In NeurIPS, 2017.
- Eraser: A benchmark to evaluate rationalized nlp models. In ACL, 2020.
- An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
- Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In EMNLP-IJCNLP, 2019.
- Interpretable explanations of black boxes by meaningful perturbation. In ICCV, 2017.
- Understanding individual decisions of cnns via contrastive backpropagation. In ACCV, 2018.
- Imagenet auto-annotation with segmentation propagation. IJCV, 110:328–348, 2014.
- Visualization of supervised and self-supervised neural networks via attribution guided factorization. In AAAI, 2021.
- Deep residual learning for image recognition. In CVPR, 2016.
- Masked autoencoders are scalable vision learners. In CVPR, 2022.
- Explaining convolutional neural networks using softmax gradient layer-wise relevance propagation. In ICCV Workshop, 2019.
- Attention is not explanation. In NAACL, 2019.
- Attention is not only a weight: Analyzing transformers with vector norms. In EMNLP, 2020.
- Learning multiple layers of features from tiny images. 2009.
- Adaptive cross-modal prototypes for cross-domain visual-language retrieval. In CVPR, 2021.
- Swin transformer v2: Scaling up capacity and resolution. In CVPR, 2022.
- A unified approach to interpreting model predictions. In NeurIPS, 2017.
- Visualizing deep convolutional neural networks using natural pre-images. IJCV, 120:233–255, 2016.
- Explaining nonlinear classification decisions with deep taylor decomposition. PR, 65:211–222, 2017.
- Dong Nguyen. Comparing automatic and human evaluation of local explanations for text classification. In NAACL, 2018.
- Attcat: Explaining transformers via attentive class activation tokens. In NeurIPS, 2022.
- Imagenet large scale visual recognition challenge. IJCV, 115:211–252, 2015.
- Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, 2017.
- Is attention interpretable? In ACL, 2019.
- Learning important features through propagating activation differences. In ICML, 2017a.
- Not just a black box: Learning important features through propagating activation differences. In ICML, 2017b.
- Smoothgrad: removing noise by adding noise. In ICML Workshop, 2017.
- Striving for simplicity: The all convolutional net. In ICLRW, 2015.
- Full-gradient representation for neural network visualization. In NeurIPS, 2019.
- Gilbert Strang. Linear algebra and its applications. Belmont, CA: Thomson, Brooks/Cole, 2006.
- Axiomatic attribution for deep networks. In ICML, 2017.
- Training data-efficient image transformers & distillation through attention. In ICML, 2021.
- Attention is all you need. In NeurIPS, 2017.
- Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In ACL, 2019.
- Attention is not not explanation. In EMNLP, 2019.
- Visualizing and understanding convolutional networks. In ECCV, 2014.
- Learning deep features for discriminative localization. In CVPR, 2016.
- Interpreting deep visual representations via network dissection. IEEE TPAMI, 41(9):2131–2145, 2018.
- Junyi Wu (15 papers)
- Bin Duan (22 papers)
- Weitai Kang (11 papers)
- Hao Tang (379 papers)
- Yan Yan (242 papers)