2000 character limit reached
CAM-Based Methods Can See through Walls (2404.01964v2)
Published 2 Apr 2024 in cs.CV and cs.LG
Abstract: CAM-based methods are widely-used post-hoc interpretability method that produce a saliency map to explain the decision of an image classification model. The saliency map highlights the important areas of the image relevant to the prediction. In this paper, we show that most of these methods can incorrectly attribute an important score to parts of the image that the model cannot see. We show that this phenomenon occurs both theoretically and experimentally. On the theory side, we analyze the behavior of GradCAM on a simple masked CNN model at initialization. Experimentally, we train a VGG-like model constrained to not use the lower part of the image and nevertheless observe positive scores in the unseen part of the image. This behavior is evaluated quantitatively on two new datasets. We believe that this is problematic, potentially leading to mis-interpretation of the model's behavior.
- Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, volume 31, 2018.
- On the convergence rate of training recurrent neural networks. Advances in Neural Information Processing Systems, 32, 2019a.
- A Convergence Theory for Deep Learning via Over-Parameterization. In Proceedings of the 36th International Conference on Machine Learning, 2019b.
- Eigen-CAM: Visual Explanations for Deep Convolutional Neural Networks. SN Computer Science, 2(1):47, 2021.
- Maxime Beauchamp. On numerical computation for the distribution of the convolution of N independent rectified Gaussian variables. Journal de la Société Française de Statistique, 2018.
- Are artificial neural networks black boxes? IEEE Transactions on Neural Networks, 8(5):1156–1164, 1997.
- Language models are few-shot learners. Advances in Neural Information Processing Systems, 2020.
- Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks. In IEEE Winter Conference on Applications of Computer Vision, pages 839–847, 2018.
- ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
- Ablation-CAM: Visual Explanations for Deep Convolutional Network via Gradient-free Localization. In IEEE Winter Conference on Applications of Computer Vision (WACV), pages 972–980, 2020.
- Use HiResCAM instead of Grad-CAM for faithful explanations of convolutional neural networks. arxiv preprint 2011.08891, 2021.
- Gradient descent finds global minima of deep neural networks. In Proceedings of the 36th International Conference on Machine Learning, 2019.
- Axiom-based Grad-CAM: Towards Accurate Visualization and Explanation of CNNs. In 31st British Machine Vision Conference, 2020.
- Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36(4):193–202, 1980.
- Interpretation of neural networks is fragile. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
- Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, pages 249–256, 2010.
- Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, pages 1026–1034, 2015.
- Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
- Fooling Neural Network Interpretations via Adversarial Model Manipulation. In Advances in Neural Information Processing Systems, volume 32, 2019.
- Layercam: Exploring hierarchical class activation maps for localization. IEEE Transactions on Image Processing, 2021.
- The (un) reliability of saliency methods. Explainable AI: Interpreting, explaining and visualizing deep learning, 2019.
- Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
- Wide neural networks of any depth evolve as linear models under gradient descent. Advances in Neural Information Processing Systems, 32, 2019.
- Explainable AI: A Review of Machine Learning Interpretability Methods. Entropy, 23(1):18, 2021.
- Zachary C. Lipton. The Mythos of Model Interpretability: In Machine Learning, the Concept of Interpretability is Both Important and Slippery. 2018.
- Zero-shot text-to-image generation. In International Conference on Machine Learning, 2021.
- Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 618–626, 2017.
- Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
- Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
- Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 111–119, 2020.
- CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features. In IEEE/CVF International Conference on Computer Vision, pages 6022–6031, 2019.
- Opti-CAM: Optimizing saliency maps for interpretability. arXiv preprint arXiv:2301.07002, 2023.
- A survey on neural network interpretability. IEEE Transactions on Emerging Topics in Computational Intelligence, 2021.
- Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2921–2929, 2016.
- Gradient descent optimizes over-parameterized deep relu networks. Machine Learning, 2020.