Papers
Topics
Authors
Recent
Search
2000 character limit reached

DORA: Exploring Outlier Representations in Deep Neural Networks

Published 9 Jun 2022 in cs.LG, cs.AI, cs.CV, and stat.ML | (2206.04530v4)

Abstract: Deep Neural Networks (DNNs) excel at learning complex abstractions within their internal representations. However, the concepts they learn remain opaque, a problem that becomes particularly acute when models unintentionally learn spurious correlations. In this work, we present DORA (Data-agnOstic Representation Analysis), the first data-agnostic framework for analyzing the representational space of DNNs. Central to our framework is the proposed Extreme-Activation (EA) distance measure, which assesses similarities between representations by analyzing their activation patterns on data points that cause the highest level of activation. As spurious correlations often manifest in features of data that are anomalous to the desired task, such as watermarks or artifacts, we demonstrate that internal representations capable of detecting such artifactual concepts can be found by analyzing relationships within neural representations. We validate the EA metric quantitatively, demonstrating its effectiveness both in controlled scenarios and real-world applications. Finally, we provide practical examples from popular Computer Vision models to illustrate that representations identified as outliers using the EA metric often correspond to undesired and spurious concepts.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (143)
  1. GitHub - weiaicunzai/pytorch-cifar100: Practice on CIFAR100— github.com. https://github.com/weiaicunzai/pytorch-cifar100, 2020. [Accessed 08-Jan-2023].
  2. Post hoc explanations may be ineffective for detecting unknown spurious correlation. In International Conference on Learning Representations, 2022.
  3. Evaluating CLIP: towards characterization of broader capabilities and downstream implications. arXiv preprint arXiv:2108.02818, 2021.
  4. Finding and removing clever hans: Using explanation methods to debug and improve deep models. Information Fusion, 77:261–295, 2022.
  5. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130140, 2015.
  6. How to explain individual classification decisions. Journal of Machine Learning Research, 11(Jun):1803–1831, 2010.
  7. BEIT: BERT pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021.
  8. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6541–6549, 2017.
  9. GAN dissection: Visualizing and understanding generative adversarial networks. arXiv preprint arXiv:1811.10597, 2018.
  10. Dimensionality reduction for visualizing single-cell data using umap. Nature biotechnology, 37(1):38–44, 2019.
  11. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
  12. Temporal kernel CCA and its application in multimodal neuronal data analysis. Machine Learning, 79(1):5–27, 2010.
  13. On taxonomies for multi-class image categorization. International Journal of Computer Vision, 99(3):281–301, 2012.
  14. Shortcomings of top-down randomization-based sanity checks for evaluations of deep neural network explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16143–16152, 2023.
  15. Natural language processing with Python: analyzing text with the natural language toolkit. O’Reilly Media, Inc., 2009.
  16. E. Bisong and E. Bisong. Google colaboratory. Building machine learning and deep learning models on google cloud platform: a comprehensive guide for beginners, pages 59–64, 2019.
  17. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
  18. Natural images are more informative for interpreting cnn activations than state-of-the-art synthetic feature visualizations. In NeurIPS 2020 Workshop SVRHM, 2020.
  19. On relevant dimensions in kernel feature spaces. The Journal of Machine Learning Research, 9:1875–1908, 2008.
  20. LOF: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD International Conference on Management of data, pages 93–104, 2000.
  21. Using explainable AI to measure feature contribution to uncertainty. In The International FLAIRS Conference Proceedings, volume 35, 2022.
  22. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901, 2020.
  23. C.-A. Brust and J. Denzler. Not just a matter of semantics: The relationship between visual and semantic similarity. In German Conference on Pattern Recognition, pages 414–427. Springer, 2019.
  24. Analysis of explainers of black box Deep Neural Networks for Computer Vision: A survey. arXiv preprint arXiv:1911.12116, 2019.
  25. Explaining Bayesian Neural Networks. arXiv preprint arXiv:2108.10346, 2021.
  26. NoiseGrad—enhancing explanations by introducing stochasticity to model weights. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6132–6140, 2022.
  27. Curve detectors. Distill, 5(6):e00024–003, 2020.
  28. Exploring Neural Networks with activation atlases. Distill., 2019.
  29. This looks like that: deep learning for interpretable image recognition. arXiv preprint arXiv:1806.10574, 2018.
  30. J. Da. A corpus-based study of character and bigram frequencies in chinese e-texts and its implications for chinese language instruction. In Proceedings of the fourth International Conference on new technologies in teaching and learning Chinese, pages 501–511. Citeseer, 2004.
  31. B. Dai and D. Lin. Contrastive learning for image captioning. Advances in Neural Information Processing Systems, 30, 2017.
  32. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255. IEEE, 2009.
  33. L. Deng. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141–142, 2012.
  34. T. Deselaers and V. Ferrari. Visual and semantic similarity in imagenet. In CVPR 2011, pages 1777–1784. IEEE, 2011.
  35. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  36. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1, 2009.
  37. I. Fogel and D. Sagi. Gabor filters as texture discriminator. Biological cybernetics, 61(2):103–113, 1989.
  38. I. Gabriel. Artificial intelligence, values, and alignment. Minds and machines, 30(3):411–437, 2020.
  39. Explainable AI in industry. In Proceedings of the 25th ACM SIGKDD International Conference on knowledge discovery & data mining, pages 3203–3204, 2019.
  40. This looks more like that: Enhancing self-explaining models by prototypical relevance propagation. arXiv preprint arXiv:2108.12204, 2021.
  41. Protovae: A trustworthy self-explainable prototypical variational model. Advances in Neural Information Processing Systems, 35:17940–17952, 2022a.
  42. Demonstrating the risk of imbalanced datasets in chest x-ray image-based diagnostics by prototypical relevance propagation. In 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pages 1–5, 2022b. doi: 10.1109/ISBI52829.2022.9761651.
  43. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.12231, 2018.
  44. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020.
  45. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on Artificial Intelligence and Statistics, pages 315–323. JMLR Workshop and Conference Proceedings, 2011.
  46. Multimodal neurons in artificial neural networks. Distill, 6(3):e30, 2021.
  47. Visualizing the diversity of representations learned by bayesian neural networks. arXiv preprint arXiv:2201.10859, 2022.
  48. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733, 2017.
  49. R. Guidotti. Evaluating local explanation methods on ground truth. Artificial Intelligence, 291:103428, 2021.
  50. A survey of methods for explaining black box models. ACM computing surveys (CSUR), 51(5):1–42, 2018.
  51. I. Guyon and A. Elisseeff. An introduction to variable and feature selection. Journal of Machine Learning Research, 3(Mar):1157–1182, 2003.
  52. Canonical Correlation Analysis: An overview with application to learning methods. Neural computation, 16:2639–64, 01 2005. doi: 10.1162/0899766042321814.
  53. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
  54. Quantus: an explainable AI toolkit for responsible evaluation of neural network explanations. arXiv preprint arXiv:2202.06861, 2022.
  55. Natural language descriptions of deep visual features. In International Conference on Learning Representations, 2021.
  56. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670, 2018.
  57. Densely Connected Convolutional Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4700–4708, 2017.
  58. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv:1602.07360, 2016.
  59. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. Proceedings of the AAAI Conference on Artificial Intelligence, 33:590–597, 07 2019.
  60. On feature learning in the presence of spurious correlations. arXiv preprint arXiv:2210.11369, 2022.
  61. P. Jackson. Introduction to expert systems. URL https://www.osti.gov/biblio/5675197. [Accessed 16-Feb-2023].
  62. A survey on contrastive self-supervised learning. Technologies, 9(1):2, 2020.
  63. H. Jiang and O. Nachum. Identifying and correcting label bias in machine learning. In S. Chiappa and R. Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 702–712. PMLR, 26–28 Aug 2020. URL https://proceedings.mlr.press/v108/jiang20a.html.
  64. I. T. Jolliffe and J. Cadima. Principal component analysis: a review and recent developments. Philosophical transactions of the royal society A: Mathematical, Physical and Engineering Sciences, 374(2065):20150202, 2016.
  65. Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519–3529. PMLR, 2019.
  66. Angle-based outlier detection in high-dimensional data. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge discovery and data mining, pages 444–452, 2008.
  67. A. Krizhevsky. Learning multiple layers of features from tiny images. pages 32–33, 2009. URL https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf.
  68. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84–90, 2017.
  69. A. Laakso. Content and cluster analysis: Assessing representational similarity in neural systems. Philosophical Psychology, 13, 05 2000. doi: 10.1080/09515080050002726.
  70. Analyzing classifiers: Fisher vectors and deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2912–2920, 2016.
  71. Unmasking clever hans predictors and assessing what machines really learn. Nature communications, 10:1096, 2019.
  72. A. Lazarevic and V. Kumar. Feature bagging for outlier detection. In Proceedings of the eleventh ACM SIGKDD International Conference on Knowledge discovery in data mining, pages 157–166, 2005.
  73. M. Le and S. Kayal. Revisiting edge detection in convolutional neural networks. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1–9. IEEE, 2021.
  74. Y. Le and X. Yang. Tiny imagenet visual recognition challenge. Stanford CS 231N, 7(7):3, 2015.
  75. C. Leacock and M. Chodorow. Combining local context and wordnet similarity for word sense identification. WordNet: An electronic lexical database, 49(2):265–283, 1998.
  76. Y. LeCun and I. Misra. Self-supervised learning: The dark matter of intelligence, 2021. URL https://ai.facebook.com/blog/self-supervised-learning-the-dark-matter-of-intelligence/. [Accessed 08-Jan-2023].
  77. Convergent learning: Do different neural networks learn the same representations? arXiv preprint arXiv:1511.07543, 2015.
  78. A whac-a-mole dilemma: Shortcuts come in multiples where mitigating one amplifies others, 2022.
  79. Isolation Forest. In 2008 8-th IEEE International Conference on Data Mining, pages 413–422. IEEE, 2008.
  80. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European Conference on Computer Vision (ECCV), pages 116–131, 2018.
  81. N. Mantel. The detection of disease clustering and a generalized regression approach. Cancer Res., 27:175–178, 1967.
  82. S. Marcel and Y. Rodriguez. Torchvision the machine-vision package of torch. In Proceedings of the 18th ACM International Conference on Multimedia, pages 1485–1488, 2010.
  83. D. Marr and H. K. Nishihara. Representation and recognition of the spatial organization of three-dimensional shapes. Proceedings of the Royal Society of London. Series B. Biological Sciences, 200(1140):269–294, 1978.
  84. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3:861, 09 2018. doi: 10.21105/joss.00861.
  85. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.
  86. G. A. Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41, 1995.
  87. Kernel analysis of deep networks. Journal of Machine Learning Research, 12(9), 2011.
  88. Methods for interpreting and understanding deep neural networks. Digital Signal Processing, 73:1–15, 2018.
  89. Insights on representational similarity in neural networks with canonical correlation. Advances in Neural Information Processing Systems, 31, 2018.
  90. Differentiable image parameterizations. Distill, 3(7):e12, 2018.
  91. J. Mu and J. Andreas. Compositional explanations of neurons. Advances in Neural Information Processing Systems, 33:17153–17163, 2020.
  92. Human alignment of neural network representations. arXiv preprint arXiv:2211.01201, 2022.
  93. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Advances in Neural Information Processing Systems, pages 3387–3395, 2016.
  94. Understanding neural networks via feature visualization: A survey. In Explainable AI: interpreting, explaining and visualizing deep learning, pages 55–76. Springer, 2019.
  95. Innovation engines: Automated creativity and improved stochastic optimization via deep learning. In Proceedings of the 2015 annual conference on genetic and evolutionary computation, pages 959–966, 2015.
  96. Do wide and deep networks learn the same things? Uncovering how neural network representations vary with width and depth. arXiv preprint arXiv:2010.15327, 2020.
  97. On the origins of the block structure phenomenon in neural network representations. arXiv preprint arXiv:2202.07184, 2022.
  98. Feature visualization. Distill, 2(11):e7, 2017.
  99. Naturally occurring equivariance in neural networks. Distill, 5(12):e00024–004, 2020.
  100. Smooth Grad-Cam++: An enhanced inference level visualization technique for deep convolutional neural network models. arXiv preprint arXiv:1908.01224, 2019.
  101. Wordnet:: Similarity-measuring the relatedness of concepts. In AAAI, volume 4, pages 25–29, 2004.
  102. X. Qin and Z. Wang. Nasnet: A neuron attention stage-by-stage net for single image deraining. arXiv preprint arXiv:1912.03151, 2019.
  103. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  104. SVCCA: Singular Vector Canonical Correlation Analysis for deep understanding and improvement. arXiv preprint arXiv:1706.05806, 2017.
  105. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34:12116–12128, 2021.
  106. Matrix correlation. Psychometrika, 49:403–423, 09 1984. doi: 10.1007/BF02306029.
  107. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, June 2022.
  108. C. Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206–215, 2019.
  109. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 109(5):756–795, 2021.
  110. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211–252, 2015.
  111. Evaluating the visualization of what a deep neural network has learned. IEEE transactions on Neural Networks and Learning Systems, 28(11):2660–2673, 2016.
  112. Explainable AI: interpreting, explaining and visualizing deep learning, volume 11700. Springer Nature, 2019.
  113. Explaining deep neural networks and beyond: A review of methods and applications. Proceedings of the IEEE, 109(3):247–278, 2021.
  114. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4510–4520, 2018.
  115. Estimating the support of a high-dimensional distribution. Neural computation, 13(7):1443–1471, 2001.
  116. Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nature Machine Intelligence, 2(8):476–486, 2020.
  117. Grad-cam: Visual explanations from deep networks via gradient-based localization. International Journal of Computer Vision, 128(2):336–359, 10 2019. ISSN 1573-1405. doi: 10.1007/s11263-019-01228-7.
  118. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  119. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825, 2017.
  120. K. O. Stanley. Compositional pattern producing networks: A novel abstraction of development. Genetic programming and evolvable machines, 8:131–162, 2007.
  121. Axiomatic attribution for deep networks. In International Conference on Machine Learning, pages 3319–3328. PMLR, 2017.
  122. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
  123. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2818–2826, 2016.
  124. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64–73, 2016.
  125. I. Tiddi et al. Directions for explainable knowledge-enabled systems. Knowledge Graphs for eXplainable Artificial intelligence: Foundations Applications and Challenges, 47:245, 2020.
  126. Spectral signatures in backdoor attacks. Advances in Neural Information Processing Systems, 31, 2018.
  127. Umap as a dimensionality reduction tool for molecular dynamics simulations of biomacromolecules: a comparison study. The Journal of Physical Chemistry B, 125(19):5022–5034, 2021.
  128. L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
  129. Opening the black box: Revealing interpretable sequence motifs in kernel-based learning algorithms. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 137–153. Springer, 2015.
  130. Feature importance measure for non-linear learning algorithms. arXiv preprint arXiv:1611.07567, 2016.
  131. D. Wallis and I. Buvat. Clever hans effect found in a widely used brain tumour mri dataset. Medical Image Analysis, 77:102368, 2022.
  132. Towards multi-grained explainability for graph neural networks. Advances in Neural Information Processing Systems, 34:18446–18458, 2021.
  133. Y. Wang. CIFAR-100 Resnet PyTorch 75.17% Accuracy — kaggle.com. https://www.kaggle.com/code/yiweiwangau/cifar-100-resnet-pytorch-75-17-accuracy, 2021. [Accessed 08-Jan-2023].
  134. A survey of transfer learning. Journal of Big data, 3(1):1–40, 2016.
  135. R. Wightman. Pytorch image models. https://github.com/rwightman/pytorch-image-models, 2019.
  136. Comparison between umap and t-sne for multiplex-immunofluorescence derived single-cell data from tissue sections. BioRxiv, page 549659, 2019.
  137. Z. Wu and M. Palmer. Verb semantics and lexical selection. arXiv preprint cmp-lg/9406033, 1994.
  138. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint arXiv:2006.09994, 2020.
  139. Explainable AI: A brief survey on history, research areas, approaches and challenges. In CCF International Conference on natural language processing and Chinese computing, pages 563–574. Springer, 2019.
  140. Large-scale Robust Deep AUC Maximization: A new surrogate loss and empirical studies on medical image classification. pages 3020–3029, 10 2021. doi: 10.1109/ICCV48922.2021.00303.
  141. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS medicine, 15(11):e1002683, 2018.
  142. M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision, pages 818–833. Springer, 2014.
  143. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1):43–76, 2020.
Citations (11)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 6 likes about this paper.

Sign Up for Free