Papers
Topics
Authors
Recent
2000 character limit reached

Does resistance to style-transfer equal Global Shape Bias? Measuring network sensitivity to global shape configuration (2310.07555v3)

Published 11 Oct 2023 in cs.CV

Abstract: Deep learning models are known to exhibit a strong texture bias, while human tends to rely heavily on global shape structure for object recognition. The current benchmark for evaluating a model's global shape bias is a set of style-transferred images with the assumption that resistance to the attack of style transfer is related to the development of global structure sensitivity in the model. In this work, we show that networks trained with style-transfer images indeed learn to ignore style, but its shape bias arises primarily from local detail. We provide a \textbf{Disrupted Structure Testbench (DiST)} as a direct measurement of global structure sensitivity. Our test includes 2400 original images from ImageNet-1K, each of which is accompanied by two images with the global shapes of the original image disrupted while preserving its texture via the texture synthesis program. We found that \textcolor{black}{(1) models that performed well on the previous cue-conflict dataset do not fare well in the proposed DiST; (2) the supervised trained Vision Transformer (ViT) lose its global spatial information from positional embedding, leading to no significant advantages over Convolutional Neural Networks (CNNs) on DiST. While self-supervised learning methods, especially mask autoencoder significantly improves the global structure sensitivity of ViT. (3) Improving the global structure sensitivity is orthogonal to resistance to style-transfer, indicating that the relationship between global shape structure and local texture detail is not an either/or relationship. Training with DiST images and style-transferred images are complementary, and can be combined to train network together to enhance the global shape sensitivity and robustness of local features.} Our code will be hosted in github: https://github.com/leelabcnbc/DiST

Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Does the brain’s ventral visual pathway compute object shape? Trends in Cognitive Sciences, 26(12):1119–1132, 2022.
  2. Perception of an object’s global shape is best described by a model of skeletal structure in human infants. elife, 11:e74943, 2022.
  3. Deep learning models fail to capture the configural nature of human shape perception. Iscience, 25(9), 2022.
  4. Francis Brochu. Increasing shape bias in imagenet-trained networks using transfer learning and domain-adversarial methods. arXiv preprint arXiv:1907.12892, 2019.
  5. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  6. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp.  9650–9660, 2021.
  7. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  8. A systematic review of robustness in deep learning for computer vision: Mind the gap? arXiv preprint arXiv:2112.00639, 2021.
  9. Intrinsic images by clustering. In Computer graphics forum, volume 31, pp.  1415–1424. Wiley Online Library, 2012.
  10. Texture synthesis using convolutional neural networks. Advances in neural information processing systems, 28, 2015.
  11. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.12231, 2018.
  12. Partial success in closing the gap between human and machine vision. Advances in Neural Information Processing Systems, 34:23885–23899, 2021.
  13. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA), pp.  80–89. IEEE, 2018.
  14. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  15. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  16000–16009, 2022.
  16. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  4700–4708, 2017.
  17. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision, pp.  1501–1510, 2017.
  18. Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32, 2019.
  19. Self-supervised intrinsic image decomposition. Advances in neural information processing systems, 30, 2017.
  20. Mobilenetv3. Convolutional Neural Networks with Swift for Tensorflow: Image Recognition and Dataset Categorization, pp.  125–144, 2021.
  21. FFCV: Accelerating training by removing data bottlenecks. In Computer Vision and Pattern Recognition (CVPR), 2023. https://github.com/libffcv/ffcv/.
  22. Emergence of shape bias in convolutional neural networks through activation sparsity. Advances in Neural Information Processing Systems (Oral Presentation), -:–, 2023.
  23. Surfgen: Adversarial 3d shape synthesis with explicit surface discriminators. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  16238–16248, 2021.
  24. Vision transformers are robust learners. In Proceedings of the AAAI conference on Artificial Intelligence, volume 36, pp.  2071–2081, 2022.
  25. Opening the black box: the promise and limitations of explainable machine learning in cardiology. Canadian Journal of Cardiology, 38(2):204–213, 2022.
  26. Perceptual categorization of cat and dog silhouettes by 3-to 4-month-old infants. Journal of experimental child psychology, 79(1):78–94, 2001a.
  27. Developmental change in form categorization in early infancy. British Journal of Developmental Psychology, 19(2):207–218, 2001b.
  28. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  779–788, 2016.
  29. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825, 2017.
  30. Inception-v4, inception-resnet and the impact of residual connections on learning. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
  31. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  2820–2828, 2019.
  32. Deit iii: Revenge of the vit. In European Conference on Computer Vision, pp.  516–533. Springer, 2022.
  33. Convnet vs transformer, supervised vs clip: Beyond imagenet accuracy, 2024.
  34. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  16133–16142, 2023.
  35. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  10687–10698, 2020.
  36. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  1492–1500, 2017.
  37. Visualizing and understanding convolutional networks. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pp.  818–833. Springer, 2014.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up