Papers
Topics
Authors
Recent
Search
2000 character limit reached

Improving generalization by mimicking the human visual diet

Published 15 Jun 2022 in cs.CV, cs.AI, and cs.GR | (2206.07802v2)

Abstract: We present a new perspective on bridging the generalization gap between biological and computer vision -- mimicking the human visual diet. While computer vision models rely on internet-scraped datasets, humans learn from limited 3D scenes under diverse real-world transformations with objects in natural context. Our results demonstrate that incorporating variations and contextual cues ubiquitous in the human visual training data (visual diet) significantly improves generalization to real-world transformations such as lighting, viewpoint, and material changes. This improvement also extends to generalizing from synthetic to real-world data -- all models trained with a human-like visual diet outperform specialized architectures by large margins when tested on natural image data. These experiments are enabled by our two key contributions: a novel dataset capturing scene context and diverse real-world transformations to mimic the human visual diet, and a transformer model tailored to leverage these aspects of the human visual diet. All data and source code can be accessed at https://github.com/Spandan-Madan/human_visual_diet.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (86)
  1. Richard Zhang. Making convolutional networks shift-invariant again. In International conference on machine learning, pages 7324–7334. PMLR, 2019.
  2. Truly shift-invariant convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3773–3783, 2021.
  3. Small in-distribution changes in 3d perspective and lighting fool both cnns and transformers. arXiv preprint arXiv:2106.16198, 2021.
  4. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pages 456–473, 2018.
  5. Adversarial relighting against face recognition. arXiv preprint arXiv:2108.07920, 2021.
  6. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
  7. Beyond pixel norm-balls: Parametric adversaries using an analytically differentiable renderer. arXiv preprint arXiv:1808.02651, 2018.
  8. Adversarial attacks beyond the image space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4302–4311, 2019.
  9. Three approaches to facilitate invariant neurons and generalization to out-of-distribution orientations and illuminations. Neural Networks, 155:119–143, 2022.
  10. Semantic adversarial attacks: Parametric transformations that fool deep classifiers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4773–4783, 2019.
  11. Colorfool: Semantic adversarial colorization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1151–1160, 2020.
  12. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023–6032, 2019.
  13. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv:1912.02781, 2019.
  14. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.
  15. Diva: Domain invariant variational autoencoders. In Medical Imaging with Deep Learning, pages 322–348. PMLR, 2020.
  16. Cross-domain face presentation attack detection via multi-domain disentangled representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6678–6687, 2020.
  17. Domain generalization via invariant feature representation. In International conference on machine learning, pages 10–18. PMLR, 2013.
  18. Domain generalization with adversarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5400–5409, 2018.
  19. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European conference on computer vision (ECCV), pages 624–639, 2018.
  20. Unified deep supervised domain adaptation and generalization. In Proceedings of the IEEE international conference on computer vision, pages 5715–5725, 2017.
  21. Multi-adversarial discriminative deep domain generalization for face presentation attack detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10023–10031, 2019.
  22. Respecting domain relations: Hypothesis invariance for domain generalization. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 9756–9763. IEEE, 2021.
  23. Sand-mask: An enhanced gradient masking strategy for the discovery of invariances in domain generalization. arXiv preprint arXiv:2106.02266, 2021.
  24. Deep CORAL: correlation alignment for deep domain adaptation. CoRR, abs/1607.01719, 2016.
  25. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019.
  26. Selfreg: Self-supervised contrastive regularization for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9619–9628, 2021.
  27. An empirical investigation of domain generalization with empirical risk minimizers. Advances in Neural Information Processing Systems, 34, 2021.
  28. Out-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning, pages 5815–5826. PMLR, 2021.
  29. Domain generalization by marginal transfer learning. arXiv preprint arXiv:1711.07910, 2017.
  30. Likelihood ratios for out-of-distribution detection. Advances in neural information processing systems, 32, 2019.
  31. Detecting out-of-distribution examples with in-distribution examples and gram matrices. arXiv e-prints, pages arXiv–1912, 2019.
  32. A survey of outlier detection methodologies. Artificial intelligence review, 22:85–126, 2004.
  33. Outlier detection for high dimensional data. In Proceedings of the 2001 ACM SIGMOD international conference on Management of data, pages 37–46, 2001.
  34. Principles of neural science, volume 4. McGraw-hill New York, 2000.
  35. Gabriel Kreiman. Biological and Computer Vision. Cambridge University Press, 2021.
  36. Seeing faces is necessary for face-domain formation. Nature neuroscience, 20(10):1404–1412, 2017.
  37. Effects of monocular deprivation in kittens. Naunyn-Schmiedebergs Archiv für Experimentelle Pathologie und Pharmakologie, 248:492–497, 1964.
  38. NW Daw and HJ Wyatt. Kittens reared in a unidirectional environment: evidence for a critical period. The Journal of physiology, 257(1):155–170, 1976.
  39. The development of invariant object recognition requires visual experience with temporally smooth objects. Cognitive Science, 42(4):1391–1406, 2018.
  40. Justin N Wood and Samantha Marie Waters Wood. The development of object recognition requires experience with the surface features of objects. bioRxiv, pages 2022–12, 2022.
  41. When and how convolutional neural networks generalize to out-of-distribution category–viewpoint combinations. Nature Machine Intelligence, 4(2):146–153, 2022.
  42. Self-supervised learning through the eyes of a child. Advances in Neural Information Processing Systems, 33:9960–9971, 2020.
  43. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017.
  44. Openrooms: An end-to-end open framework for photorealistic indoor scene datasets. arXiv preprint arXiv:2007.12868, 2020.
  45. ilab-20m: A large-scale controlled object dataset to investigate deep learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2221–2230, 2016.
  46. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision, pages 1501–1510, 2017.
  47. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  48. Learning multiple layers of features from tiny images. 2009.
  49. When pigs fly: Contextual reasoning in synthetic and natural scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 255–264, 2021.
  50. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  51. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017.
  52. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  53. Three approaches to facilitate dnn generalization to objects in out-of-distribution orientations and illuminations: late-stopping, tuning batch normalization and invariance loss. arXiv preprint arXiv:2111.00131, 2021.
  54. To which out-of-distribution object orientations are dnns capable of generalizing? arXiv preprint arXiv:2109.13445, 2021.
  55. Investigating the nature of 3d generalization in deep neural networks. arXiv preprint arXiv:2304.09358, 2023.
  56. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
  57. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
  58. Putting visual object recognition in context. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12985–12994, 2020.
  59. Domain generalization in vision: A survey. arXiv preprint arXiv:2103.02503, 2021.
  60. Kentaro Wada. labelme: Image polygonal annotation with python. https://github.com/wkentaro/labelme, 2018.
  61. Real-world visual statistics and infants’ first-learned object names. Philosophical Transactions of the Royal Society B: Biological Sciences, 372(1711):20160055, 2017.
  62. Visual search asymmetry: Deep nets and humans share similar inherent biases. Advances in neural information processing systems, 34:6946–6959, 2021.
  63. Scene context impairs perception of semantically congruent objects. Psychological Science, 33(2):299–313, 2022. PMID: 35020519.
  64. Michelle R Greene. Statistics of high-level scene context. Frontiers in psychology, 4:777, 2013.
  65. Peter De Graef. Scene-context effects and models of real-world perception. In Eye movements and visual cognition, pages 243–259. Springer, 1992.
  66. Contextual guidance of eye movements and attention in real-world scenes: the role of global features in object search. Psychological review, 113(4):766, 2006.
  67. John M Henderson. Object identification in context: the visual processing of natural scenes. Canadian Journal of Psychology/Revue canadienne de psychologie, 46(3):319, 1992.
  68. Katherine Mary Mathis. Does scene context automatically influence object recognition? Evidence from an interference task. State University of New York at Albany, 1998.
  69. Context-dependent generalization. Frontiers in Human Neuroscience, 7:171, 2013.
  70. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261, 2019.
  71. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pages 5542–5550, 2017.
  72. Unbiased look at dataset bias. In CVPR 2011, pages 1521–1528. IEEE, 2011.
  73. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5018–5027, 2017.
  74. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1406–1415, 2019.
  75. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2475–2484, 2020.
  76. Deep sky modeling for single image outdoor lighting estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6927–6935, 2019.
  77. Spatiotemporal contrastive video representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6964–6974, 2021.
  78. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  79. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
  80. Variational prototype replays for continual learning. arXiv preprint arXiv:1905.09447, 2019.
  81. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33:18661–18673, 2020.
  82. Generalisation in humans and deep neural networks. Advances in neural information processing systems, 31, 2018.
  83. Human-aided saliency maps improve generalization of deep learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2735–2744, 2022.
  84. Densenet: Implementing efficient convnet descriptor pyramids. arXiv preprint arXiv:1404.1869, 2014.
  85. Joost CF De Winter. Using the student’s t-test with extremely small sample sizes. Practical Assessment, Research, and Evaluation, 18(1):10, 2019.
  86. Harry O Posten. Two-sample wilcoxon power over the pearson system and comparison with the t-test. Journal of Statistical Computation and Simulation, 16(1):1–18, 1982.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up