Combining Self-Supervised and Supervised Learning with Noisy Labels (2011.08145v2)
Abstract: Since convolutional neural networks (CNNs) can easily overfit noisy labels, which are ubiquitous in visual classification tasks, it has been a great challenge to train CNNs against them robustly. Various methods have been proposed for this challenge. However, none of them pay attention to the difference between representation and classifier learning of CNNs. Thus, inspired by the observation that classifier is more robust to noisy labels while representation is much more fragile, and by the recent advances of self-supervised representation learning (SSRL) technologies, we design a new method, i.e., CS$3$NL, to obtain representation by SSRL without labels and train the classifier directly with noisy labels. Extensive experiments are performed on both synthetic and real benchmark datasets. Results demonstrate that the proposed method can beat the state-of-the-art ones by a large margin, especially under a high noisy level.
- “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
- Deep learning, MIT press, 2016.
- “Deep residual learning for image recognition,” in CVPR, 2016.
- “Going deeper with convolutions,” in CVPR, 2015, pp. 1–9.
- “Faster r-cnn: Towards real-time object detection with region proposal networks,” NeurIPS, vol. 28, 2015.
- “Mask R-CNN,” in ICCV, 2017.
- “Imagenet large scale visual recognition challenge,” IJCV, vol. 115, pp. 211–252, 2015.
- “Microsoft coco: Common objects in context,” in ECCV. Springer, 2014, pp. 740–755.
- “A closer look at memorization in deep networks,” in ICML. PMLR, 2017, pp. 233–242.
- “Exploring generalization in deep learning,” in NeurIPS, 2017.
- “Understanding deep learning requires rethinking generalization,” in ICLR, 2017.
- “Beyond synthetic noise: Deep learning on controlled noisy labels,” in ICML. PMLR, 2020, pp. 4804–4815.
- “A survey of label-noise representation learning: Past, present and future,” Tech. Rep., arXiv:2011.04406, 2020.
- “Training convolutional networks with noisy labels,” in ICLR, 2015.
- “Learning from massive noisy labeled data for image classification,” in CVPR, 2015.
- “Making deep neural networks robust to label noise: A loss correction approach,” in CVPR, 2017.
- “Unsupervised label noise modeling and loss correction,” in ICML, 2019.
- “Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels,” in ICML. PMLR, 2018, pp. 2304–2313.
- “Co-teaching: Robust training of deep neural networks with extremely noisy labels,” in NeurIPS, 2018.
- “A simple framework for contrastive learning of visual representations,” in ICML, 2020.
- “Momentum contrast for unsupervised visual representation learning,” in CVPR, 2020.
- “Contrastive learning of global and local features for medical image segmentation with limited annotations,” NeurIPS, vol. 33, pp. 12546–12558, 2020.
- T. M Mitchell, Machine learning, vol. 1, McGraw-hill New York, 1997.
- “Bootstrap your own latent-a new approach to self-supervised learning,” NeurIPS, vol. 33, pp. 21271–21284, 2020.
- “Dividemix: Learning with noisy labels as semi-supervised learning,” in ICLR, 2020.
- D. A Reynolds et al., “Gaussian mixture models.,” Encyclopedia of biometrics, vol. 741, no. 659-663, 2009.
- “MixUp: Beyond empirical risk minimization,” in ICLR, 2018.
- “Deep self-learning from noisy labels,” in ICCV, 2019.
- “Learning multiple layers of features from tiny images,” 2009.
- “Webvision database: Visual learning and understanding from web data,” Tech. Rep., 2017.
- “Learning to learn from noisy labeled data,” in CVPR, 2019, pp. 5051–5059.
- “Early-learning regularization prevents memorization of noisy labels,” NeurIPS, vol. 33, pp. 20331–20342, 2020.
- “Augmentation strategies for learning with noisy labels,” in CVPR, 2021, pp. 8022–8031.
- “Understanding and utilizing deep neural networks trained with noisy labels,” in ICML, 2019.
- “Dimensionality-driven learning with noisy labels,” in ICML, 2018.