Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Noisy Label Processing for Classification: A Survey (2404.04159v1)

Published 5 Apr 2024 in cs.CV and cs.AI

Abstract: In recent years, deep neural networks (DNNs) have gained remarkable achievement in computer vision tasks, and the success of DNNs often depends greatly on the richness of data. However, the acquisition process of data and high-quality ground truth requires a lot of manpower and money. In the long, tedious process of data annotation, annotators are prone to make mistakes, resulting in incorrect labels of images, i.e., noisy labels. The emergence of noisy labels is inevitable. Moreover, since research shows that DNNs can easily fit noisy labels, the existence of noisy labels will cause significant damage to the model training process. Therefore, it is crucial to combat noisy labels for computer vision tasks, especially for classification tasks. In this survey, we first comprehensively review the evolution of different deep learning approaches for noisy label combating in the image classification task. In addition, we also review different noise patterns that have been proposed to design robust algorithms. Furthermore, we explore the inner pattern of real-world label noise and propose an algorithm to generate a synthetic label noise pattern guided by real-world data. We test the algorithm on the well-known real-world dataset CIFAR-10N to form a new real-world data-guided synthetic benchmark and evaluate some typical noise-robust methods on the benchmark.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (88)
  1. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” NeurIPS, vol. 25, pp. 1097–1105, 2012.
  2. X. Yu, T. Liu, M. Gong, and D. Tao, “Learning with biased complementary labels,” in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 68–83.
  3. X. Xie, J. Mao, Y. Liu, M. de Rijke, Q. Ai, Y. Huang, M. Zhang, and S. Ma, “Improving web image search with contextual information,” in Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 2019, pp. 1683–1692.
  4. R. V. Lloyd, L. A. Erickson, M. B. Casey, K. Y. Lam, C. M. Lohse, S. L. Asa, J. K. Chan, R. A. DeLellis, H. R. Harach, K. Kakudo et al., “Observer variation in the diagnosis of follicular variant of papillary thyroid carcinoma,” The American journal of surgical pathology, vol. 28, no. 10, pp. 1336–1340, 2004.
  5. C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understanding deep learning (still) requires rethinking generalization,” Communications of the ACM, vol. 64, no. 3, pp. 107–115, 2021.
  6. B. Frénay and M. Verleysen, “Classification in the presence of label noise: a survey,” IEEE transactions on neural networks and learning systems, vol. 25, no. 5, pp. 845–869, 2013.
  7. F. R. Cordeiro and G. Carneiro, “A survey on deep learning with noisy labels: How to train your model when you cannot trust on the annotations?” in 2020 33rd SIBGRAPI conference on graphics, patterns and images (SIBGRAPI).   IEEE, 2020, pp. 9–16.
  8. D. Karimi, H. Dou, S. K. Warfield, and A. Gholipour, “Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis,” Medical image analysis, vol. 65, p. 101759, 2020.
  9. H. Song, M. Kim, D. Park, Y. Shin, and J.-G. Lee, “Learning from noisy labels with deep neural networks: A survey,” IEEE Transactions on Neural Networks and Learning Systems, 2022.
  10. Y. Kim, J. Yim, J. Yun, and J. Kim, “Nlnl: Negative learning for noisy labels,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 101–110.
  11. G. Patrini, A. Rozza, A. Krishna Menon, R. Nock, and L. Qu, “Making deep neural networks robust to label noise: A loss correction approach,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1944–1952.
  12. Z. Zhang and M. R. Sabuncu, “Generalized cross entropy loss for training deep neural networks with noisy labels,” in 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018.
  13. X. Xia, T. Liu, B. Han, N. Wang, M. Gong, H. Liu, G. Niu, D. Tao, and M. Sugiyama, “Part-dependent label noise: Towards instance-dependent label noise,” Advances in Neural Information Processing Systems, vol. 33, pp. 7597–7610, 2020.
  14. Y. Zhang, S. Zheng, P. Wu, M. Goswami, and C. Chen, “Learning with feature-dependent label noise: A progressive approach,” arXiv preprint arXiv:2103.07756, 2021.
  15. P. Chen, J. Ye, G. Chen, J. Zhao, and P.-A. Heng, “Beyond class-conditional assumption: A primary attempt to combat instance-dependent label noise.” in Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
  16. J. Wei, Z. Zhu, H. Cheng, T. Liu, G. Niu, and Y. Liu, “Learning with noisy labels revisited: A study using real-world human annotations,” arXiv preprint arXiv:2110.12088, 2021.
  17. S. Sukhbaatar, J. Bruna, M. Paluri, L. Bourdev, and R. Fergus, “Training convolutional networks with noisy labels,” arXiv preprint arXiv:1406.2080, 2014.
  18. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” The journal of machine learning research, vol. 15, no. 1, pp. 1929–1958, 2014.
  19. J. Goldberger and E. Ben-Reuven, “Training deep neural-networks using a noise adaptation layer,” in 5th International Conference on Learning Representations, 2017.
  20. X. Chen and A. Gupta, “Webly supervised learning of convolutional networks,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 1431–1439.
  21. D. Hendrycks, M. Mazeika, D. Wilson, and K. Gimpel, “Using trusted data to train deep networks on labels corrupted by severe noise,” Advances in neural information processing systems, vol. 31, 2018.
  22. M. S. Bucarelli, L. Cassano, F. Siciliano, A. Mantrach, and F. Silvestri, “Leveraging inter-rater agreement for classification in the presence of noisy labels,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 3439–3448.
  23. T. Xiao, T. Xia, Y. Yang, C. Huang, and X. Wang, “Learning from massive noisy labeled data for image classification,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 2691–2699.
  24. B. Han, J. Yao, G. Niu, M. Zhou, I. Tsang, Y. Zhang, and M. Sugiyama, “Masking: A new perspective of noisy supervision,” Advances in neural information processing systems, vol. 31, 2018.
  25. J. Yao, J. Wang, I. W. Tsang, Y. Zhang, J. Sun, C. Zhang, and R. Zhang, “Deep learning from noisy image labels with quality embedding,” IEEE Transactions on Image Processing, vol. 28, no. 4, pp. 1909–1922, 2018.
  26. K. Lee, S. Yun, K. Lee, H. Lee, B. Li, and J. Shin, “Robust inference via generative classifiers for handling noisy labels,” in International conference on machine learning.   PMLR, 2019, pp. 3763–3772.
  27. M. Collier, B. Mustafa, E. Kokiopoulou, R. Jenatton, and J. Berent, “Correlated input-dependent label noise in large-scale image classification,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 1551–1560.
  28. X. Xia, B. Han, N. Wang, J. Deng, J. Li, Y. Mao, and T. Liu, “Extended t𝑡titalic_t t: Learning with mixed closed-set and open-set noisy labels,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 3, pp. 3047–3058, 2022.
  29. J. Han, P. Luo, and X. Wang, “Deep self-learning from noisy labels,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 5138–5147.
  30. R. Tanno, A. Saeedi, S. Sankaranarayanan, D. C. AleXaNder, and N. Silberman, “Learning from noisy labels by regularized estimation of annotator confusion,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  31. S. Liu, J. Niles-Weed, N. Razavian, and C. Fernandez-Granda, “Early-learning regularization prevents memorization of noisy labels,” 2020.
  32. D. Li and H. Zhang, “Improved regularization and robustness for fine-tuning in neural networks,” Advances in Neural Information Processing Systems, vol. 34, pp. 27 249–27 262, 2021.
  33. S. Jenni and P. Favaro, “Deep bilevel learning,” in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 618–633.
  34. D. Hendrycks, K. Lee, and M. Mazeika, “Using pre-training can improve model robustness and uncertainty,” in International conference on machine learning.   PMLR, 2019, pp. 2712–2721.
  35. A. K. Menon, A. S. Rawat, S. J. Reddi, and S. Kumar, “Can gradient clipping mitigate label noise?” in International Conference on Learning Representations, 2019.
  36. X. Xia, T. Liu, B. Han, C. Gong, N. Wang, Z. Ge, and Y. Chang, “Robust early-learning: Hindering the memorization of noisy labels,” in International conference on learning representations, 2020.
  37. H. Wei, L. Tao, R. Xie, and B. An, “Open-set label noise can improve robustness against inherent label noise,” Advances in Neural Information Processing Systems, vol. 34, pp. 7978–7992, 2021.
  38. I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” arXiv preprint arXiv:1412.6572, 2014.
  39. H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” arXiv preprint arXiv:1710.09412, 2017.
  40. G. Pereyra, G. Tucker, J. Chorowski, Ł. Kaiser, and G. Hinton, “Regularizing neural networks by penalizing confident output distributions,” arXiv preprint arXiv:1701.06548, 2017.
  41. M. Lukasik, S. Bhojanapalli, A. Menon, and S. Kumar, “Does label smoothing mitigate label noise?” in International Conference on Machine Learning.   PMLR, 2020, pp. 6448–6458.
  42. C.-B. Zhang, P.-T. Jiang, Q. Hou, Y. Wei, Q. Han, Z. Li, and M.-M. Cheng, “Delving deep into label smoothing,” IEEE Transactions on Image Processing, vol. 30, pp. 5984–5996, 2021.
  43. J. Lienen and E. Hüllermeier, “From label smoothing to label relaxation,” in Proceedings of the AAAI conference on artificial intelligence, vol. 35, no. 10, 2021, pp. 8583–8591.
  44. J. Ko, B. Yi, and S.-Y. Yun, “A gift from label smoothing: robust training with adaptive label smoothing via auxiliary classifier under label noise,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 7, 2023, pp. 8325–8333.
  45. Y. Qu, S. Mo, and J. Niu, “Dat: Training deep networks robust to label-noise by matching the feature distributions,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 6821–6829.
  46. H. Cheng, Z. Zhu, X. Sun, and Y. Liu, “Mitigating memorization of noisy labels via regularization between representations,” in International Conference on Learning Representations (ICLR), 2023.
  47. L. Jiang, Z. Zhou, T. Leung, L.-J. Li, and L. Fei-Fei, “Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels,” in International Conference on Machine Learning.   PMLR, 2018, pp. 2304–2313.
  48. B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama, “Co-teaching: Robust training of deep neural networks with extremely noisy labels,” in Advances in neural information processing systems, 2018, pp. 8527–8537.
  49. P. Chen, B. Liao, G. Chen, and S. Zhang, “Understanding and utilizing deep neural networks trained with noisy labels,” arXiv preprint arXiv:1905.05040, 2019.
  50. J. Huang, L. Qu, R. Jia, and B. Zhao, “O2u-net: A simple noisy label detection approach for deep neural networks,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 3326–3334.
  51. E. Malach and S. Shalev-Shwartz, “Decoupling" when to update" from" how to update",” Advances in neural information processing systems, vol. 30, 2017.
  52. X. Yu, B. Han, J. Yao, G. Niu, I. Tsang, and M. Sugiyama, “How does disagreement help generalization against label corruption?” in International Conference on Machine Learning.   PMLR, 2019, pp. 7164–7173.
  53. H. Wei, L. Feng, X. Chen, and B. An, “Combating noisy labels by agreement: A joint training method with co-regularization,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 13 726–13 735.
  54. Y. Shen and S. Sanghavi, “Learning with bad training data via iterative trimmed loss minimization,” in International Conference on Machine Learning.   PMLR, 2019, pp. 5739–5748.
  55. Y. Wang, W. Liu, X. Ma, J. Bailey, H. Zha, L. Song, and S.-T. Xia, “Iterative learning with open-set noisy labels,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8688–8696.
  56. H. Song, M. Kim, D. Park, Y. Shin, and J.-G. Lee, “Robust learning by self-transition for handling noisy labels,” in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021, pp. 1490–1500.
  57. P. Wu, S. Zheng, M. Goswami, D. Metaxas, and C. Chen, “A topological filter for learning with label noise,” Advances in neural information processing systems, vol. 33, pp. 21 382–21 393, 2020.
  58. Z.-F. Wu, T. Wei, J. Jiang, C. Mao, M. Tang, and Y.-F. Li, “Ngc: A unified framework for learning with open-world noisy data,” arXiv preprint arXiv:2108.11035, 2021.
  59. J. Li, R. Socher, and S. C. Hoi, “Dividemix: Learning with noisy labels as semi-supervised learning,” arXiv preprint arXiv:2002.07394, 2020.
  60. D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel, “Mixmatch: A holistic approach to semi-supervised learning,” Advances in neural information processing systems, vol. 32, 2019.
  61. F. R. Cordeiro, R. Sachdeva, V. Belagiannis, I. Reid, and G. Carneiro, “Longremix: Robust learning with high confidence samples in a noisy label environment,” Pattern Recognition, vol. 133, p. 109013, 2023.
  62. K. Nishi, Y. Ding, A. Rich, and T. Hllerer, “Augmentation strategies for learning with noisy labels,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  63. E. Zheltonozhskii, C. Baskin, A. Mendelson, A. M. Bronstein, and O. Litany, “Contrast to divide: Self-supervised pre-training for learning with noisy labels,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp. 1657–1667.
  64. H. Song, M. Kim, and J.-G. Lee, “Selfie: Refurbishing unclean samples for robust deep learning,” in International Conference on Machine Learning.   PMLR, 2019, pp. 5907–5915.
  65. D. T. Nguyen, C. K. Mummadi, T. P. N. Ngo, T. H. P. Nguyen, L. Beggel, and T. Brox, “Self: Learning to filter noisy labels with self-ensembling,” arXiv preprint arXiv:1910.01842, 2019.
  66. T. Zhou, S. Wang, and J. Bilmes, “Robust curriculum learning: From clean label detection to noisy label self-correction,” in International Conference on Learning Representations, 2020.
  67. M. Chen, H. Cheng, Y. Du, M. Xu, W. Jiang, and C. Wang, “Two wrongs don’t make a right: Combating confirmation bias in learning with label noise,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 12, 2023, pp. 14 765–14 773.
  68. Y. Kim, J. Yun, H. Shon, and J. Kim, “Joint negative and positive learning for noisy labels,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 9442–9451.
  69. A. Silva, L. Luo, S. Karunasekera, and C. Leckie, “Noise-robust learning from multiple unsupervised sources of inferred labels,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 8, 2022, pp. 8315–8323.
  70. Y. Bai, E. Yang, B. Han, Y. Yang, J. Li, Y. Mao, G. Niu, and T. Liu, “Understanding and improving early stopping for learning with noisy labels,” Advances in Neural Information Processing Systems, vol. 34, pp. 24 392–24 403, 2021.
  71. Y. Wang, X. Sun, and Y. Fu, “Scalable penalized regression for noise detection in learning with noisy labels,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 346–355.
  72. H. Chen, R. Tao, Y. Fan, Y. Wang, J. Wang, B. Schiele, X. Xie, B. Raj, and M. Savvides, “Softmatch: Addressing the quantity-quality trade-off in semi-supervised learning,” arXiv preprint arXiv:2301.10921, 2023.
  73. P. Chen, J. Ye, G. Chen, J. Zhao, and P.-A. Heng, “Robustness of accuracy metric and its inspirations in learning with noisy labels,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 13, 2021, pp. 11 451–11 461.
  74. X.-J. Gui, W. Wang, and Z.-H. Tian, “Towards understanding deep learning from noisy labels with small-loss criterion,” arXiv preprint arXiv:2106.09291, 2021.
  75. T. Kim, J. Ko, J. Choi, S.-Y. Yun et al., “Fine samples for learning with noisy labels,” Advances in Neural Information Processing Systems, vol. 34, pp. 24 137–24 149, 2021.
  76. Q. Wang, B. Han, T. Liu, G. Niu, J. Yang, and C. Gong, “Tackling instance-dependent label noise via a universal probabilistic model,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 11, 2021, pp. 10 183–10 191.
  77. Y. Li, H. Han, S. Shan, and X. Chen, “Disc: Learning from noisy labels via dynamic instance-specific selection and correction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 24 070–24 079.
  78. C. Feng, Y. Ren, and X. Xie, “Ot-filter: An optimal transport filter for learning with noisy labels,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 16 164–16 174.
  79. Y. Xu, X. Niu, J. Yang, S. Drew, J. Zhou, and R. Chen, “Usdnl: uncertainty-based single dropout in noisy label learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 9, 2023, pp. 10 648–10 656.
  80. Y. Yao, Z. Sun, C. Zhang, F. Shen, Q. Wu, J. Zhang, and Z. Tang, “Jo-src: A contrastive approach for combating noisy labels,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 5192–5201.
  81. D. Ortego, E. Arazo, P. Albert, N. E. O’Connor, and K. McGuinness, “Multi-objective interpolation training for robustness to label noise,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 6606–6615.
  82. N. Karim, M. N. Rizve, N. Rahnavard, A. Mian, and M. Shah, “Unicon: Combating label noise through uniform selection and contrastive learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 9676–9686.
  83. S. Li, X. Xia, S. Ge, and T. Liu, “Selective-supervised contrastive learning with noisy labels,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 316–325.
  84. P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan, “Supervised contrastive learning,” Advances in neural information processing systems, vol. 33, pp. 18 661–18 673, 2020.
  85. Z. Huang, J. Zhang, and H. Shan, “Twin contrastive learning with noisy labels,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 11 661–11 670.
  86. L. Jiang, D. Huang, M. Liu, and W. Yang, “Beyond synthetic noise: Deep learning on controlled noisy labels,” in International conference on machine learning.   PMLR, 2020, pp. 4804–4815.
  87. C. Zhu, H. Dong, and S. Zhang, “Feature fusion for image retrieval with adaptive bitrate allocation and hard negative mining,” IEEE Access, vol. 7, pp. 161 858–161 870, 2019.
  88. X. Ma, H. Huang, Y. Wang, S. Romano, S. Erfani, and J. Bailey, “Normalized loss functions for deep learning with noisy labels,” in International conference on machine learning.   PMLR, 2020, pp. 6543–6553.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Mengting Li (11 papers)
  2. Chuang Zhu (33 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now