Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Coordinated Sparse Recovery of Label Noise (2404.04800v1)

Published 7 Apr 2024 in cs.LG, cs.CV, and stat.ML

Abstract: Label noise is a common issue in real-world datasets that inevitably impacts the generalization of models. This study focuses on robust classification tasks where the label noise is instance-dependent. Estimating the transition matrix accurately in this task is challenging, and methods based on sample selection often exhibit confirmation bias to varying degrees. Sparse over-parameterized training (SOP) has been theoretically effective in estimating and recovering label noise, offering a novel solution for noise-label learning. However, this study empirically observes and verifies a technical flaw of SOP: the lack of coordination between model predictions and noise recovery leads to increased generalization error. To address this, we propose a method called Coordinated Sparse Recovery (CSR). CSR introduces a collaboration matrix and confidence weights to coordinate model predictions and noise recovery, reducing error leakage. Based on CSR, this study designs a joint sample selection strategy and constructs a comprehensive and powerful learning framework called CSR+. CSR+ significantly reduces confirmation bias, especially for datasets with more classes and a high proportion of instance-specific noise. Experimental results on simulated and real-world noisy datasets demonstrate that both CSR and CSR+ achieve outstanding performance compared to methods at the same level.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understanding deep learning (still) requires rethinking generalization,” Communications of the ACM, vol. 64, no. 3, pp. 107–115, 2021.
  2. D. Arpit, S. Jastrzębski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. Courville, Y. Bengio et al., “A closer look at memorization in deep networks,” in International conference on machine learning.   PMLR, 2017, pp. 233–242.
  3. L. Jiang, Z. Zhou, T. Leung, L.-J. Li, and L. Fei-Fei, “Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels,” in International conference on machine learning.   PMLR, 2018, pp. 2304–2313.
  4. H. Song, M. Kim, D. Park, Y. Shin, and J.-G. Lee, “Learning from noisy labels with deep neural networks: A survey,” IEEE Transactions on Neural Networks and Learning Systems, 2022.
  5. N. Natarajan, I. S. Dhillon, P. K. Ravikumar, and A. Tewari, “Learning with noisy labels,” Advances in neural information processing systems, vol. 26, 2013.
  6. N. Manwani and P. Sastry, “Noise tolerance under risk minimization,” IEEE transactions on cybernetics, vol. 43, no. 3, pp. 1146–1151, 2013.
  7. B. Van Rooyen, A. Menon, and R. C. Williamson, “Learning with symmetric label noise: The importance of being unhinged,” Advances in neural information processing systems, vol. 28, 2015.
  8. A. Ghosh, H. Kumar, and P. S. Sastry, “Robust loss functions under label noise for deep neural networks,” in Proceedings of the AAAI conference on artificial intelligence, vol. 31, no. 1, 2017.
  9. Z. Zhang and M. Sabuncu, “Generalized cross entropy loss for training deep neural networks with noisy labels,” Advances in neural information processing systems, vol. 31, 2018.
  10. Y. Wang, X. Ma, Z. Chen, Y. Luo, J. Yi, and J. Bailey, “Symmetric cross entropy for robust learning with noisy labels,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 322–330.
  11. X. Ma, H. Huang, Y. Wang, S. Romano, S. Erfani, and J. Bailey, “Normalized loss functions for deep learning with noisy labels,” in International conference on machine learning.   PMLR, 2020, pp. 6543–6553.
  12. Y. Wang, X. Sun, and Y. Fu, “Scalable penalized regression for noise detection in learning with noisy labels,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 346–355.
  13. C. M. Bishop, “Training with noise is equivalent to tikhonov regularization,” Neural Computation, vol. 7, no. 1, pp. 108–116, 1995.
  14. L. Gu and T. Kanade, “A generative shape regularization model for robust face alignment,” in Computer Vision–ECCV 2008: 10th European Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part I 10.   Springer, 2008, pp. 413–426.
  15. J. Hoffman, D. A. Roberts, and S. Yaida, “Robust learning with jacobian regularization,” arXiv preprint arXiv:1908.02729, 2019.
  16. X. Xia, T. Liu, B. Han, N. Wang, M. Gong, H. Liu, G. Niu, D. Tao, and M. Sugiyama, “Part-dependent label noise: Towards instance-dependent label noise,” Advances in Neural Information Processing Systems, vol. 33, pp. 7597–7610, 2020.
  17. S. Yang, E. Yang, B. Han, Y. Liu, M. Xu, G. Niu, and T. Liu, “Estimating instance-dependent label-noise transition matrix using dnns,” 2021.
  18. E. Arazo, D. Ortego, P. Albert, N. O’Connor, and K. McGuinness, “Unsupervised label noise modeling and loss correction,” in International conference on machine learning.   PMLR, 2019, pp. 312–321.
  19. H. Cheng, Z. Zhu, X. Li, Y. Gong, X. Sun, and Y. Liu, “Learning with instance-dependent label noise: A sample sieve approach,” arXiv preprint arXiv:2010.02347, 2020.
  20. E. Englesson and H. Azizpour, “Generalized jensen-shannon divergence loss for learning with noisy labels,” Advances in Neural Information Processing Systems, vol. 34, pp. 30 284–30 297, 2021.
  21. B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama, “Co-teaching: Robust training of deep neural networks with extremely noisy labels,” Advances in neural information processing systems, vol. 31, 2018.
  22. J. Li, R. Socher, and S. C. Hoi, “Dividemix: Learning with noisy labels as semi-supervised learning,” arXiv preprint arXiv:2002.07394, 2020.
  23. N. Karim, M. N. Rizve, N. Rahnavard, A. Mian, and M. Shah, “Unicon: Combating label noise through uniform selection and contrastive learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 9676–9686.
  24. J. Li, C. Xiong, and S. C. Hoi, “Learning from noisy data with robust representation learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 9485–9494.
  25. S. Liu, J. Niles-Weed, N. Razavian, and C. Fernandez-Granda, “Early-learning regularization prevents memorization of noisy labels,” Advances in neural information processing systems, vol. 33, pp. 20 331–20 342, 2020.
  26. D. Tanaka, D. Ikami, T. Yamasaki, and K. Aizawa, “Joint optimization framework for learning with noisy labels,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 5552–5560.
  27. X. Yu, B. Han, J. Yao, G. Niu, I. Tsang, and M. Sugiyama, “How does disagreement help generalization against label corruption?” in International Conference on Machine Learning.   PMLR, 2019, pp. 7164–7173.
  28. Y. Zhang, S. Zheng, P. Wu, M. Goswami, and C. Chen, “Learning with feature-dependent label noise: A progressive approach,” arXiv preprint arXiv:2103.07756, 2021.
  29. Z. Zhu, F. Liu, G. Chrysos, and V. Cevher, “Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization),” Advances in Neural Information Processing Systems, vol. 35, pp. 36 094–36 107, 2022.
  30. H. Wei, L. Tao, R. Xie, and B. An, “Open-set label noise can improve robustness against inherent label noise,” Advances in Neural Information Processing Systems, vol. 34, pp. 7978–7992, 2021.
  31. X. Wang, Y. Hua, E. Kodirov, and N. M. Robertson, “Imae for noise-robust learning: Mean absolute error does not treat examples equally and gradient magnitude’s variance matters,” arXiv preprint arXiv:1903.12141, 2019.
  32. Y. Xu, P. Cao, Y. Kong, and Y. Wang, “L_dmi: A novel information-theoretic loss function for training deep nets robust to label noise,” Advances in neural information processing systems, vol. 32, 2019.
  33. J. Goldberger and E. Ben-Reuven, “Training deep neural-networks using a noise adaptation layer,” in International conference on learning representations, 2016.
  34. X. Xia, T. Liu, N. Wang, B. Han, C. Gong, G. Niu, and M. Sugiyama, “Are anchor points really indispensable in label-noise learning?” Advances in neural information processing systems, vol. 32, 2019.
  35. G. Patrini, A. Rozza, A. Krishna Menon, R. Nock, and L. Qu, “Making deep neural networks robust to label noise: A loss correction approach,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1944–1952.
  36. R. Tanno, A. Saeedi, S. Sankaranarayanan, D. C. Alexander, and N. Silberman, “Learning from noisy labels by regularized estimation of annotator confusion,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 11 244–11 253.
  37. E. Malach and S. Shalev-Shwartz, “Decoupling" when to update" from" how to update",” Advances in neural information processing systems, vol. 30, 2017.
  38. J. Liu, R. Li, and C. Sun, “Co-correcting: noise-tolerant medical image classification via mutual label correction,” IEEE Transactions on Medical Imaging, vol. 40, no. 12, pp. 3580–3592, 2021.
  39. D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel, “Mixmatch: A holistic approach to semi-supervised learning,” Advances in neural information processing systems, vol. 32, 2019.
  40. D. Patel and P. Sastry, “Adaptive sample selection for robust learning under label noise,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023, pp. 3932–3942.
  41. Y. Li, H. Han, S. Shan, and X. Chen, “Disc: Learning from noisy labels via dynamic instance-specific selection and correction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 24 070–24 079.
  42. P. Nakkiran, G. Kaplun, Y. Bansal, T. Yang, B. Barak, and I. Sutskever, “Deep double descent: Where bigger models and more data hurt,” Journal of Statistical Mechanics: Theory and Experiment, vol. 2021, no. 12, p. 124003, 2021.
  43. K. Wen, J. Teng, and J. Zhang, “Benign overfitting in classification: Provably counter label noise with larger models,” in The Eleventh International Conference on Learning Representations, 2022.
  44. D. A. Nguyen, R. Levie, J. Lienen, G. Kutyniok, and E. Hüllermeier, “Memorization-dilation: Modeling neural collapse under noise,” arXiv preprint arXiv:2206.05530, 2022.
  45. S. Liu, Z. Zhu, Q. Qu, and C. You, “Robust training under label noise by over-parameterization,” in International Conference on Machine Learning.   PMLR, 2022, pp. 14 153–14 172.
  46. C. You, Z. Zhu, Q. Qu, and Y. Ma, “Robust recovery via implicit bias of discrepant learning rates for double over-parameterization,” Advances in Neural Information Processing Systems, vol. 33, pp. 17 733–17 744, 2020.
  47. F. A. Wani, M. S. Bucarelli, and F. Silvestri, “Combining distance to class centroids and outlier discounting for improved learning with noisy labels,” arXiv preprint arXiv:2303.09470, 2023.
  48. X. C. F. Z. Y. L. Wei Yang, Luhui Xu, “Chi-squared distance metric learning for histogram data,” Mathematical Problems in Engineering, 2015.
  49. H. Wang, R. Xiao, Y. Dong, L. Feng, and J. Zhao, “Promix: combating label noise via maximizing clean sample utility,” arXiv preprint arXiv:2207.10276, 2022.
  50. A. Krizhevsky, G. Hinton et al., “Learning multiple layers of features from tiny images,” 2009.
  51. J. Wei, Z. Zhu, H. Cheng, T. Liu, G. Niu, and Y. Liu, “Learning with noisy labels revisited: A study using real-world human annotations,” arXiv preprint arXiv:2110.12088, 2021.
  52. T. Xiao, T. Xia, Y. Yang, C. Huang, and X. Wang, “Learning from massive noisy labeled data for image classification,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 2691–2699.
  53. W. Li, L. Wang, W. Li, E. Agustsson, and L. Van Gool, “Webvision database: Visual learning and understanding from web data,” arXiv preprint arXiv:1708.02862, 2017.
  54. P.-T. De Boer, D. P. Kroese, S. Mannor, and R. Y. Rubinstein, “A tutorial on the cross-entropy method,” Annals of operations research, vol. 134, pp. 19–67, 2005.
  55. C. J. Willmott and K. Matsuura, “Advantages of the mean absolute error (mae) over the root mean square error (rmse) in assessing average model performance,” Climate research, vol. 30, no. 1, pp. 79–82, 2005.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets