Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Multi-Level Label Correction by Distilling Proximate Patterns for Semi-supervised Semantic Segmentation (2404.02065v2)

Published 2 Apr 2024 in cs.CV

Abstract: Semi-supervised semantic segmentation relieves the reliance on large-scale labeled data by leveraging unlabeled data. Recent semi-supervised semantic segmentation approaches mainly resort to pseudo-labeling methods to exploit unlabeled data. However, unreliable pseudo-labeling can undermine the semi-supervision processes. In this paper, we propose an algorithm called Multi-Level Label Correction (MLLC), which aims to use graph neural networks to capture structural relationships in Semantic-Level Graphs (SLGs) and Class-Level Graphs (CLGs) to rectify erroneous pseudo-labels. Specifically, SLGs represent semantic affinities between pairs of pixel features, and CLGs describe classification consistencies between pairs of pixel labels. With the support of proximate pattern information from graphs, MLLC can rectify incorrectly predicted pseudo-labels and can facilitate discriminative feature representations. We design an end-to-end network to train and perform this effective label corrections mechanism. Experiments demonstrate that MLLC can significantly improve supervised baselines and outperforms state-of-the-art approaches in different scenarios on Cityscapes and PASCAL VOC 2012 datasets. Specifically, MLLC improves the supervised baseline by at least 5% and 2% with DeepLabV2 and DeepLabV3+ respectively under different partition protocols.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431–3440, 2015.
  2. L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” IEEE transactions on pattern analysis and machine intelligence, vol. 40, no. 4, pp. 834–848, 2017.
  3. A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick, “Segment anything,” arXiv:2304.02643, 2023.
  4. M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213–3223, 2016.
  5. A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” arXiv preprint arXiv:1703.01780, 2017.
  6. C. Chen, S. Dong, Y. Tian, K. Cao, L. Liu, and Y. Guo, “Temporal self-ensembling teacher for semi-supervised object detection,” IEEE Transactions on Multimedia, vol. 24, pp. 3679–3692, 2021.
  7. Y. Li, Z. Zhao, H. Sun, Y. Cen, and Z. He, “Snowball: Iterative model evolution and confident sample discovery for semi-supervised learning on very small labeled datasets,” IEEE Transactions on Multimedia, vol. 23, pp. 1354–1366, 2020.
  8. G. French, S. Laine, T. Aila, M. Mackiewicz, and G. Finlayson, “Semi-supervised semantic segmentation needs strong, varied perturbations,” in British Machine Vision Conference, no. 31, 2020.
  9. V. Olsson, W. Tranheden, J. Pinto, and L. Svensson, “Classmix: Segmentation-based data augmentation for semi-supervised learning,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1369–1378, 2021.
  10. Y. Zhu, Z. Zhang, C. Wu, Z. Zhang, T. He, H. Zhang, R. Manmatha, M. Li, and A. Smola, “Improving semantic segmentation via self-training,” arXiv preprint arXiv:2004.14960, 2020.
  11. C. Wei, K. Sohn, C. Mellina, A. Yuille, and F. Yang, “Crest: A class-rebalancing self-training framework for imbalanced semi-supervised learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10857–10866, 2021.
  12. E. W. Teh, T. DeVries, B. Duke, R. Jiang, P. Aarabi, and G. W. Taylor, “The gist and rist of iterative self-training for semi-supervised segmentation,” arXiv preprint arXiv:2103.17105, 2021.
  13. R. Ke, A. I. Aviles-Rivero, S. Pandey, S. Reddy, and C.-B. Schönlieb, “A three-stage self-training framework for semi-supervised semantic segmentation,” IEEE Transactions on Image Processing, vol. 31, pp. 1805–1815, 2022.
  14. S. Mittal, M. Tatarchenko, and T. Brox, “Semi-supervised semantic segmentation with high-and low-level consistency,” IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 43, no. 04, pp. 1369–1379, 2021.
  15. W. C. Hung, Y. H. Tsai, Y. T. Liou, Y.-Y. Lin, and M. H. Yang, “Adversarial learning for semi-supervised semantic segmentation,” in 29th British Machine Vision Conference, BMVC 2018, 2018.
  16. E. Arazo, D. Ortego, P. Albert, N. E. O’Connor, and K. McGuinness, “Pseudo-labeling and confirmation bias in deep semi-supervised learning,” in International Joint Conference on Neural Networks, pp. 1–8, IEEE, 2020.
  17. Y. Zhou, H. Xu, W. Zhang, B. Gao, and P.-A. Heng, “C3-semiseg: Contrastive semi-supervised segmentation via cross-set learning and dynamic class-balancing,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7036–7045, 2021.
  18. R. Mendel, L. A. De Souza, D. Rauber, J. P. Papa, and C. Palm, “Semi-supervised segmentation based on error-correcting supervision,” in Proceedings of the European Conference on Computer Vision, pp. 141–157, Springer, 2020.
  19. Z. Ke, D. Qiu, K. Li, Q. Yan, and R. W. Lau, “Guided collaborative training for pixel-wise semi-supervised learning,” in Proceedings of the European conference on computer vision, pp. 429–445, 2020.
  20. B. Zhang, J. Xiao, J. Jiao, Y. Wei, and Y. Zhao, “Affinity attention graph neural network for weakly supervised semantic segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 11, pp. 8082–8096, 2021.
  21. T. Yan, K. Zhu, G. Zhu, M. Tang, J. Wang, et al., “Plug-and-play pseudo label correction network for unsupervised person re-identification,” arXiv preprint arXiv:2206.06607, 2022.
  22. Y. Luo, J. Zhu, M. Li, Y. Ren, and B. Zhang, “Smooth neighbors on teacher graphs for semi-supervised learning,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8896–8905, 2018.
  23. Z. Song, X. Yang, Z. Xu, and I. King, “Graph-based semi-supervised learning: A comprehensive review,” IEEE Transactions on Neural Networks and Learning Systems, 2022.
  24. C. Gong, T. Liu, D. Tao, K. Fu, E. Tu, and J. Yang, “Deformed graph laplacian for semisupervised learning,” IEEE transactions on neural networks and learning systems, vol. 26, no. 10, pp. 2261–2274, 2015.
  25. L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in Proceedings of the European conference on computer vision, pp. 801–818, 2018.
  26. H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2881–2890, 2017.
  27. P. Luc, C. Couprie, S. Chintala, and J. Verbeek, “Semantic segmentation using adversarial networks,” arXiv preprint arXiv:1611.08408, 2016.
  28. K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li, “Fixmatch: Simplifying semi-supervised learning with consistency and confidence,” Advances in neural information processing systems, vol. 33, pp. 596–608, 2020.
  29. Z. Feng, Q. Zhou, Q. Gu, X. Tan, G. Cheng, X. Lu, J. Shi, and L. Ma, “Dmt: Dynamic mutual training for semi-supervised learning,” Pattern Recognition, p. 108777, 2022.
  30. Y. Ouali, C. Hudelot, and M. Tami, “Semi-supervised semantic segmentation with cross-consistency training,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12674–12684, 2020.
  31. Y. Xu, L. Shang, J. Ye, Q. Qian, Y.-F. Li, B. Sun, H. Li, and R. Jin, “Dash: Semi-supervised learning with dynamic thresholding,” in International Conference on Machine Learning, pp. 11525–11536, PMLR, 2021.
  32. Y. Liu, Y. Tian, Y. Chen, F. Liu, V. Belagiannis, and G. Carneiro, “Perturbed and strict mean teachers for semi-supervised semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4258–4267, 2022.
  33. Z. Wang, Z. Zhao, X. Xing, D. Xu, X. Kong, and L. Zhou, “Conflict-based cross-view consistency for semi-supervised semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19585–19595, 2023.
  34. L. Yang, L. Qi, L. Feng, W. Zhang, and Y. Shi, “Revisiting weak-to-strong consistency in semi-supervised semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7236–7246, 2023.
  35. D.-H. Lee et al., “Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks,” in Workshop on challenges in representation learning, ICML, vol. 3, p. 896, 2013.
  36. B. Zhang, Y. Wang, W. Hou, H. Wu, J. Wang, M. Okumura, and T. Shinozaki, “Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling,” Advances in Neural Information Processing Systems, vol. 34, pp. 18408–18419, 2021.
  37. H. Hu, F. Wei, H. Hu, Q. Ye, J. Cui, and L. Wang, “Semi-supervised semantic segmentation via adaptive equalization learning,” Advances in Neural Information Processing Systems, vol. 34, pp. 22106–22118, 2021.
  38. Y. Wang, H. Wang, Y. Shen, J. Fei, W. Li, G. Jin, L. Wu, R. Zhao, and X. Le, “Semi-supervised semantic segmentation using unreliable pseudo-labels,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4248–4257, 2022.
  39. D. Guan, J. Huang, A. Xiao, and S. Lu, “Unbiased subclass regularization for semi-supervised semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9968–9978, 2022.
  40. Y. Jin, J. Wang, and D. Lin, “Semi-supervised semantic segmentation via gentle teaching assistant,” Advances in Neural Information Processing Systems, vol. 35, pp. 2803–2816, 2022.
  41. L. Wu, L. Fang, X. He, M. He, J. Ma, and Z. Zhong, “Querying labeled for unlabeled: Cross-image semantic consistency guided semi-supervised semantic segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  42. P. Tu, Y. Huang, F. Zheng, Z. He, L. Cao, and L. Shao, “Guidedmix-net: Semi-supervised semantic segmentation by using labeled images as reference,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 2379–2387, 2022.
  43. S. Fan, F. Zhu, Z. Feng, Y. Lv, M. Song, and F.-Y. Wang, “Conservative-progressive collaborative learning for semi-supervised semantic segmentation,” IEEE Transactions on Image Processing, 2023.
  44. P. Qiao, Z. Wei, Y. Wang, Z. Wang, G. Song, F. Xu, X. Ji, C. Liu, and J. Chen, “Fuzzy positive learning for semi-supervised semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15465–15474, 2023.
  45. X. Wang, B. Zhang, L. Yu, and J. Xiao, “Hunting sparsity: Density-guided contrastive learning for semi-supervised semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3114–3123, 2023.
  46. H. Xu, L. Liu, Q. Bian, and Z. Yang, “Semi-supervised semantic segmentation with prototype-based consistency regularization,” Advances in Neural Information Processing Systems, vol. 35, pp. 26007–26020, 2022.
  47. Z. Zhao, L. Yang, S. Long, J. Pi, L. Zhou, and J. Wang, “Augmentation matters: A simple-yet-effective approach to semi-supervised semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11350–11359, 2023.
  48. M. Everingham, S. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes challenge: A retrospective,” International journal of computer vision, vol. 111, no. 1, pp. 98–136, 2015.
  49. Z. Feng, Q. Zhou, G. Cheng, X. Tan, J. Shi, and L. Ma, “Semi-supervised semantic segmentation via dynamic self-training and class-balanced curriculum,” arXiv e-prints, pp. arXiv–2004, 2020.
  50. Z. Feng, Q. Zhou, Q. Gu, X. Tan, G. Cheng, X. Lu, J. Shi, and L. Ma, “Dmt: Dynamic mutual training for semi-supervised learning,” arXiv preprint arXiv:2004.08514, 2020.
  51. I. Alonso, A. Sabater, D. Ferstl, L. Montesano, and A. C. Murillo, “Semi-supervised semantic segmentation with pixel-level contrastive learning from a class-wise memory bank,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8219–8228, 2021.
  52. X. Wang, J. Xiao, B. Zhang, and L. Yu, “Card: Semi-supervised semantic segmentation via class-agnostic relation based denoising,” in Proc. IJCAI, pp. 1451–1457, 2022.
  53. X. Chen, Y. Yuan, G. Zeng, and J. Wang, “Semi-supervised semantic segmentation with cross pseudo supervision,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2613–2622, 2021.
  54. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255, Ieee, 2009.
  55. Y. Zou, Z. Zhang, H. Zhang, C.-L. Li, X. Bian, J.-B. Huang, and T. Pfister, “Pseudoseg: Designing pseudo labels for semantic segmentation,” International Conference on Learning Representations, 2021.
  56. E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “Segformer: Simple and efficient design for semantic segmentation with transformers,” Advances in Neural Information Processing Systems, vol. 34, pp. 12077–12090, 2021.
  57. L. Hoyer, D. Dai, and L. Van Gool, “DAFormer: Improving network architectures and training strategies for domain-adaptive semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9924–9935, 2022.
  58. T. Miyato, S.-I. Maeda, M. Koyama, and S. Ishii, “Virtual adversarial training: A regularization method for supervised and semi-supervised learning,” IEEE transactions on pattern analysis and machine intelligence, vol. 41, no. 8, pp. 1979–1993, 2019.
  59. Y. Zhong, B. Yuan, H. Wu, Z. Yuan, J. Peng, and Y.-X. Wang, “Pixel contrastive-consistent semi-supervised semantic segmentation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7273–7282, 2021.
  60. L. Yang, W. Zhuo, L. Qi, Y. Shi, and Y. Gao, “St++: Make self-training work better for semi-supervised semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4268–4277, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Hui Xiao (31 papers)
  2. Yuting Hong (3 papers)
  3. Li Dong (154 papers)
  4. Diqun Yan (15 papers)
  5. Jiayan Zhuang (3 papers)
  6. Junjie Xiong (5 papers)
  7. Dongtai Liang (1 paper)
  8. Chengbin Peng (8 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now