Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

BaCon: Boosting Imbalanced Semi-supervised Learning via Balanced Feature-Level Contrastive Learning (2403.12986v2)

Published 4 Mar 2024 in cs.CV and cs.LG

Abstract: Semi-supervised Learning (SSL) reduces the need for extensive annotations in deep learning, but the more realistic challenge of imbalanced data distribution in SSL remains largely unexplored. In Class Imbalanced Semi-supervised Learning (CISSL), the bias introduced by unreliable pseudo-labels can be exacerbated by imbalanced data distributions. Most existing methods address this issue at instance-level through reweighting or resampling, but the performance is heavily limited by their reliance on biased backbone representation. Some other methods do perform feature-level adjustments like feature blending but might introduce unfavorable noise. In this paper, we discuss the bonus of a more balanced feature distribution for the CISSL problem, and further propose a Balanced Feature-Level Contrastive Learning method (BaCon). Our method directly regularizes the distribution of instances' representations in a well-designed contrastive manner. Specifically, class-wise feature centers are computed as the positive anchors, while negative anchors are selected by a straightforward yet effective mechanism. A distribution-related temperature adjustment is leveraged to control the class-wise contrastive degrees dynamically. Our method demonstrates its effectiveness through comprehensive experiments on the CIFAR10-LT, CIFAR100-LT, STL10-LT, and SVHN-LT datasets across various settings. For example, BaCon surpasses instance-level method FixMatch-based ABC on CIFAR10-LT with a 1.21% accuracy improvement, and outperforms state-of-the-art feature-level method CoSSL on CIFAR100-LT with a 0.63% accuracy improvement. When encountering more extreme imbalance degree, BaCon also shows better robustness than other methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (40)
  1. ReMixMatch: Semi-Supervised Learning with Distribution Matching and Augmentation Anchoring. In International Conference on Learning Representations.
  2. Mixmatch: A holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems 32, 5050–5060.
  3. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems 33, virtual.
  4. A simple framework for contrastive learning of visual representations. In International conference on machine learning, 1597–1607.
  5. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 15750–15758.
  6. An Analysis of Single-Layer Networks in Unsupervised Feature Learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of JMLR Proceedings, 215–223.
  7. Cossl: Co-learning of representation and classifier for imbalanced semi-supervised learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 14574–14584.
  8. Semi-supervised learning objectives as log-likelihoods in a generative model of data curation. arXiv:2008.05913.
  9. Bootstrap your own latent-a new approach to self-supervised learning. In Advances in Neural Information Processing Systems 33, virtual.
  10. Class-Imbalanced Semi-Supervised Learning with Adaptive Thresholding. In International Conference on Machine Learning, 8082–8094.
  11. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, 297–304. JMLR Workshop and Conference Proceedings.
  12. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9729–9738.
  13. Distribution Aligning Refinery of Pseudo-label for Imbalanced Semi-supervised Learning. In Advances in Neural Information Processing Systems 33, virtual.
  14. Learning Multiple Layers of Features from Tiny Images. Technical Report.
  15. Smoothed Adaptive Weighting for Imbalanced Semi-Supervised Learning: Improve Reliability Against Unknown Distribution Data. In International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, 11828–11843.
  16. Temporal Ensembling for Semi-Supervised Learning. In 5th International Conference on Learning Representations.
  17. Lee, D.-H.; et al. 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, 896. Atlanta.
  18. ABC: Auxiliary Balanced Classifier for Class-imbalanced Semi-supervised Learning. In Advances in Neural Information Processing Systems 34, virtual, 7082–7094.
  19. Comatch: Semi-supervised learning with contrastive graph regularization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 9475–9484.
  20. Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(8): 1979–1993.
  21. Reading Digits in Natural Images with Unsupervised Feature Learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning.
  22. DASO: Distribution-Aware Semantics-Oriented Pseudo-label for Imbalanced Semi-Supervised Learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 9776–9786.
  23. An Overview of Deep Semi-Supervised Learning. arXiv:2006.05278.
  24. FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence. In Advances in Neural Information Processing Systems, Virtual.
  25. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems 30, 1195–1204.
  26. Contrastive multiview coding. In European Conference on Computer Vision, 776–794.
  27. Representation Learning with Contrastive Predictive Coding. arXiv:1807.03748.
  28. Visualizing data using t-SNE. Journal of machine learning research, 9(11).
  29. Debiased Learning from Naturally Imbalanced Pseudo-Labels. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14627–14637.
  30. USB: A Unified Semi-supervised Learning Benchmark for Classification. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
  31. FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
  32. Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4238–4247.
  33. CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, 10857–10866.
  34. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, 3733–3742.
  35. Unsupervised Data Augmentation for Consistency Training. In Advances in Neural Information Processing Systems, Virtual.
  36. Rethinking the value of labels for improving class-imbalanced learning. In Advances in Neural Information Processing Systems 33, virtual.
  37. Wide Residual Networks. In Proceedings of the British Machine Vision Conference 2016.
  38. FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling. In Advances in Neural Information Processing, Virtual, 18408–18419.
  39. Simmatch: Semi-supervised learning with similarity matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14471–14481.
  40. Zhu, X. J. 2005. Semi-supervised learning literature survey. Technical Report.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Qianhan Feng (4 papers)
  2. Lujing Xie (2 papers)
  3. Shijie Fang (11 papers)
  4. Tong Lin (24 papers)
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now