Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Scaling Up Semi-supervised Learning with Unconstrained Unlabelled Data (2306.01222v2)

Published 2 Jun 2023 in cs.LG and cs.CV

Abstract: We propose UnMixMatch, a semi-supervised learning framework which can learn effective representations from unconstrained unlabelled data in order to scale up performance. Most existing semi-supervised methods rely on the assumption that labelled and unlabelled samples are drawn from the same distribution, which limits the potential for improvement through the use of free-living unlabeled data. Consequently, the generalizability and scalability of semi-supervised learning are often hindered by this assumption. Our method aims to overcome these constraints and effectively utilize unconstrained unlabelled data in semi-supervised learning. UnMixMatch consists of three main components: a supervised learner with hard augmentations that provides strong regularization, a contrastive consistency regularizer to learn underlying representations from the unlabelled data, and a self-supervised loss to enhance the representations that are learnt from the unlabelled data. We perform extensive experiments on 4 commonly used datasets and demonstrate superior performance over existing semi-supervised methods with a performance boost of 4.79%. Extensive ablation and sensitivity studies show the effectiveness and impact of each of the proposed components of our method.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. In International Conference on Learning Representations.
  2. Remixmatch: Semi-Supervised learning with distribution matching and augmentation anchoring. In International Conference on Learning Representations.
  3. Mixmatch: A holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems.
  4. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 1597–1607.
  5. Exploring simple siamese representation learning. In Computer Vision and Pattern Recognition, 15750–15758.
  6. An analysis of single-layer networks in unsupervised feature learning. In International Conference on Artificial Intelligence and Statistics, 215–223.
  7. Randaugment: Practical automated data augmentation with a reduced search space. In Computer Vision and Pattern Recognition Workshops, 702–703.
  8. Imagenet: A large-scale hierarchical image database. In International Conference on Computer Vision, 248–255.
  9. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations.
  10. Bootstrap your own latent-a new approach to self-supervised learning. In Advances in Neural Information Processing Systems, 21271–21284.
  11. Improving barely supervised learning by discriminating unlabeled samples with super-class. In Advances in Neural Information Processing Systems.
  12. Safe deep semi-supervised learning for unseen-class unlabeled data. In International Conference on Machine Learning, 3897–3906.
  13. Conmatch: Semi-supervised learning with confidence-guided consistency regularization. In European Conference on Computer Vision, 674–690.
  14. Learning multiple layers of features from tiny images. Technical report, University of Toronto.
  15. Contrastive regularization for semi-supervised learning. In Computer Vision and Pattern Recognition, 3911–3920.
  16. Lee, D.-H.; et al. 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, ICML.
  17. Comatch: Semi-supervised learning with contrastive graph regularization. In International Conference on Computer Vision, 9475–9484.
  18. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, (8): 1979–1993.
  19. Reading digits in natural images with unsupervised feature learning. In NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning.
  20. Realistic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information Processing Systems.
  21. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
  22. Impact of labelled set selection and supervision policies on semi-supervised learning. arXiv preprint arXiv:2211.14912.
  23. Openmatch: Open-set consistency regularization for semi-supervised learning with outliers. In Advances in Neural Information Processing Systems.
  24. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In Advances in Neural Information Processing Systems.
  25. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In Advances in Neural Information Processing Systems, 596–608.
  26. A realistic evaluation of semi-supervised learning for fine-grained classification. In Computer Vision and Pattern Recognition, 12966–12975.
  27. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems.
  28. Unsupervised data augmentation for consistency training. In Advances in Neural Information Processing Systems, 6256–6268.
  29. Self-training with noisy student improves imagenet classification. In Computer Vision and Pattern Recognition, 10687–10698.
  30. Class-aware contrastive semi-supervised learning. In Computer Vision and Pattern Recognition, 14421–14430.
  31. Classification-reconstruction learning for open-set recognition. In Computer Vision and Pattern Recognition, 4016–4025.
  32. Multi-task curriculum framework for open-set semi-supervised learning. In European Conference on Computer Vision, 438–454.
  33. Wide residual networks. In British Machine Vision Conference.
  34. S4l: Self-supervised semi-supervised learning. In International Conference on Computer Vision, 1476–1485.
  35. Flexmatch: boosting semi-supervised learning with curriculum pseudo labeling. In Advances in Neural Information Processing Systems, 18408–18419.
  36. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations.
  37. Colorful image colorization. In European Conference on Computer Vision, 649–666.
  38. SimMatch: Semi-supervised learning with similarity matching. In Computer Vision and Pattern Recognition, 14471–14481.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Shuvendu Roy (18 papers)
  2. Ali Etemad (118 papers)
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now