Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
173 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Exploration and Exploitation of Unlabeled Data for Open-Set Semi-Supervised Learning (2306.17699v1)

Published 30 Jun 2023 in cs.CV

Abstract: In this paper, we address a complex but practical scenario in semi-supervised learning (SSL) named open-set SSL, where unlabeled data contain both in-distribution (ID) and out-of-distribution (OOD) samples. Unlike previous methods that only consider ID samples to be useful and aim to filter out OOD ones completely during training, we argue that the exploration and exploitation of both ID and OOD samples can benefit SSL. To support our claim, i) we propose a prototype-based clustering and identification algorithm that explores the inherent similarity and difference among samples at feature level and effectively cluster them around several predefined ID and OOD prototypes, thereby enhancing feature learning and facilitating ID/OOD identification; ii) we propose an importance-based sampling method that exploits the difference in importance of each ID and OOD sample to SSL, thereby reducing the sampling bias and improving the training. Our proposed method achieves state-of-the-art in several challenging benchmarks, and improves upon existing SSL methods even when ID samples are totally absent in unlabeled data.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (53)
  1. M. Sajjadi, M. Javanmardi, and T. Tasdizen, “Regularization with stochastic transformations and perturbations for deep semi-supervised learning,” Advances in Neural Information Processing Systems, vol. 29, 2016.
  2. S. Laine and T. Aila, “Temporal ensembling for semi-supervised learning,” in International Conference on Learning Representations, 2017.
  3. D.-H. Lee et al., “Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks,” in Workshop on challenges in representation learning, ICML, vol. 3, no. 2, 2013.
  4. H. Pham, Z. Dai, Q. Xie, and Q. V. Le, “Meta pseudo labels,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 11 557–11 568.
  5. Y. Grandvalet and Y. Bengio, “Semi-supervised learning by entropy minimization,” Advances in Neural Information Processing Systems, vol. 17, 2004.
  6. K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li, “Fixmatch: Simplifying semi-supervised learning with consistency and confidence,” Advances in Neural Information Processing Systems, vol. 33, pp. 596–608, 2020.
  7. Q. Yu, D. Ikami, G. Irie, and K. Aizawa, “Multi-task curriculum framework for open-set semi-supervised learning,” in European Conference on Computer Vision.   Springer, 2020, pp. 438–454.
  8. S. Liang, Y. Li, and R. Srikant, “Enhancing the reliability of out-of-distribution image detection in neural networks,” in International Conference on Learning Representations, 2018.
  9. D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel, “Mixmatch: A holistic approach to semi-supervised learning,” Advances in Neural Information Processing Systems, vol. 32, 2019.
  10. J. Winkens, R. Bunel, A. G. Roy, R. Stanforth, V. Natarajan, J. R. Ledsam, P. MacWilliams, P. Kohli, A. Karthikesalingam, S. Kohl et al., “Contrastive training for improved out-of-distribution detection,” arXiv preprint arXiv:2007.05566, 2020.
  11. A. Krizhevsky, G. Hinton et al., “Learning multiple layers of features from tiny images,” 2009.
  12. Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, “Reading digits in natural images with unsupervised feature learning,” 2011.
  13. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition.   Ieee, 2009, pp. 248–255.
  14. X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang, “Moment matching for multi-source domain adaptation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 1406–1415.
  15. P. Bachman, O. Alsharif, and D. Precup, “Learning with pseudo-ensembles,” Advances in Neural Information Processing Systems, vol. 27, 2014.
  16. X. Zhai, A. Oliver, A. Kolesnikov, and L. Beyer, “S4l: Self-supervised semi-supervised learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 1476–1485.
  17. S. Park, J. Park, S.-J. Shin, and I.-C. Moon, “Adversarial dropout for supervised and semi-supervised learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, 2018.
  18. S. Wager, S. Wang, and P. S. Liang, “Dropout training as adaptive regularization,” Advances in Neural Information Processing Systems, vol. 26, 2013.
  19. D. Berthelot, N. Carlini, E. D. Cubuk, A. Kurakin, K. Sohn, H. Zhang, and C. Raffel, “Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring,” in International Conference on Learning Representations, 2019.
  20. A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” Advances in Neural Information Processing Systems, vol. 30, 2017.
  21. B. Zhang, Y. Wang, W. Hou, H. Wu, J. Wang, M. Okumura, and T. Shinozaki, “Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling,” Advances in Neural Information Processing Systems, vol. 34, pp. 18 408–18 419, 2021.
  22. J. Li, P. Zhou, C. Xiong, and S. Hoi, “Prototypical contrastive learning of unsupervised representations,” in International Conference on Learning Representations, 2020.
  23. T. DeVries and G. W. Taylor, “Learning confidence for out-of-distribution detection in neural networks,” arXiv preprint arXiv:1802.04865, 2018.
  24. Y. Ming, Z. Cai, J. Gu, Y. Sun, W. Li, and Y. Li, “Delving into out-of-distribution detection with vision-language representations,” in Advances in Neural Information Processing Systems, 2022.
  25. X. Du, G. Gozum, Y. Ming, and Y. Li, “Siren: Shaping representations for detecting out-of-distribution objects,” in Advances in Neural Information Processing Systems, 2022.
  26. J. Yang, P. Wang, D. Zou, Z. Zhou, K. Ding, W. PENG, H. Wang, G. Chen, B. Li, Y. Sun et al., “Openood: Benchmarking generalized out-of-distribution detection,” in Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
  27. C. Chow, “On optimum recognition error and reject tradeoff,” IEEE Transactions on information theory, vol. 16, no. 1, pp. 41–46, 1970.
  28. P. Vincent and Y. Bengio, “Manifold parzen windows,” Advances in Neural Information Processing Systems, pp. 849–856, 2003.
  29. A. Ghoting, S. Parthasarathy, and M. E. Otey, “Fast mining of distance-based outliers in high-dimensional datasets,” Data Mining and Knowledge Discovery, vol. 16, no. 3, pp. 349–364, 2008.
  30. D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” in International Conference on Learning Representations, 2017.
  31. K. Lee, K. Lee, H. Lee, and J. Shin, “A simple unified framework for detecting out-of-distribution samples and adversarial attacks,” Advances in neural information processing systems, vol. 31, 2018.
  32. W. Liu, X. Wang, J. Owens, and Y. Li, “Energy-based out-of-distribution detection,” Advances in Neural Information Processing Systems, vol. 33, pp. 21 464–21 475, 2020.
  33. Y.-C. Hsu, Y. Shen, H. Jin, and Z. Kira, “Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 10 951–10 960.
  34. Y. Sun, C. Guo, and Y. Li, “React: Out-of-distribution detection with rectified activations,” in Advances in Neural Information Processing Systems, 2021.
  35. A. Oliver, A. Odena, C. A. Raffel, E. D. Cubuk, and I. Goodfellow, “Realistic evaluation of deep semi-supervised learning algorithms,” Advances in Neural Information Processing Systems, vol. 31, 2018.
  36. H. Luo, H. Cheng, Y. Gao, K. Li, M. Zhang, F. Meng, X. Guo, F. Huang, and X. Sun, “On the consistency training for open-set semi-supervised learning,” arXiv preprint arXiv:2101.08237, 2021.
  37. Q. Xie, Z. Dai, E. Hovy, T. Luong, and Q. Le, “Unsupervised data augmentation for consistency training,” in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., vol. 33.   Curran Associates, Inc., 2020, pp. 6256–6268.
  38. Y. Chen, X. Zhu, W. Li, and S. Gong, “Semi-supervised learning under class distribution mismatch,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 04, 2020, pp. 3569–3576.
  39. J. Huang, C. Fang, W. Chen, Z. Chai, X. Wei, P. Wei, L. Lin, and G. Li, “Trash to treasure: Harvesting ood data with cross-modal matching for open-set semi-supervised learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 8310–8319.
  40. T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in International Conference on Machine Learning.   PMLR, 2020, pp. 1597–1607.
  41. L.-Z. Guo, Z.-Y. Zhang, Y. Jiang, Y.-F. Li, and Z.-H. Zhou, “Safe deep semi-supervised learning for unseen-class unlabeled data,” in International Conference on Machine Learning.   PMLR, 2020, pp. 3897–3906.
  42. K. Saito, D. Kim, and K. Saenko, “Openmatch: Open-set consistency regularization for semi-supervised learning with outliers,” in Advances in Neural Information Processing Systems, 2021.
  43. S. Zagoruyko and N. Komodakis, “Wide residual networks,” in British Machine Vision Conference 2016.   British Machine Vision Association, 2016.
  44. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.
  45. Z. Akata, S. Reed, D. Walter, H. Lee, and B. Schiele, “Evaluation of output embeddings for fine-grained image classification,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 2927–2936.
  46. Z. Yang, T. Luo, D. Wang, Z. Hu, J. Gao, and L. Wang, “Learning to navigate for fine-grained classification,” in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 420–435.
  47. A. Dubey, O. Gupta, R. Raskar, and N. Naik, “Maximum-entropy fine grained classification,” Advances in neural information processing systems, vol. 31, 2018.
  48. T. Syeda-Mahmood, K. C. Wong, Y. Gur, J. T. Wu, A. Jadhav, S. Kashyap, A. Karargyris, A. Pillai, A. Sharma, A. B. Syed et al., “Chest x-ray report generation through fine-grained label learning,” in International Conference on Medical Image Computing and Computer-Assisted Intervention.   Springer, 2020, pp. 561–571.
  49. Y. Zhu, X. Deng, and S. Newsam, “Fine-grained land use classification at the city scale using ground-level images,” IEEE Transactions on Multimedia, vol. 21, no. 7, pp. 1825–1838, 2019.
  50. J.-C. Su, Z. Cheng, and S. Maji, “A realistic evaluation of semi-supervised learning for fine-grained classification,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 12 966–12 975.
  51. J.-C. Su and S. Maji, “The semi-supervised inaturalist-aves challenge at fgvc7 workshop,” arXiv preprint arXiv:2103.06937, 2021.
  52. 2018 FGVCx Fungi Classification Challenge, https://github.com/visipedia/fgvcx_fungi_comp.
  53. P. Cascante-Bonilla, F. Tan, Y. Qi, and V. Ordonez, “Curriculum labeling: Revisiting pseudo-labeling for semi-supervised learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 8, 2021, pp. 6912–6920.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now