Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Bayesian Learning-driven Prototypical Contrastive Loss for Class-Incremental Learning (2405.11067v3)

Published 17 May 2024 in cs.CV and cs.AI

Abstract: The primary objective of methods in continual learning is to learn tasks in a sequential manner over time (sometimes from a stream of data), while mitigating the detrimental phenomenon of catastrophic forgetting. This paper proposes a method to learn an effective representation between previous and newly encountered class prototypes. We propose a prototypical network with a Bayesian learning-driven contrastive loss (BLCL), tailored specifically for class-incremental learning scenarios. We introduce a contrastive loss that incorporates novel classes into the latent representation by reducing intra-class and increasing inter-class distance. Our approach dynamically adapts the balance between the cross-entropy and contrastive loss functions with a Bayesian learning technique. Experimental results conducted on the CIFAR-10, CIFAR-100, and ImageNet100 datasets for image classification and images of a GNSS-based dataset for interference classification validate the efficacy of our method, showcasing its superiority over existing state-of-the-art approaches. Git: https://gitlab.cc-asp.fraunhofer.de/darcy_gnss/gnss_class_incremental_learning

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Rainbow Memory: Continual Learning with a Memory of Diverse Samples. In IEEE/CVF Intl. Conf. on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, June 2021.
  2. Online Continual Learning on a Contaminated Data Stream with Blurry Task Boundaries. In IEEE/CVF Intl. Conf. on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, June 2022.
  3. Continual Lifelong Learning in Natural Language Processing: A Survey. In Proc. of the Intl. Conf. on Computational Linguistics, pages 6523–6541, Barcelona, Spain, Dec. 2020.
  4. Multimodal Learning for Reliable Interference Classification in GNSS Signals. In Proc. of the Intl. Technical Meeting of the Satellite Division of the Institute of Navigation (ION GNSS+), pages 3210–3234, Denver, CO, Sept. 2022.
  5. A Dendrite Method for Cluster Analysis. In Communications in Statistics, volume 3(1), pages 1–28, Sept. 1972.
  6. Co2⁢LsuperscriptCo2L\text{Co}^{2}\text{L}Co start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT L: Contrastive Continual Learning. In Intl. Conf. on Computer Vision (ICCV), Montreal, QC, Oct. 2021.
  7. Sy-CON: Symmetric Contrastive Loss for Continual Self-Supervised Representation Learning. In arXiv preprint arXiv:2306.05101, June 2023.
  8. A Simple Framework for Contrastive Learning of Visual Representations. In Intl. Conf. on Machine Learning (ICML), volume 149, pages 1597–1607, July 2020.
  9. Beyond Triplet Loss: A Deep Quadruplet Network for Person Re-identification. In IEEE/CVF Intl. Conf. on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, July 2017.
  10. Learning with Augmented Class by Exploiting Unlabeled Data. In Proc. of the Intl. Conf. on Artificial Intelligence (AAAI), volume 28, 2014.
  11. A Cluster Separation Measure. In IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI), volume PAMI-1(2), pages 224–227, Apr. 1979.
  12. ImageNet: A Large-scale Hierarchical Image Database. In IEEE/CVF Intl. Conf. on Computer Vision and Pattern Recognition (CVPR), Miami, FL, June 2009.
  13. A Theoretically Sound Upper Bound on the Triplet Loss for Improving the Efficiency of Deep Distance Metric Learning. In IEEE/CVF Intl. Conf. on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, June 2019.
  14. DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition. In Journal for Machine Learning Research (JMLR), volume 32(1), pages 647–655, 2014.
  15. PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning. In IEEE/CVF Europ. Conf. on Computer Vision (ECCV), volume 12365, pages 86–102, Nov. 2020.
  16. DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion. In IEEE/CVF Intl. Conf. on Computer Vision and Pattern Recognition (CVPR), pages 9285–9295, 2022.
  17. Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. In IEEE/CVF Intl. Conf. on Computer Vision and Pattern Recognition (CVPR), pages 580–587, 2014.
  18. On Clustering Validation Techniques. In Journal of Intelligent Information Systems, volume 17, pages 107–145, Dec. 2001.
  19. Bayesian Graph Contrastive Learning. In arXiv preprint arXiv:2112.07823, Aug. 2022.
  20. Deep Residual Learning for Image Recognition. In IEEE/CVF Intl. Conf. on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, June 2016.
  21. MHTN: Modal-Adversarial Hybrid Transfer Network for Cross-Modal Retrieval. In Trans. on Cybernetics, volume 50(3), pages 1047–1059, 2020.
  22. CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks. In Intl. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 6871–6883, Nov. 2021.
  23. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? In Advances in Neural Information Processing Systems (NIPS), Long Beach, CA, 2017.
  24. Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics. In IEEE/CVF Intl. Conf. on Computer Vision and Pattern Recognition (CVPR), 2018.
  25. Overcoming Catastrophic Forgetting in Neural Networks. In Applied Mathematics, volume 114(13), pages 3521–3526, Mar. 2017.
  26. Uncertainty-aware Evaluation of Time-Series Classification for Online Handwriting Recognition with Domain Shift. In IJCAI-ECAI Intl. Workshop on Spatio-Temporal Reasoning and Learning (STRL), volume 3190, Vienna, Austria, July 2022.
  27. Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inference. In Intl. Conf. on Learning Representations (ICLR), 2022.
  28. Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. 2009.
  29. From N to N+1: Multiclass Transfer Incremental Learning. In IEEE/CVF Intl. Conf. on Computer Vision and Pattern Recognition (CVPR), Portland, OR, June 2013.
  30. A Continual Learning Survey: Defying Forgetting in Classification Tasks. In IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI), volume 44(7), July 2022.
  31. Catastrophic Interference in Neural Networks: Causes, Solutions, and Data. In F. N. Dempster & C. J. Brainerd (Eds.), Interference and Inhibition in Cognition, pages 329–361, 1995.
  32. Learning Without Forgetting. In IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI), volume 40(12), pages 2935–2947, Dec. 2018.
  33. Bayesian Contrastive Learning with Manifold Regularization for Self-Supervised Skeleton Based Action Recognition. In IEEE Intl. Symposium on Circuits and Systems (ISCAS), Monterey, CA, May 2023.
  34. Bayesian Self-Supervised Contrastive Learning. In arXiv preprint arXiv:2301.11673, Jan. 2024.
  35. RMM: Reinforced Memory Management for Class-Incremental Learning. In Advances in Neural Information Processing Systems (NIPS), 2021.
  36. Gradient Episodic Memory for Continual Learning. In Advances in Neural Information Processing Systems (NIPS), Long Beach, CA, 2017.
  37. Online Continual Learning in Image Classification: An Empirical Survey. In arXiv preprint arXiv:2101.10423, Oct. 2021.
  38. Class-Incremental Learning: Survey and Performance Evaluation on Image Classification. In IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI), volume 45(5), May 2023.
  39. Few-Shot Learning with Uncertainty-based Quadruplet Selection for Interference Classification in GNSS Data. In Intl. Conf. on Localization and GNSS (ICL GNSS), Feb. 2024.
  40. Auxiliary Cross-Modal Representation Learning with Triplet Loss Functions for Online Handwriting Recognition. In IEEE Access, volume 11, pages 94148–94172, Aug. 2023.
  41. Machine Learning-assisted GNSS Interference Monitoring Through Crowdsourcing. In Proc. of the Intl. Technical Meeting of the Satellite Division of the Institute of Navigation (ION GNSS+), pages 1151–1175, Denver, CO, Sept. 2022.
  42. A New Approach to Cross-Modal Multimedia Retrieval. In Proc. of the ACM Intl. Conf. on Multimedia (ACMMM), pages 251–260, Oct. 2010.
  43. R. Ratcliff. Connectionist Models of Recognition Memory: Constraints Imposed by Learning and Forgetting Functions. In Psychological Review, volume 97(2), pages 285–308, Apr. 1997.
  44. iCaRL: Incremental Classifier and Representation Learning. In IEEE Intl. Conf. on Computer Vision and Pattern Recognition (CVPR), pages 2011–2010, Honolulu, HI, July 2017.
  45. Experience Replay for Continual Learning. In Advances in Neural Information Processing Systems (NIPS), 2019.
  46. Adversarial Representation Learning for Text-to-Image Matching. In IEEE/CVF Intl. Conf. on Computer Vision (ICCV), pages 5814–5824, Seoul, Korea, 2019.
  47. FaceNet: A Unified Embedding for Face Recognition and Clustering. In IEEE/CVF Intl. Conf. on Computer Vision and Pattern Recognition (CVPR), Boston, MA, June 2015.
  48. Prototypical Networks for Few-shot Learning. In Advances in Neural Information Processing Systems (NIPS), pages 4080–4090, Dec. 2017.
  49. Contrastive Domain Adaptation. In IEEE/CVF Intl. Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW), Nashville, TN, June 2021.
  50. Brain-inspired Replay for Continual Learning with Artificial Neural Networks. In Nature Communications, volume 11(4069), Aug. 2020.
  51. Laurens van der Maaten and Geoffrey Hinton. Visualizing Data Using t-SNE. In Journal for Machine Learning Research (JMLR), volume 9(86), pages 2579–2605, Nov. 2008.
  52. Low-Cost COTS GNSS Interference Monitoring, Detection, and Classification System. In MDPI Sensors, volume 23(7), 3452, Mar. 2023.
  53. FOSTER: Feature Boosting and Compression for Class-Incremental Learning. In IEEE/CVF Europ. Conf. on Computer Vision (ECCV), volume 13685, pages 398–414, Oct. 2022.
  54. Learning to Prompt for Continual Learning. In IEEE/CVF Intl. Conf. on Computer Vision and Pattern Recognition (CVPR), pages 139–149, 2022.
  55. Cross-Modal Retrieval with CNN Visual Features: A New Baseline. In Trans. on Cybernetics, volume 47(2), pages 449–460, Mar. 2016.
  56. Large Scale Incremental Learning. In IEEE/CVF Intl. Conf. on Computer Vision and Pattern Recognition (CVPR), pages 374–382, 2019.
  57. General Incremental Learning with Domain-aware Categorical Representations. In IEEE/CVF Intl. Conf. on Computer Vision and Pattern Recognition (CVPR), pages 14351–14360, 2022.
  58. DER: Dynamically Expandable Representation for Class Incremental Learning. In IEEE/CVF Intl. Conf. on Computer Vision and Pattern Recognition (CVPR), pages 3014–3023, 2021.
  59. Maintaining Discrimination and Fairness in Class Incremental Learning. In IEEE/CVF Intl. Conf. on Computer Vision and Pattern Recognition (CVPR), pages 13205–13214, 2020.
  60. Deep Class-Incremental Learning: A Survey. In arXiv preprint arXiv:2302.03648, Feb. 2023.
  61. A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning. In arXiv preprint arXiv:2205.13218, Feb. 2023.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets