Papers
Topics
Authors
Recent
Search
2000 character limit reached

CUCL: Codebook for Unsupervised Continual Learning

Published 25 Nov 2023 in cs.CV | (2311.14911v1)

Abstract: The focus of this study is on Unsupervised Continual Learning (UCL), as it presents an alternative to Supervised Continual Learning which needs high-quality manual labeled data. The experiments under the UCL paradigm indicate a phenomenon where the results on the first few tasks are suboptimal. This phenomenon can render the model inappropriate for practical applications. To address this issue, after analyzing the phenomenon and identifying the lack of diversity as a vital factor, we propose a method named Codebook for Unsupervised Continual Learning (CUCL) which promotes the model to learn discriminative features to complete the class boundary. Specifically, we first introduce a Product Quantization to inject diversity into the representation and apply a cross quantized contrastive loss between the original representation and the quantized one to capture discriminative information. Then, based on the quantizer, we propose an effective Codebook Rehearsal to address catastrophic forgetting. This study involves conducting extensive experiments on CIFAR100, TinyImageNet, and MiniImageNet benchmark datasets. Our method significantly boosts the performances of supervised and unsupervised methods. For instance, on TinyImageNet, our method led to a relative improvement of 12.76% and 7% when compared with Simsiam and BYOL, respectively.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (41)
  1. Memory Aware Synapses: Learning What (not) to Forget. In ECCV, Vol. 11207. 144–161.
  2. Ali Ayub and Alan R. Wagner. 2021. EEC: Learning to Encode and Regenerate Images for Continual Learning. In ICLR.
  3. Signature Verification Using a Siamese Time Delay Neural Network. In NeurIPS. 737–744.
  4. Dark Experience for General Continual Learning: a Strong, Simple Baseline. In NeurIPS.
  5. Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. In NeurIPS.
  6. Efficient Lifelong Learning with A-GEM. In ICLR.
  7. Class Gradient Projection For Continual Learning. In ACM MM. 5575–5583.
  8. A Simple Framework for Contrastive Learning of Visual Representations. In ICML, Vol. 119. 1597–1607.
  9. Xinlei Chen and Kaiming He. 2021. Exploring Simple Siamese Representation Learning. In CVPR. 15750–15758.
  10. An Empirical Study of Training Self-Supervised Vision Transformers. In ICCV. 9620–9629.
  11. ImageNet: A large-scale hierarchical image database. In CVPR. 248–255.
  12. Examine before You Answer: Multi-task Learning with Adaptive-attentions for Multiple-choice VQA. In ACM MM. 1742–1750.
  13. Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning. In NeurIPS.
  14. Masked Autoencoders Are Scalable Vision Learners. In CVPR. 15979–15988.
  15. Momentum Contrast for Unsupervised Visual Representation Learning. In CVPR. 9726–9735.
  16. Deep Residual Learning for Image Recognition. In CVPR. 770–778.
  17. Learning deep representations by mutual information estimation and maximization. In ICLR.
  18. Young Kyun Jang and Nam Ik Cho. 2021. Self-supervised Product Quantization for Deep Unsupervised Image Retrieval. In ICCV. 12065–12074.
  19. Overcoming catastrophic forgetting in neural networks. CoRR abs/1612.00796 (2016).
  20. Learning multiple layers of features from tiny images. (2009).
  21. Learnable Aggregating Net with Diversity Learning for Video Question Answering. In ACM MM. 1166–1174.
  22. Zhizhong Li and Derek Hoiem. 2018. Learning without Forgetting. TPAMI 40 (2018), 2935–2947.
  23. David Lopez-Paz and Marc’Aurelio Ranzato. 2017. Gradient Episodic Memory for Continual Learning. In NeurIPS. 6467–6476.
  24. Adaptive Fine-Grained Predicates Learning for Scene Graph Generation. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).
  25. Representational Continuity for Unsupervised Continual Learning. In ICLR.
  26. Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation. Vol. 24. 109–165.
  27. Continual Unsupervised Representation Learning. In NeurIPS. 7645–7655.
  28. iCaRL: Incremental Classifier and Representation Learning. In CVPR. 5533–5542.
  29. Mark B. Ring. 1998. Child: A First Step Towards Continual Learning. In Learning to Learn. 261–292.
  30. Progressive Neural Networks. CoRR abs/1606.04671 (2016).
  31. Overcoming Catastrophic Forgetting with Hard Attention to the Task. In ICML, Vol. 80. 4555–4564.
  32. Memory-based Parameter Adaptation. In ICLR.
  33. Contrastive Multiview Coding. In ECCV, Vol. 12356. 776–794.
  34. Efficient Continual Learning with Modular Networks and Task-Driven Priors. In ICLR.
  35. Matching Networks for One Shot Learning. In NeurIPS. 3630–3638.
  36. Unsupervised Feature Learning via Non-Parametric Instance Discrimination. In CVPR. 3733–3742.
  37. Unsupervised Embedding Learning via Invariant and Spreading Instance Feature. In CVPR. 6210–6219.
  38. Product Quantization Network for Fast Visual Search. IJCV 128 (2020), 2325–2343.
  39. Barlow Twins: Self-Supervised Learning via Redundancy Reduction. In ICML, Vol. 139. 12310–12320.
  40. S2 Transformer for Image Captioning. In IJCAI, Luc De Raedt (Ed.). 1608–1614.
  41. Continual Learning Through Synaptic Intelligence. In ICML, Vol. 70. 3987–3995.
Citations (5)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now