Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Density Distribution-based Learning Framework for Addressing Online Continual Learning Challenges (2311.13623v1)

Published 22 Nov 2023 in cs.LG and cs.CV

Abstract: In this paper, we address the challenges of online Continual Learning (CL) by introducing a density distribution-based learning framework. CL, especially the Class Incremental Learning, enables adaptation to new test distributions while continuously learning from a single-pass training data stream, which is more in line with the practical application requirements of real-world scenarios. However, existing CL methods often suffer from catastrophic forgetting and higher computing costs due to complex algorithm designs, limiting their practical use. Our proposed framework overcomes these limitations by achieving superior average accuracy and time-space efficiency, bridging the performance gap between CL and classical machine learning. Specifically, we adopt an independent Generative Kernel Density Estimation (GKDE) model for each CL task. During the testing stage, the GKDEs utilize a self-reported max probability density value to determine which one is responsible for predicting incoming test instances. A GKDE-based learning objective can ensure that samples with the same label are grouped together, while dissimilar instances are pushed farther apart. Extensive experiments conducted on multiple CL datasets validate the effectiveness of our proposed framework. Our method outperforms popular CL approaches by a significant margin, while maintaining competitive time-space efficiency, making our framework suitable for real-world applications. Code will be available at https://github.com/xxxx/xxxx.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. Conditional channel gated networks for task-aware continual learning. In CVPR, pages 3931–3940, 2020.
  2. Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems, 2019a.
  3. Gradient based sample selection for online continual learning. Advances in Neural Information Processing Systems, 32, 2019b.
  4. Dark experience for general continual learning: a strong, simple baseline. Advances in Neural Information Processing Systems, 33:15920–15930, 2020.
  5. New insights on reducing abrupt representation change in online continual learning. arXiv:2203.03798, 2022.
  6. Co22{}^{\mbox{2}}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPTl: Contrastive continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9516–9525, 2021.
  7. Efficient lifelong learning with A-GEM. arXiv:1812.00420, 2018.
  8. On tiny episodic memories in continual learning. arXiv:1902.10486, 2019.
  9. Online continual learning from imbalanced data. In International Conference on Machine Learning, pages 1952–1961, 2020.
  10. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(11), 2008.
  11. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv:2010.11929, 2020.
  12. PathNet: Evolution channels gradient descent in super neural networks. arXiv:1701.08734, 2017.
  13. Self-supervised models are continual learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9621–9630, 2022.
  14. Robert M French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4):128–135, 1999.
  15. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv:1312.6211, 2013.
  16. Not just selection, but exploration: Online class-incremental continual learning via dual view consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7442–7451, 2022.
  17. Online continual learning through mutual information maximization. International Conference on Machine Learning, pages 8109–8126, 2022.
  18. Exemplar-free online continual learning. In 2022 IEEE International Conference on Image Processing (ICIP), pages 541–545, 2022.
  19. Incremental learning in online scenario. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13926–13935, 2020.
  20. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
  21. Posterior meta-replay for continual learning. NeurIPS, 34, 2021.
  22. Compacting, picking and growing for unforgetting continual learning. In NeurIPS, 2019.
  23. Learning multiple layers of features from tiny images. Technical Report TR-2009, University of Toronto, Toronto, 2009.
  24. Continual prototype evolution: Learning online from non-stationary data streams. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8250–8259, 2021.
  25. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3366–3385, 2021.
  26. Tiny ImageNet visual recognition challenge. CS 231N, 7(7):3, 2015.
  27. Uncertainty modeling for out-of-distribution generalization. In International Conference on Learning Representations, 2022.
  28. Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 3589–3599, 2021.
  29. Online continual learning in image classification: An empirical survey. Neurocomputing, 469:28–51, 2022.
  30. PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning. arXiv preprint arXiv:1711.05769, 2017.
  31. Class-incremental learning: survey and performance evaluation on image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  32. Continual lifelong learning with neural networks: A review. Neural networks, 113:54–71, 2019.
  33. Gdumb: A simple approach that questions our progress in continual learning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 524–540, 2020.
  34. itaml: An incremental task-agnostic meta-learning approach. In CVPR, 2020.
  35. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001–2010, 2017.
  36. Progressive neural networks. arXiv:1606.04671, 2016.
  37. Overcoming catastrophic forgetting with hard attention to the task. In ICML, 2018.
  38. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pages 4548–4557, 2018.
  39. Online class-incremental continual learning with adversarial shapley value. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9630–9638, 2021.
  40. Continual learning with hypernetworks. ICLR, 2020.
  41. Online prototype learning for online continual learning. In ICCV, 2023.
  42. Supermasks in superposition. In NeurIPS, 2020.
  43. Delving deep into the generalization of vision transformers under distribution shifts. CoRR, abs/2106.07617, 2021a.
  44. Deep stable learning for out-of-distribution generalization. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5368–5378, 2021b.
  45. Prototype augmentation and self-supervision for incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5871–5880, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Shilin Zhang (8 papers)
  2. Jiahui Wang (46 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now