Papers
Topics
Authors
Recent
2000 character limit reached

Preserving Linear Separability in Continual Learning by Backward Feature Projection (2303.14595v3)

Published 26 Mar 2023 in cs.LG, cs.AI, and cs.CV

Abstract: Catastrophic forgetting has been a major challenge in continual learning, where the model needs to learn new tasks with limited or no access to data from previously seen tasks. To tackle this challenge, methods based on knowledge distillation in feature space have been proposed and shown to reduce forgetting. However, most feature distillation methods directly constrain the new features to match the old ones, overlooking the need for plasticity. To achieve a better stability-plasticity trade-off, we propose Backward Feature Projection (BFP), a method for continual learning that allows the new features to change up to a learnable linear transformation of the old features. BFP preserves the linear separability of the old classes while allowing the emergence of new feature directions to accommodate new classes. BFP can be integrated with existing experience replay methods and boost performance by a significant margin. We also demonstrate that BFP helps learn a better representation space, in which linear separability is well preserved during continual learning and linear probing achieves high classification accuracy. The code can be found at https://github.com/rvl-lab-utoronto/BFP

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. Memory aware synapses: Learning what (not) to forget. In ECCV, 2018.
  2. Online continual learning with maximal interfered retrieval. In NeurIPS, 2019.
  3. Learning fast, learning slow: A general continual learning method based on complementary learning system. In ICLR, 2022.
  4. Contrastive supervised distillation for continual representation learning. In ICIAP, pages 597–609. Springer, 2022.
  5. Measuring and regularizing networks in function space. In ICLR, 2019.
  6. Continual lifelong learning in natural language processing: A survey. arXiv preprint arXiv:2012.09823, 2020.
  7. Class-incremental continual learning into the extended der-verse. arXiv preprint arXiv:2201.00766, 2022.
  8. Dark experience for general continual learning: a strong, simple baseline. NeurIPS, 2020.
  9. Rethinking experience replay: a bag of tricks for continual learning. In ICPR, 2021.
  10. New insights on reducing abrupt representation change in online continual learning. In ICLR, 2022.
  11. Co2l: Contrastive continual learning. In ICCV, 2021.
  12. Riemannian walk for incremental learning: understanding forgetting and intransigence. In ECCV, 2018.
  13. Efficient lifelong learning with a-gem. In ICLR, 2019.
  14. Continual learning with tiny episodic memories. In ICML, 2019.
  15. Probing representation forgetting in supervised and unsupervised continual learning. In CVPR, 2022.
  16. Learning without memorizing. In CVPR, 2019.
  17. Efficient parametric approximations of neural network function space distance. In ICML, 2023.
  18. A theoretical analysis of catastrophic forgetting through the ntk overlap matrix. In AISTATS, pages 1072–1080. PMLR, 2021.
  19. Podnet: Pooled outputs distillation for small-tasks incremental learning. In ECCV, pages 86–102. Springer, 2020.
  20. Representational drift: Emerging theories for continual learning and experimental future directions. Current Opinion in Neurobiology, 76:102609, 2022.
  21. Self-supervised models are continual learners. In CVPR, 2022.
  22. Continually learning self-supervised representations with projected functional regularization. In CVPR Workshop, 2022.
  23. Deep residual learning for image recognition. In CVPR, 2016.
  24. Learning a unified classifier incrementally via rebalancing. In ICCV, 2019.
  25. Towards continual knowledge learning of language models. arXiv preprint arXiv:2110.03215, 2021.
  26. Less-forgetting learning in deep neural networks. arXiv preprint arXiv:1607.00122, 2016.
  27. Class-incremental learning by knowledge distillation with adaptive feature consolidation. In CVPR, 2022.
  28. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  29. Similarity of neural network representations revisited. In ICML, pages 3519–3529. PMLR, 2019.
  30. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
  31. Overcoming catastrophic forgetting by incremental moment matching. In NeurIPS, 2017.
  32. Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges. Information Fusion, 2020.
  33. Learning without forgetting. PAMI, 2017.
  34. Generative feature replay for class-incremental learning. In CVPR Workshops, 2020.
  35. Gradient episodic memory for continual learning. In NeurIPS, 2017.
  36. Packnet: Adding multiple tasks to a single network by iterative pruning. In CVPR, 2018.
  37. Class-incremental learning: survey and performance evaluation on image classification. arXiv preprint arXiv:2010.15277, 2020.
  38. Class-incremental learning: survey and performance evaluation on image classification. PAMI, 2022.
  39. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier, 1989.
  40. Karl Pearson. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science, 2(11):559–572, 1901.
  41. Anatomy of catastrophic forgetting: Hidden representations and task semantics. arXiv preprint arXiv:2007.07400, 2020.
  42. icarl: Incremental classifier and representation learning. In CVPR, 2017.
  43. Learning to learn without forgetting by maximizing transfer and minimizing interference. In ICLR, 2019.
  44. Progressive neural networks. arXiv, 2016.
  45. Space: Structured compression and sharing of representational space for continual learning. IEEE Access, 9:150480–150494, 2021.
  46. Continual learning with deep generative replay. In NeurIPS, 2017.
  47. On learning the geodesic path for incremental learning. In CVPR, pages 1591–1600, 2021.
  48. Stanford. Tiny imagenet challenge, cs231n course., CS231N.
  49. Gido M Van de Ven and Andreas S Tolias. Generative replay with feedback connections as a general strategy for continual learning. arXiv preprint arXiv:1809.10635, 2018.
  50. Wanderlust: Online continual object detection in the real world. In ICCV, pages 10829–10838, 2021.
  51. Large scale incremental learning. In CVPR, 2019.
  52. Der: Dynamically expandable representation for class incremental learning. In CVPR, pages 3014–3023, 2021.
  53. Semantic drift compensation for class-incremental learning. In CVPR, 2020.
  54. Continual learning through synaptic intelligence. In ICML, 2017.
  55. Monitoring multimode processes: A modified pca algorithm with continual learning ability. Journal of Process Control, 103:76–86, 2021.
  56. Feature forgetting in continual representation learning. arXiv preprint arXiv:2205.13359, 2022.
  57. Class-incremental learning via dual augmentation. NeurIPS, 34:14306–14318, 2021.
Citations (8)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Video Overview

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up