Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
12 tokens/sec
GPT-4o
12 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
37 tokens/sec
DeepSeek R1 via Azure Pro
33 tokens/sec
2000 character limit reached

Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning (2403.12030v1)

Published 18 Mar 2024 in cs.CV and cs.LG

Abstract: Class-Incremental Learning (CIL) requires a learning system to continually learn new classes without forgetting. Despite the strong performance of Pre-Trained Models (PTMs) in CIL, a critical issue persists: learning new classes often results in the overwriting of old ones. Excessive modification of the network causes forgetting, while minimal adjustments lead to an inadequate fit for new classes. As a result, it is desired to figure out a way of efficient model updating without harming former knowledge. In this paper, we propose ExpAndable Subspace Ensemble (EASE) for PTM-based CIL. To enable model updating without conflict, we train a distinct lightweight adapter module for each new task, aiming to create task-specific subspaces. These adapters span a high-dimensional feature space, enabling joint decision-making across multiple subspaces. As data evolves, the expanding subspaces render the old class classifiers incompatible with new-stage spaces. Correspondingly, we design a semantic-guided prototype complement strategy that synthesizes old classes' new features without using any old class instance. Extensive experiments on seven benchmark datasets verify EASE's state-of-the-art performance. Code is available at: https://github.com/sun-hailong/CVPR24-Ease

Definition Search Book Streamline Icon: https://streamlinehq.com
References (82)
  1. Memory aware synapses: Learning what (not) to forget. In ECCV, pages 139–154, 2018.
  2. Task-free continual learning. In CVPR, pages 11254–11263, 2019a.
  3. Gradient based sample selection for online continual learning. In NeurIPS, pages 11816–11825, 2019b.
  4. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. NeurIPS, 32, 2019.
  5. Il2m: Class incremental learning with dual memory. In ICCV, pages 583–592, 2019.
  6. Efficient lifelong learning with a-gem. In ICLR, 2018.
  7. Large-margin contrastive learning with distance polarization regularizer. In ICML, pages 1673–1683, 2021.
  8. Adaptformer: Adapting vision transformers for scalable visual recognition. In NeurIPS, 2022a.
  9. Learning contrastive embedding in low-dimensional space. NeurIPS, 35:6345–6357, 2022b.
  10. Dynamic residual classifier for class incremental learning. In ICCV, pages 18743–18752, 2023.
  11. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255, 2009.
  12. Learning without memorizing. In CVPR, pages 5138–5146, 2019.
  13. Federated class-incremental learning. In CVPR, pages 10164–10173, 2022.
  14. Federated incremental semantic segmentation. In CVPR, pages 3934–3943, 2023.
  15. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
  16. Podnet: Pooled outputs distillation for small-tasks incremental learning. In ECCV, pages 86–102, 2020.
  17. Dytox: Transformers for continual learning with dynamic token expansion. In CVPR, pages 9285–9295, 2022.
  18. Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128–135, 1999.
  19. Modeling time perception in rats: Evidence for catastrophic interference in animal learning. In Proceedings of the 21st Annual Conference of the Cognitive Science Conference, pages 173–178. Citeseer, 1999.
  20. R-DFCIL: relation-guided representation learning for data-free class incremental learning. In ECCV, pages 423–439, 2022.
  21. A unified continual learning framework with general parameter-efficient tuning. In ICCV, pages 11483–11493, 2023.
  22. Fecam: Exploiting the heterogeneity of class distributions in exemplar-free continual learning. NeurIPS, 36, 2023.
  23. Stephen T Grossberg. Studies of mind and brain: Neural principles of learning, perception, development, cognition, and motor control. Springer Science & Business Media, 2012.
  24. Pre-trained models: Past, present and future. AI Open, 2:225–250, 2021.
  25. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, pages 8340–8349, 2021a.
  26. Natural adversarial examples. In CVPR, pages 15262–15271, 2021b.
  27. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  28. Parameter-efficient transfer learning for nlp. In ICML, pages 2790–2799, 2019.
  29. Dense network expansion for class incremental learning. In CVPR, pages 11858–11867, 2023.
  30. Resolving task confusion in dynamic expansion architectures for class incremental learning. In AAAI, pages 908–916, 2023.
  31. Visual prompt tuning. In ECCV, pages 709–727, 2022.
  32. Generating instance-level prompts for rehearsal-free continual learning. In ICCV, pages 11847–11857, 2023.
  33. Leonid V Kantorovich. Mathematical methods of organizing and planning production. Management science, 6(4):366–422, 1960.
  34. Overcoming catastrophic forgetting in neural networks. PNAS, 114(13):3521–3526, 2017.
  35. Learning multiple layers of features from tiny images. Technical report, 2009.
  36. Learning without forgetting. TPAMI, 40(12):2935–2947, 2017.
  37. Mnemonics training: Multi-class incremental learning without forgetting. In CVPR, pages 12245–12254, 2020.
  38. Rmm: Reinforced memory management for class-incremental learning. NeurIPS, 34:3478–3490, 2021.
  39. Premonition: Using generative models to preempt future data changes in continual learning. arXiv preprint arXiv:2403.07356, 2024a.
  40. Ranpac: Random projections and pre-trained models for continual learning. NeurIPS, 36, 2024b.
  41. Rf-badge: Vital sign-based authentication via rfid tag array on badges. IEEE Transactions on Mobile Computing, 22(02):1170–1184, 2023.
  42. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, pages 8026–8037, 2019.
  43. Continual normalization: Rethinking batch normalization for online continual learning. In ICLR, 2022.
  44. Audio-visual class-incremental learning. In ICCV, pages 7799–7811, 2023.
  45. Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285, 1990.
  46. icarl: Incremental classifier and representation learning. In CVPR, pages 2001–2010, 2017.
  47. Mimicking the oracle: An initial phase decorrelation approach for class incremental learning. In CVPR, pages 16722–16731, 2022.
  48. On learning the geodesic path for incremental learning. In CVPR, pages 1591–1600, 2021.
  49. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In CVPR, pages 11909–11919, 2023.
  50. Prototypical networks for few-shot learning. In NIPS, pages 4080–4090, 2017.
  51. Pilot: A pre-trained model-based continual learning toolbox. arXiv preprint arXiv:2309.07117, 2023.
  52. Topology-preserving class-incremental learning. In ECCV, pages 254–270, 2020.
  53. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9(11), 2008.
  54. Pivot: Prompting for video continual learning. In CVPR, pages 24214–24223, 2023.
  55. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.
  56. Foster: Feature boosting and compression for class-incremental learning. In ECCV, pages 398–414, 2022a.
  57. BEEF: Bi-compatible class-incremental learning via energy-based expansion and fusion. In ICLR, 2023a.
  58. Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality. NeurIPS, 36, 2023b.
  59. Few-shot class-incremental learning via training-free prototype calibration. NeurIPS, 36, 2023c.
  60. S-prompts learning with pre-trained transformers: An occam’s razor for domain incremental learning. NeurIPS, 35:5682–5695, 2022b.
  61. Dualprompt: Complementary prompting for rehearsal-free continual learning. In ECCV, pages 631–648, 2022c.
  62. Learning to prompt for continual learning. In CVPR, pages 139–149, 2022d.
  63. Large scale incremental learning. In CVPR, pages 374–382, 2019.
  64. Der: Dynamically expandable representation for class incremental learning. In CVPR, pages 3014–3023, 2021.
  65. Rectify heterogeneous models with semantic mapping. In ICML, pages 5630–5639, 2018.
  66. Learning multiple local metrics: Global consideration helps. IEEE transactions on pattern analysis and machine intelligence, 42(7):1698–1712, 2019.
  67. Semantic drift compensation for class-incremental learning. In CVPR, pages 6982–6991, 2020.
  68. Continual learning through synaptic intelligence. In ICML, pages 3987–3995, 2017.
  69. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867, 2019.
  70. Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. In ICCV, pages 19148–19158, 2023.
  71. Class-incremental learning via deep model consolidation. In WACV, pages 1131–1140, 2020.
  72. Benchmarking omni-vision representation through the lens of visual realms. In ECCV, pages 594–611, 2022.
  73. Maintaining discrimination and fairness in class incremental learning. In CVPR, pages 13208–13217, 2020.
  74. Mgsvf: Multi-grained slow versus fast framework for few-shot class-incremental learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(3):1576–1588, 2021a.
  75. Memory-efficient class-incremental learning for image classification. IEEE Transactions on Neural Networks and Learning Systems, 33(10):5966–5977, 2021b.
  76. Pycil: a python toolbox for class-incremental learning. SCIENCE CHINA Information Sciences, 66(9):197101–, 2023a.
  77. A model or 603 exemplars: Towards memory-efficient class-incremental learning. In ICLR, 2023b.
  78. Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need. arXiv preprint arXiv:2303.07338, 2023c.
  79. Continual learning with pre-trained models: A survey. arXiv preprint arXiv:2401.16386, 2024.
  80. Prototype augmentation and self-supervision for incremental learning. In CVPR, pages 5871–5880, 2021.
  81. Acil: Analytic class-incremental learning with absolute memorization and privacy protection. NeurIPS, 35:11602–11614, 2022.
  82. Gkeal: Gaussian kernel embedded analytic learning for few-shot class incremental task. In CVPR, pages 7746–7755, 2023.
Citations (20)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now