Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

The Importance of Robust Features in Mitigating Catastrophic Forgetting (2306.17091v1)

Published 29 Jun 2023 in cs.LG and cs.CV

Abstract: Continual learning (CL) is an approach to address catastrophic forgetting, which refers to forgetting previously learned knowledge by neural networks when trained on new tasks or data distributions. The adversarial robustness has decomposed features into robust and non-robust types and demonstrated that models trained on robust features significantly enhance adversarial robustness. However, no study has been conducted on the efficacy of robust features from the lens of the CL model in mitigating catastrophic forgetting in CL. In this paper, we introduce the CL robust dataset and train four baseline models on both the standard and CL robust datasets. Our results demonstrate that the CL models trained on the CL robust dataset experienced less catastrophic forgetting of the previously learned tasks than when trained on the standard dataset. Our observations highlight the significance of the features provided to the underlying CL models, showing that CL robust features can alleviate catastrophic forgetting.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366–3385, 2021.
  2. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier, 1989.
  3. The stability-plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects, 2013.
  4. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), pages 139–154, 2018.
  5. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  6. Overcoming catastrophic forgetting by incremental moment matching. Advances in neural information processing systems, 30, 2017.
  7. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pages 3987–3995. PMLR, 2017.
  8. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7765–7773, 2018.
  9. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734, 2017.
  10. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.01547, 2017.
  11. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In International Conference on Machine Learning, pages 3925–3934. PMLR, 2019.
  12. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016.
  13. Dytox: Transformers for continual learning with dynamic token expansion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9285–9295, 2022.
  14. Continual learning via neural pruning. arXiv preprint arXiv:1903.04476, 2019.
  15. Compacting, picking and growing for unforgetting continual learning. Advances in Neural Information Processing Systems, 32, 2019.
  16. Routing networks with co-training for continual learning. arXiv preprint arXiv:2009.04381, 2020.
  17. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. arXiv preprint arXiv:2002.06715, 2020.
  18. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
  19. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486, 2019.
  20. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001–2010, 2017.
  21. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
  22. Gradient based sample selection for online continual learning. Advances in neural information processing systems, 32, 2019.
  23. Selective experience replay for lifelong learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  24. Generative feature replay for class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 226–227, 2020.
  25. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
  26. Pseudo-recursal: Solving the catastrophic forgetting problem in deep neural networks. arXiv preprint arXiv:1802.03875, 2018.
  27. Continual classification learning using generative models. arXiv preprint arXiv:1810.10612, 2018.
  28. Class-incremental learning by knowledge distillation with adaptive feature consolidation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16071–16080, 2022.
  29. Less-forgetting learning in deep neural networks. arXiv preprint arXiv:1607.00122, 2016.
  30. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017.
  31. Distilled replay: Overcoming forgetting through synthetic samples. In Continual Semi-Supervised Learning: First International Workshop, CSSL 2021, Virtual Event, August 19–20, 2021, Revised Selected Papers, pages 104–117. Springer, 2022.
  32. Remember the past: Distilling datasets into addressable memories for neural networks. arXiv preprint arXiv:2206.02916, 2022.
  33. Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32, 2019.
  34. Adversarial robustness as a prior for learned representations. arXiv preprint arXiv:1906.00945, 2019.
  35. Brain tumor segmentation and surveillance with deep artificial neural networks. Deep Learning for Biomedical Data Analysis: Techniques, Approaches, and Applications, pages 311–350, 2021.
  36. Explainable ai: Rotorcraft attitude prediction. In Vertical Flight Society’s 76th Annual Forum and Technology Display, 2020.
  37. Exploring robust architectures for deep artificial neural networks. Communications Engineering, 1(1):46, 2022.
  38. Multimodal data integration for oncology in the era of deep neural networks: A review. arXiv preprint arXiv:2303.06471, 2023.
  39. Deep ensemble for rotorcraft attitude prediction. In 77th Annual Vertical Flight Society Forum and Technology Display: The Future of Vertical Flight, FORUM 2021. Vertical Flight Society, 2021.
  40. Semantic drift compensation for class-incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6982–6991, 2020.
  41. Timothée Lesort. Continual feature selection: Spurious features in continual learning. arXiv preprint arXiv:2203.01012, 2022.
  42. Learning multiple layers of features from tiny images. 2009.
  43. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3014–3023, 2021.
  44. Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734, 2019.
  45. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XX 16, pages 86–102. Springer, 2020.
  46. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  47. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010.
  48. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016.
  49. Adversarially robust continual learning. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Hikmat Khan (7 papers)
  2. Nidhal C. Bouaynaya (8 papers)
  3. Ghulam Rasool (32 papers)
Citations (6)
View on Semantic Scholar