Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Methodology-Oriented Study of Catastrophic Forgetting in Incremental Deep Neural Networks (2405.08015v1)

Published 11 May 2024 in cs.LG and cs.AI

Abstract: Human being and different species of animals having the skills to gather, transferring knowledge, processing, fine-tune and generating information throughout their lifetime. The ability of learning throughout their lifespan is referred as continuous learning which is using neurocognition mechanism. Consequently, in real world computational system of incremental learning autonomous agents also needs such continuous learning mechanism which provide retrieval of information and long-term memory consolidation. However, the main challenge in artificial intelligence is that the incremental learning of the autonomous agent when new data confronted. In such scenarios, the main concern is catastrophic forgetting(CF), i.e., while learning the sequentially, neural network underfits the old data when it confronted with new data. To tackle this CF problem many numerous studied have been proposed, however it is very difficult to compare their performance due to dissimilarity in their evaluation mechanism. Here we focus on the comparison of all algorithms which are having similar type of evaluation mechanism. Here we are comparing three types of incremental learning methods: (1) Exemplar based methods, (2) Memory based methods, and (3) Network based method. In this survey paper, methodology oriented study for catastrophic forgetting in incremental deep neural network is addressed. Furthermore, it contains the mathematical overview of impact-full methods which can be help researchers to deal with CF.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (76)
  1. Uncertainty-based continual learning with adaptive regularization. Advances in neural information processing systems, 32, 2019.
  2. Expert gate: Lifelong learning with a network of experts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3366–3375, 2017.
  3. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), pages 139–154, 2018.
  4. Task-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11254–11263, 2019a.
  5. Online continual learning with no task boundaries. arXiv preprint arXiv:1903.08671, 2019b.
  6. Pseudo-recursal: Solving the catastrophic forgetting problem in deep neural networks. arxiv 2018. arXiv preprint arXiv:1802.03875, 2, 1802.
  7. E. Belouadah and A. Popescu. Deesil: Deep-shallow incremental learning. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0–0, 2018.
  8. A comprehensive study of class incremental learning algorithms for visual tasks. Neural Networks, 135:38–54, 2021.
  9. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV), pages 532–547, 2018a.
  10. Efficient lifelong learning with a-gem. arXiv preprint arXiv:1812.00420, 2018b.
  11. Continual learning with tiny episodic memories. 2019.
  12. Gan memory with no forgetting. Advances in Neural Information Processing Systems, 33:16481–16494, 2020.
  13. M. De Lange and T. Tuytelaars. Continual prototype evolution: Learning online from non-stationary data streams. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8250–8259, 2021.
  14. Adversarial continual learning. In European Conference on Computer Vision, pages 386–402. Springer, 2020.
  15. M. Elsayed and A. R. Mahmood. Addressing loss of plasticity and catastrophic forgetting in continual learning. arXiv preprint arXiv:2404.00781, 2024.
  16. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pages 1407–1416. PMLR, 2018.
  17. S. Farquhar and Y. Gal. Towards robust evaluations of continual learning. arXiv preprint arXiv:1805.09733, 2018.
  18. Multi-layered gradient boosting decision trees. Advances in neural information processing systems, 31, 2018.
  19. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734, 2017.
  20. Generative adversarial nets. Advances in neural information processing systems, 27, 2014a.
  21. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211, 2013.
  22. Qualitatively characterizing neural network optimization problems. arXiv preprint arXiv:1412.6544, 2014b.
  23. Exemplar-supported generative reproduction for class incremental learning. In BMVC, page 98, 2018.
  24. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2(7), 2015.
  25. Lifelong learning via progressive distillation and retrospection. In Proceedings of the European Conference on Computer Vision (ECCV), pages 437–452, 2018.
  26. D. Isele and A. Cosgun. Selective experience replay for lifelong learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  27. Less-forgetting learning in deep neural networks. arXiv preprint arXiv:1607.00122, 2016.
  28. Fine-tuning deep neural networks in continuous learning scenarios. In Asian Conference on Computer Vision, pages 588–605. Springer, 2016.
  29. R. Kemker and C. Kanan. Fearnet: Brain-inspired model for incremental learning. arXiv preprint arXiv:1711.10563, 2017.
  30. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  31. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84–90, 2017.
  32. Unsupervised model personalization while preserving privacy and scalability: An open problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14463–14472, 2020.
  33. Continual classification learning using generative models. arXiv preprint arXiv:1810.10612, 2018.
  34. Overcoming catastrophic forgetting by incremental moment matching. Advances in neural information processing systems, 30, 2017.
  35. Z. Li and D. Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017.
  36. Rotate your networks: Better weight consolidation and less catastrophic forgetting. In 2018 24th International Conference on Pattern Recognition (ICPR), pages 2262–2268. IEEE, 2018.
  37. Tail: Task-specific adapters for imitation learning with large pretrained models. arXiv preprint arXiv:2310.05905, 2023.
  38. D. Lopez-Paz and M. Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
  39. A. Mallya and S. Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7765–7773, 2018.
  40. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In Proceedings of the European Conference on Computer Vision (ECCV), pages 67–82, 2018.
  41. Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization. Proceedings of the National Academy of Sciences, 115(44):E10467–E10475, 2018.
  42. Variational continual learning. arXiv preprint arXiv:1710.10628, 2017.
  43. Random path selection for incremental learning. Advances in Neural Information Processing Systems, 2019.
  44. Lifelong generative modeling. Neurocomputing, 404:381–400, 2020.
  45. Encoder based lifelong learning. In Proceedings of the IEEE International Conference on Computer Vision, pages 1320–1328, 2017.
  46. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001–2010, 2017.
  47. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
  48. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  49. A. Rosenfeld and J. K. Tsotsos. Incremental learning through deep adaptation. IEEE transactions on pattern analysis and machine intelligence, 42(3):651–663, 2018.
  50. S. Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017.
  51. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211–252, 2015.
  52. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016.
  53. Progress & compress: A scalable framework for continual learning. In International Conference on Machine Learning, pages 4528–4537. PMLR, 2018.
  54. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pages 4548–4557. PMLR, 2018.
  55. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
  56. Incremental learning of object detectors without catastrophic forgetting. In Proceedings of the IEEE international conference on computer vision, pages 3400–3409, 2017.
  57. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140–1144, 2018.
  58. The task rehearsal method of life-long learning: Overcoming impoverished data. In Conference of the Canadian Society for Computational Studies of Intelligence, pages 90–101. Springer, 2002.
  59. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  60. G. M. Van de Ven and A. S. Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734, 2019.
  61. A strategy for an uncompromising incremental learner. arXiv preprint arXiv:1705.00744, 2017.
  62. Task difficulty aware parameter allocation & regularization for lifelong learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7776–7785, 2023.
  63. Incremental classifier learning with generative adversarial networks. arXiv preprint arXiv:1802.00853, 2018.
  64. Large scale incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 374–382, 2019.
  65. J. Xu and Z. Zhu. Reinforced continual learning. Advances in Neural Information Processing Systems, 31, 2018.
  66. Complex object classification: A multi-modal multi-instance multi-label deep network with optimal transport. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2594–2603, 2018.
  67. Adaptive deep models for incremental learning: Considering capacity scalability and sustainability. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 74–82, 2019.
  68. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.01547, 2017.
  69. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pages 3987–3995. PMLR, 2017.
  70. Task agnostic continual learning using online variational bayes. arXiv preprint arXiv:1803.10123, 2018.
  71. Lifelong gan: Continual learning for conditional image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2759–2768, 2019.
  72. Piggyback gan: Efficient lifelong learning for image conditioned generation. In European Conference on Computer Vision, pages 397–413. Springer, 2020.
  73. Hyper-lifelonggan: scalable lifelong learning for image conditioned generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2246–2255, 2021.
  74. Class-incremental learning via deep model consolidation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1131–1140, 2020.
  75. Continual sequence generation with adaptive compositional modules. arXiv preprint arXiv:2203.10652, 2022.
  76. Memory efficient class-incremental learning for image classification. IEEE Transactions on Neural Networks and Learning Systems, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Ashutosh Kumar (53 papers)
  2. Sonali Agarwal (38 papers)
  3. D Jude Hemanth (1 paper)