Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

FOCIL: Finetune-and-Freeze for Online Class Incremental Learning by Training Randomly Pruned Sparse Experts (2403.14684v1)

Published 13 Mar 2024 in cs.CV and cs.LG

Abstract: Class incremental learning (CIL) in an online continual learning setting strives to acquire knowledge on a series of novel classes from a data stream, using each data point only once for training. This is more realistic compared to offline modes, where it is assumed that all data from novel class(es) is readily available. Current online CIL approaches store a subset of the previous data which creates heavy overhead costs in terms of both memory and computation, as well as privacy issues. In this paper, we propose a new online CIL approach called FOCIL. It fine-tunes the main architecture continually by training a randomly pruned sparse subnetwork for each task. Then, it freezes the trained connections to prevent forgetting. FOCIL also determines the sparsity level and learning rate per task adaptively and ensures (almost) zero forgetting across all tasks without storing any replay data. Experimental results on 10-Task CIFAR100, 20-Task CIFAR100, and 100-Task TinyImagenet, demonstrate that our method outperforms the SOTA by a large margin. The code is publicly available at https://github.com/muratonuryildirim/FOCIL.

View on arXiv
Definition Search Book Streamline Icon: https://streamlinehq.com
References (81)
  1. Expert gate: Lifelong learning with a network of experts. In CVPR, 2017.
  2. Memory aware synapses: Learning what (not) to forget. In ECCV, 2018.
  3. Online continual learning with maximal interfered retrieval. In NeurIPS, 2019a.
  4. Gradient based sample selection for online continual learning. In NeurIPS, 2019b.
  5. Learning fast, learning slow: A general continual learning method based on complementary learning system. In ICLR, 2022.
  6. Sparseness and expansion in sensory representations. Neuron, 83(5), 2014.
  7. Rainbow memory: Continual learning with a memory of diverse samples. In CVPR, 2021.
  8. Random search for hyper-parameter optimization. Journal of machine learning research, 13(2), 2012.
  9. Algorithms for hyper-parameter optimization. In NeurIPS, 2011.
  10. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In ECCV, 2018.
  11. Efficient lifelong learning with a-gem. In ICLR, 2019a.
  12. On tiny episodic memories in continual learning. arXiv:1902.10486, 2019b.
  13. Continual prototype evolution: Learning online from non-stationary data streams. In ICCV, 2021.
  14. Continual prune-and-select: class-incremental learning with specialized subnetworks. Applied Intelligence, 2023.
  15. Dytox: Transformers for continual learning with dynamic token expansion. In CVPR, 2022.
  16. Rigging the lottery: Making all tickets winners. In ICML, 2020.
  17. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2019.
  18. Why random pruning is all we need to start sparse. In ICML, 2023.
  19. The state of sparsity in deep neural networks. arXiv:1902.09574, 2019.
  20. Continual learning via neural pruning. In NeurIPS, 2019.
  21. Not just selection, but exploration: Online class-incremental continual learning via dual view consistency. In CVPR, 2022.
  22. Adaptive orthogonal projection for batch and online continual learning. In AAAI, 2022a.
  23. Online continual learning through mutual information maximization. In ICML, 2022b.
  24. Nispa: Neuro-inspired stability-plasticity adaptation for continual learning in sparse networks. In ICML, 2022.
  25. Exemplar-supported generative reproduction for class incremental learning. In BMVC, 2018.
  26. Deep residual learning for image recognition. In CVPR, 2016.
  27. Channel pruning for accelerating very deep neural networks. In ICCV, 2017.
  28. Lifelong learning via progressive distillation and retrospection. In ECCV, 2018.
  29. Overcoming catastrophic forgetting for continual learning via model adaptation. In ICLR, 2019.
  30. Automated machine learning: methods, systems, challenges. Springer Nature, 2019.
  31. Forget-free continual learning with winning subnetworks. In ICML, 2022a.
  32. On the soft-subnetwork for few-shot class incremental learning. In ICLR, 2023.
  33. Class-incremental learning by knowledge distillation with adaptive feature consolidation. In CVPR, 2022b.
  34. Kirkpatrick, J. et al. Overcoming catastrophic forgetting in neural networks. PNAS, 2017.
  35. Learning multiple layers of features from tiny images. 2009.
  36. Tiny imagenet visual recognition challenge. 2015.
  37. Mnist handwritten digit database. ATT Labs, 2, 2010.
  38. Continual learning with extended kronecker-factored approximate curvature. In CVPR, 2020.
  39. Snip: Single-shot network pruning based on connection sensitivity. In ICLR, 2019.
  40. Massively parallel hyperparameter tuning. arXiv:1810.05934, 2018.
  41. Learning without forgetting. TPAMI, 2017.
  42. Topological insights into sparse neural networks. In ECML PKDD, 2020.
  43. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. In ICLR, 2022.
  44. Adaptive aggregation networks for class-incremental learning. In CVPR, 2021a.
  45. Rmm: Reinforced memory management for class-incremental learning. In NeurIPS, 2021b.
  46. Gradient episodic memory for continual learning. In NeurIPS, 2017.
  47. Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning. In CVPR Workshops, 2021.
  48. Packnet: Adding multiple tasks to a single network by iterative pruning. In CVPR, 2018.
  49. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In ECCV, 2018.
  50. Diversity networks: Neural network compression using determinantal point processes. arXiv:1511.05077, 2015.
  51. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation. Elsevier, 1989.
  52. Wide neural networks forget less catastrophically. In ICML, 2022a.
  53. Architecture matters in continual learning. arXiv:2202.00275, 2022b.
  54. A topological insight into restricted boltzmann machines. Machine Learning, 104:243–270, 2016a.
  55. Online contrastive divergence with generative replay: Experience replay without storing data. arXiv:1610.05555, 2016b.
  56. Mocanu, D. C. et al. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 2018.
  57. Fetril: Feature translation for exemplar-free class-incremental learning. In WACV, 2023.
  58. Dualnet: Continual learning, fast and slow. In NeurIPS, 2021.
  59. Gdumb: A simple approach that questions our progress in continual learning. In ECCV, 2020.
  60. itaml: An incremental task-agnostic meta-learning approach. In CVPR, 2020.
  61. icarl: Incremental classifier and representation learning. In CVPR, 2017.
  62. Error sensitivity modulation based experience replay: Mitigating abrupt representation drift in continual learning. In ICLR, 2023.
  63. Online class-incremental continual learning with adversarial shapley value. In AAAI, 2021.
  64. Continual learning with deep generative replay. In NeurIPS, 2017.
  65. Spacenet: Make free space for continual learning. Neurocomputing, 2021.
  66. Avoiding forgetting and allowing forward transfer in continual learning via sparse networks. In ECML PKDD, 2023.
  67. Pruning neural networks without any data by iteratively conserving synaptic flow. In NeurIPS, 2020.
  68. Three scenarios for continual learning. arXiv:1904.07734, 2019.
  69. Picking winning tickets before training by preserving gradient flow. In ICLR, 2020.
  70. Foster: Feature boosting and compression for class-incremental learning. In ECCV, 2022a.
  71. Task difficulty aware parameter allocation & regularization for lifelong learning. In CVPR, 2023.
  72. Sparcl: Sparse continual learning on the edge. In NeurIPS, 2022b.
  73. Online prototype learning for online continual learning. In ICCV, 2023.
  74. Supermasks in superposition. In NeurIPS, 2020.
  75. Large scale incremental learning. In CVPR, 2019.
  76. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
  77. Der: Dynamically expandable representation for class incremental learning. In CVPR, 2021.
  78. Continual learning with dynamic sparse training: Exploring algorithms for effective model updates. In CPAL, 2023.
  79. Continual learning through synaptic intelligence. In ICML, 2017.
  80. Maintaining discrimination and fairness in class incremental learning. In CVPR, 2020.
  81. A model or 603 exemplars: Towards memory-efficient class-incremental learning. In ICLR, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now