Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Random Representations Outperform Online Continually Learned Representations (2402.08823v3)

Published 13 Feb 2024 in cs.CV and cs.LG

Abstract: Continual learning has primarily focused on the issue of catastrophic forgetting and the associated stability-plasticity tradeoffs. However, little attention has been paid to the efficacy of continually learned representations, as representations are learned alongside classifiers throughout the learning process. Our primary contribution is empirically demonstrating that existing online continually trained deep networks produce inferior representations compared to a simple pre-defined random transforms. Our approach projects raw pixels using a fixed random transform, approximating an RBF-Kernel initialized before any data is seen. We then train a simple linear classifier on top without storing any exemplars, processing one sample at a time in an online continual learning setting. This method, called RanDumb, significantly outperforms state-of-the-art continually learned representations across all standard online continual learning benchmarks. Our study reveals the significant limitations of representation learning, particularly in low-exemplar and online continual learning scenarios. Extending our investigation to popular exemplar-free scenarios with pretrained models, we find that training only a linear classifier on top of pretrained representations surpasses most continual fine-tuning and prompt-tuning strategies. Overall, our investigation challenges the prevailing assumptions about effective representation learning in online continual learning. Our code is available at://github.com/drimpossible/RanDumb.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (65)
  1. Ss-il: Separated softmax for incremental learning. In ICCV, 2021.
  2. Online continual learning with maximally interfered retrieval. In NeurIPS, 2019a.
  3. Gradient based sample selection for online continual learning. In NeurIPS, 2019b.
  4. Anonymous. Meta continual learning revisited: Implicitly enhancing online hessian approximation via variance reduction. In ICLR, 2024.
  5. Learning fast, learning slow: A general continual learning method based on complementary learning system. In ICLR, 2022.
  6. Rainbow memory: Continual learning with a memory of diverse samples. In CVPR, 2021.
  7. Edges are the’independent components’ of natural scenes. In NeurIPS, 1996.
  8. Class-incremental continual learning into the extended der-verse. TPAMI, 2022.
  9. Dark experience for general continual learning: a strong, simple baseline. In NeurIPS, 2020.
  10. New insights on reducing abrupt representation change in online continual learning. In ICLR, 2022.
  11. Co2l: Contrastive continual learning. In ICCV, 2021.
  12. Efficient lifelong learning with a-gem. In ICLR, 2019a.
  13. Continual learning with tiny episodic memories. In ICML-W, 2019b.
  14. Online bias correction for task-free continual learning. In ICLR, 2023.
  15. Continual prototype evolution: Learning online from non-stationary data streams. In ICCV, 2021.
  16. Continual learning: A comparative study on how to defy forgetting in classification tasks. In TPAMI, 2020.
  17. Tic-clip: Continual training of clip models. ArXiv, 2023.
  18. Online continual learning through mutual information maximization. In ICML, 2022.
  19. Look-ahead meta learning for continual learning. NeurIPS, 2020.
  20. Lifelong machine learning with deep streaming linear discriminant analysis. In CVPR-W, 2020.
  21. Knowledge accumulation in continually learned representations and the issue of feature forgetting. arXiv preprint arXiv:2304.00933, 2023.
  22. A simple baseline that questions the use of pretrained-models in continual learning. In NeurIPS-W, 2022.
  23. Gradient based memory editing for task-free continual learning. In NeurIPS, 2021.
  24. Overcoming catastrophic forgetting in neural networks. PNAS, 2017.
  25. A neural dirichlet process mixture model for task-free continual learning. In ICLR, 2020.
  26. Learning without forgetting. TPAMI, 2017.
  27. Gradient episodic memory for continual learning. In NeurIPS, 2017.
  28. Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning. In CVPR, 2021.
  29. Online continual learning in image classification: An empirical survey. Neurocomputing, 2022.
  30. Ranpac: Random projections and pre-trained models for continual learning. In NeurIPS, 2023.
  31. Goeffrey J McLachlan. Mahalanobis distance. Resonance, 4(6):20–26, 1999.
  32. Geoffrey J McLachlan. Discriminant analysis and statistical pattern recognition. John Wiley & Sons, 2005.
  33. Distance-based image classification: Generalizing to new classes at near-zero cost. TPAMI, 2013.
  34. Architecture matters in continual learning. arXiv preprint arXiv:2202.00275, 2022.
  35. Essentials for class incremental learning. In CVPR, 2021.
  36. Fine-tuning can cripple your foundation model; preserving features may be the solution. arXiv preprint arXiv:2308.13320, 2023.
  37. A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning. Neural Networks, 2023.
  38. A review of kernel methods for feature extraction in nonlinear process monitoring. Processes, 8(1), 2020.
  39. Gdumb: A simple approach that questions our progress in continual learning. In ECCV, 2020.
  40. Online continual learning without the storage constraint. arXiv preprint arXiv:2305.09253, 2023a.
  41. Computationally budgeted continual learning: What does matter? In CVPR, 2023b.
  42. Random features for large-scale kernel machines. NeurIPS, 2007.
  43. Continual unsupervised representation learning. NeurIPS, 32, 2019.
  44. icarl: Incremental classifier and representation learning. In CVPR, 2017.
  45. Learning to learn without forgetting by maximizing transfer and minimizing interference. In ICLR, 2019.
  46. Online class-incremental continual learning with adversarial shapley value. In AAAI, 2021.
  47. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In CVPR, 2023a.
  48. A closer look at rehearsal-free continual learning. In CVPR-W, 2023b.
  49. Gido M van de Ven and Andreas S Tolias. Three scenarios for continual learning. In NeurIPS-W, 2018.
  50. Class-incremental learning with generative classifiers. In CVPR-W, 2021.
  51. Continual learning: Applications and the road forward. arXiv preprint arXiv:2311.11908, 2023.
  52. Foster: Feature boosting and compression for class-incremental learning. In ECCV, 2022a.
  53. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision (ECCV), 2022b.
  54. Learning to prompt for continual learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022c.
  55. Online prototype learning for online continual learning. In ICCV, 2023.
  56. Large scale incremental learning. In CVPR, 2019.
  57. Continual variational autoencoder learning via online cooperative memorization. In ECCV, 2022.
  58. Self-evolved dynamic expansion model for task-free continual learning. In ICCV, 2023.
  59. Prediction error-based classification for class-incremental learning. ICLR, 2024.
  60. Continual learning through synaptic intelligence. In ICML, 2017.
  61. Task agnostic continual learning using online variational bayes. arXiv preprint arXiv:1803.10123, 2018.
  62. Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. In ICCV, 2023.
  63. A model or 603 exemplars: Towards memory-efficient class-incremental learning. arXiv preprint arXiv:2205.13218, 2022.
  64. Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need. arXiv preprint arXiv:2303.07338, 2023.
  65. Class-incremental learning via dual augmentation. NeurIPS, 2021.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets