Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Prediction Error-based Classification for Class-Incremental Learning (2305.18806v2)

Published 30 May 2023 in cs.LG, cs.AI, cs.CV, and stat.ML

Abstract: Class-incremental learning (CIL) is a particularly challenging variant of continual learning, where the goal is to learn to discriminate between all classes presented in an incremental fashion. Existing approaches often suffer from excessive forgetting and imbalance of the scores assigned to classes that have not been seen together during training. In this study, we introduce a novel approach, Prediction Error-based Classification (PEC), which differs from traditional discriminative and generative classification paradigms. PEC computes a class score by measuring the prediction error of a model trained to replicate the outputs of a frozen random neural network on data from that class. The method can be interpreted as approximating a classification rule based on Gaussian Process posterior variance. PEC offers several practical advantages, including sample efficiency, ease of tuning, and effectiveness even when data are presented one class at a time. Our empirical results show that PEC performs strongly in single-pass-through-data CIL, outperforming other rehearsal-free baselines in all cases and rehearsal-based methods with moderate replay buffer size in most cases across multiple benchmarks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems, volume 32, 2019a.
  2. Task-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  11254–11263, 2019b.
  3. Generative vs. discriminative: Rethinking the meta-continual learning. In Advances in Neural Information Processing Systems, volume 34, pp.  21592–21604, 2021.
  4. A comprehensive study of class incremental learning algorithms for visual tasks. Neural Networks, 135:38–54, 2021.
  5. Class-incremental continual learning into the extended der-verse. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  6. Exploration by random network distillation. arXiv preprint arXiv:1810.12894, 2018.
  7. Dark experience for general continual learning: a strong, simple baseline. In Advances in Neural Information Processing Systems, volume 33, pp.  15920–15930, 2020.
  8. New insights on reducing abrupt representation change in online continual learning. In International Conference on Learning Representations, 2021.
  9. Efficient lifelong learning with A-GEM. In International Conference on Learning Representations, 2019a.
  10. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486, 2019b.
  11. Conservative uncertainty estimation by fitting prior networks. In International Conference on Learning Representations, 2020.
  12. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3366–3385, 2022.
  13. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp.  248–255. IEEE, 2009.
  14. Li Deng. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141–142, 2012.
  15. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp.  249–256. JMLR Workshop and Conference Proceedings, 2010.
  16. Deep Learning. MIT press, 2016.
  17. Embracing change: Continual learning in deep neural networks. Trends in Cognitive Sciences, 24(12):1028–1040, 2020.
  18. Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation, 9(2):159–195, 2001.
  19. Lifelong machine learning with deep streaming linear discriminant analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp.  220–221, 2020.
  20. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp.  1026–1034, 2015.
  21. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.  770–778, 2016.
  22. Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415, 2016.
  23. Posterior meta-replay for continual learning. Advances in Neural Information Processing Systems, 34:14135–14149, 2021.
  24. Generalization in deep learning. arXiv preprint arXiv:1710.05468, 2017.
  25. Continual learning based on ood detection and task masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp.  3856–3866, 2022a.
  26. A theoretical study on solving continual learning. Advances in Neural Information Processing Systems, 35:5065–5079, 2022b.
  27. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  28. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  29. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017.
  30. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
  31. Biological underpinnings for lifelong learning machines. Nature Machine Intelligence, 4(3):196–210, 2022.
  32. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935–2947, 2017.
  33. CORe50: a new dataset and benchmark for continuous object recognition. In Conference on Robot Learning, pp.  17–26. PMLR, 2017.
  34. Gradient episodic memory for continual learning. Advances in Neural Information Processing Systems, 30, 2017.
  35. Continuous learning in single-incremental-task scenarios. Neural Networks, 116:56–73, 2019.
  36. Class-incremental learning: survey and performance evaluation on image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5513–5533, 2023.
  37. Radford M Neal. Priors for infinite networks. In Bayesian Learning for Neural Networks, pp.  29–53. Springer, 1996.
  38. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
  39. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54–71, 2019.
  40. PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
  41. Gaussian Processes for Machine Learning. MIT press Cambridge, MA, 2006.
  42. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.  2001–2010, 2017.
  43. Continual learning with deep generative replay. Advances in Neural Information Processing Systems, 30, 2017.
  44. A comprehensive empirical evaluation on online continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pp.  3518–3528, 2023.
  45. Class-incremental learning with generative classifiers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp.  3611–3620, 2021.
  46. Three types of incremental learning. Nature Machine Intelligence, 4:1185–1197, 2022.
  47. Matching networks for one shot learning. Advances in Neural Information Processing Systems, 29, 2016.
  48. Large scale incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  374–382, 2019.
  49. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
  50. Greg Yang. Wide feedforward or recurrent neural networks of any architecture are gaussian processes. Advances in Neural Information Processing Systems, 32, 2019.
  51. Christoph Zauner. Implementation and benchmarking of perceptual image hash functions, 2010.
  52. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pp.  3987–3995. PMLR, 2017.
  53. Task agnostic continual learning using online variational bayes. arXiv preprint arXiv:1803.10123, 2018.
  54. Deep class-incremental learning: A survey. arXiv preprint arXiv:2302.03648, 2023.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets