Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Calibrating Higher-Order Statistics for Few-Shot Class-Incremental Learning with Pre-trained Vision Transformers (2404.06622v1)

Published 9 Apr 2024 in cs.CV

Abstract: Few-shot class-incremental learning (FSCIL) aims to adapt the model to new classes from very few data (5 samples) without forgetting the previously learned classes. Recent works in many-shot CIL (MSCIL) (using all available training data) exploited pre-trained models to reduce forgetting and achieve better plasticity. In a similar fashion, we use ViT models pre-trained on large-scale datasets for few-shot settings, which face the critical issue of low plasticity. FSCIL methods start with a many-shot first task to learn a very good feature extractor and then move to the few-shot setting from the second task onwards. While the focus of most recent studies is on how to learn the many-shot first task so that the model generalizes to all future few-shot tasks, we explore in this work how to better model the few-shot data using pre-trained models, irrespective of how the first task is trained. Inspired by recent works in MSCIL, we explore how using higher-order feature statistics can influence the classification of few-shot classes. We identify the main challenge of obtaining a good covariance matrix from few-shot data and propose to calibrate the covariance matrix for new classes based on semantic similarity to the many-shot base classes. Using the calibrated feature statistics in combination with existing methods significantly improves few-shot continual classification on several FSCIL benchmarks. Code is available at https://github.com/dipamgoswami/FSCIL-Calibration.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Few-shot class incremental learning leveraging self-supervised features. In Conference on computer vision and pattern recognition, 2022.
  2. Subspace regularizers for few-shot class incremental learning. In International Conference on Learning Representations (ICLR), 2022.
  3. Il2m: Class incremental learning with dual memory. In International Conference on Computer Vision (ICCV), 2019.
  4. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 2022.
  5. Continual prototype evolution: Learning online from non-stationary data streams. In International Conference on Computer Vision (ICCV), 2021.
  6. A continual learning survey: Defying forgetting in classification tasks. Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 2021.
  7. Expanding hyperspherical space for few-shot class-incremental learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024.
  8. Learning without memorizing. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  9. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
  10. Podnet: Pooled outputs distillation for small-tasks incremental learning. In European Conference on Computer Vision (ECCV), 2020.
  11. Fecam: Exploiting the heterogeneity of class distributions in exemplar-free continual learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  12. Lifelong machine learning with deep streaming linear discriminant analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 220–221, 2020.
  13. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  14. Constrained few-shot class-incremental learning. In Conference on Computer Vision and Pattern Recognition, 2022.
  15. Learning a unified classifier incrementally via rebalancing. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  16. A simple baseline that questions the use of pretrained-models in continual learning. arXiv preprint arXiv:2210.04428, 2022.
  17. S3c: Self-supervised stochastic classifiers for few-shot class-incremental learning. In European Conference on Computer Vision, 2022.
  18. On the soft-subnetwork for few-shot class incremental learning. In The Eleventh International Conference on Learning Representations, 2022.
  19. Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI conference on artificial intelligence, 2018.
  20. Warping the space: Weight space rotation for class-incremental few-shot learning. In The Eleventh International Conference on Learning Representations, 2022.
  21. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences (PNAS), 2017.
  22. 3d object representations for fine-grained categorization. In International Conference on Computer Vision (ICCV-W) Workshops, 2013.
  23. Alex Krizhevsky. Learning multiple layers of features from tiny images. pages 32–33, 2009.
  24. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 2017.
  25. Learnable distribution calibration for few-shot class-incremental learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023a.
  26. Rotate your networks: Better weight consolidation and less catastrophic forgetting. In 24th International Conference on Pattern Recognition (ICPR). IEEE, 2018.
  27. Augmented box replay: Overcoming foreground shift for incremental object detection. In International Conference on Computer Vision (ICCV), 2023b.
  28. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.
  29. Class-incremental learning: survey and performance evaluation. Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 2022.
  30. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation. Elsevier, 1989.
  31. Ranpac: Random projections and pre-trained models for continual learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  32. First session adaptation: A strong replay-free baseline for class-incremental learning. In International Conference on Computer Vision (ICCV), 2023.
  33. Few-shot class-incremental learning from an open-set perspective. In European Conference on Computer Vision (ECCV), 2022.
  34. Fetril: Feature translation for exemplar-free class-incremental learning. In Winter Conference on Applications of Computer Vision (WACV), 2023.
  35. icarl: Incremental classifier and representation learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  36. Imagenet-21k pretraining for the masses. In Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
  37. Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 1995.
  38. One-shot learning with a hierarchical nonparametric bayesian model. In Proceedings of ICML Workshop on Unsupervised and Transfer Learning. JMLR Workshop and Conference Proceedings, 2012.
  39. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems (NeurIPS), 2017.
  40. Learning with fantasy: Semantic-aware virtual contrastive constraint for few-shot class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  41. On the importance of cross-task features for class-incremental learning. International Conference on Machine Learning (ICML) Workshops, 2021.
  42. Pilot: A pre-trained model-based continual learning toolbox. arXiv preprint arXiv:2309.07117, 2023.
  43. Few-shot class-incremental learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  44. Pl-fscil: Harnessing the power of prompts for few-shot class-incremental learning. arXiv preprint arXiv:2401.14807, 2024a.
  45. A survey on few-shot class-incremental learning. Neural Networks, 2024b.
  46. Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734, 2019.
  47. Matching networks for one shot learning. Advances in Neural Information Processing Systems (NeurIPS), 2016.
  48. The caltech-ucsd birds-200-2011 dataset. 2011.
  49. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302.00487, 2023a.
  50. Few-shot class-incremental learning via training-free prototype calibration. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b.
  51. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 2020.
  52. Free lunch for few-shot learning: Distribution calibration. In International Conference on Learning Representations (ICLR), 2021.
  53. Neural collapse inspired feature-classifier alignment for few-shot class-incremental learning. In The Eleventh International Conference on Learning Representations, 2022.
  54. Few-shot incremental learning with continually evolved classifiers. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  55. Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
  56. Few-shot class-incremental learning via class-aware bilateral distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  57. Forward compatible few-shot class-incremental learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022a.
  58. Few-shot class-incremental learning by sampling multi-phase tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022b.
  59. Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need. arXiv preprint arXiv:2303.07338, 2023.
  60. Continual learning with pre-trained models: A survey. arXiv preprint arXiv:2401.16386, 2024.
  61. Prototype augmentation and self-supervision for incremental learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Dipam Goswami (10 papers)
  2. Bartłomiej Twardowski (37 papers)
  3. Joost van de Weijer (133 papers)
Citations (2)

Summary

We haven't generated a summary for this paper yet.

X Twitter Logo Streamline Icon: https://streamlinehq.com