Calibrating Higher-Order Statistics for Few-Shot Class-Incremental Learning with Pre-trained Vision Transformers (2404.06622v1)
Abstract: Few-shot class-incremental learning (FSCIL) aims to adapt the model to new classes from very few data (5 samples) without forgetting the previously learned classes. Recent works in many-shot CIL (MSCIL) (using all available training data) exploited pre-trained models to reduce forgetting and achieve better plasticity. In a similar fashion, we use ViT models pre-trained on large-scale datasets for few-shot settings, which face the critical issue of low plasticity. FSCIL methods start with a many-shot first task to learn a very good feature extractor and then move to the few-shot setting from the second task onwards. While the focus of most recent studies is on how to learn the many-shot first task so that the model generalizes to all future few-shot tasks, we explore in this work how to better model the few-shot data using pre-trained models, irrespective of how the first task is trained. Inspired by recent works in MSCIL, we explore how using higher-order feature statistics can influence the classification of few-shot classes. We identify the main challenge of obtaining a good covariance matrix from few-shot data and propose to calibrate the covariance matrix for new classes based on semantic similarity to the many-shot base classes. Using the calibrated feature statistics in combination with existing methods significantly improves few-shot continual classification on several FSCIL benchmarks. Code is available at https://github.com/dipamgoswami/FSCIL-Calibration.
- Few-shot class incremental learning leveraging self-supervised features. In Conference on computer vision and pattern recognition, 2022.
- Subspace regularizers for few-shot class incremental learning. In International Conference on Learning Representations (ICLR), 2022.
- Il2m: Class incremental learning with dual memory. In International Conference on Computer Vision (ICCV), 2019.
- Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 2022.
- Continual prototype evolution: Learning online from non-stationary data streams. In International Conference on Computer Vision (ICCV), 2021.
- A continual learning survey: Defying forgetting in classification tasks. Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 2021.
- Expanding hyperspherical space for few-shot class-incremental learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024.
- Learning without memorizing. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
- An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
- Podnet: Pooled outputs distillation for small-tasks incremental learning. In European Conference on Computer Vision (ECCV), 2020.
- Fecam: Exploiting the heterogeneity of class distributions in exemplar-free continual learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
- Lifelong machine learning with deep streaming linear discriminant analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 220–221, 2020.
- Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
- Constrained few-shot class-incremental learning. In Conference on Computer Vision and Pattern Recognition, 2022.
- Learning a unified classifier incrementally via rebalancing. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
- A simple baseline that questions the use of pretrained-models in continual learning. arXiv preprint arXiv:2210.04428, 2022.
- S3c: Self-supervised stochastic classifiers for few-shot class-incremental learning. In European Conference on Computer Vision, 2022.
- On the soft-subnetwork for few-shot class incremental learning. In The Eleventh International Conference on Learning Representations, 2022.
- Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI conference on artificial intelligence, 2018.
- Warping the space: Weight space rotation for class-incremental few-shot learning. In The Eleventh International Conference on Learning Representations, 2022.
- Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences (PNAS), 2017.
- 3d object representations for fine-grained categorization. In International Conference on Computer Vision (ICCV-W) Workshops, 2013.
- Alex Krizhevsky. Learning multiple layers of features from tiny images. pages 32–33, 2009.
- Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 2017.
- Learnable distribution calibration for few-shot class-incremental learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023a.
- Rotate your networks: Better weight consolidation and less catastrophic forgetting. In 24th International Conference on Pattern Recognition (ICPR). IEEE, 2018.
- Augmented box replay: Overcoming foreground shift for incremental object detection. In International Conference on Computer Vision (ICCV), 2023b.
- Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.
- Class-incremental learning: survey and performance evaluation. Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 2022.
- Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation. Elsevier, 1989.
- Ranpac: Random projections and pre-trained models for continual learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
- First session adaptation: A strong replay-free baseline for class-incremental learning. In International Conference on Computer Vision (ICCV), 2023.
- Few-shot class-incremental learning from an open-set perspective. In European Conference on Computer Vision (ECCV), 2022.
- Fetril: Feature translation for exemplar-free class-incremental learning. In Winter Conference on Applications of Computer Vision (WACV), 2023.
- icarl: Incremental classifier and representation learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
- Imagenet-21k pretraining for the masses. In Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
- Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 1995.
- One-shot learning with a hierarchical nonparametric bayesian model. In Proceedings of ICML Workshop on Unsupervised and Transfer Learning. JMLR Workshop and Conference Proceedings, 2012.
- Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems (NeurIPS), 2017.
- Learning with fantasy: Semantic-aware virtual contrastive constraint for few-shot class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
- On the importance of cross-task features for class-incremental learning. International Conference on Machine Learning (ICML) Workshops, 2021.
- Pilot: A pre-trained model-based continual learning toolbox. arXiv preprint arXiv:2309.07117, 2023.
- Few-shot class-incremental learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
- Pl-fscil: Harnessing the power of prompts for few-shot class-incremental learning. arXiv preprint arXiv:2401.14807, 2024a.
- A survey on few-shot class-incremental learning. Neural Networks, 2024b.
- Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734, 2019.
- Matching networks for one shot learning. Advances in Neural Information Processing Systems (NeurIPS), 2016.
- The caltech-ucsd birds-200-2011 dataset. 2011.
- A comprehensive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302.00487, 2023a.
- Few-shot class-incremental learning via training-free prototype calibration. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b.
- Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 2020.
- Free lunch for few-shot learning: Distribution calibration. In International Conference on Learning Representations (ICLR), 2021.
- Neural collapse inspired feature-classifier alignment for few-shot class-incremental learning. In The Eleventh International Conference on Learning Representations, 2022.
- Few-shot incremental learning with continually evolved classifiers. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
- Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
- Few-shot class-incremental learning via class-aware bilateral distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
- Forward compatible few-shot class-incremental learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022a.
- Few-shot class-incremental learning by sampling multi-phase tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022b.
- Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need. arXiv preprint arXiv:2303.07338, 2023.
- Continual learning with pre-trained models: A survey. arXiv preprint arXiv:2401.16386, 2024.
- Prototype augmentation and self-supervision for incremental learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
- Dipam Goswami (10 papers)
- Bartłomiej Twardowski (37 papers)
- Joost van de Weijer (133 papers)