Papers
Topics
Authors
Recent
Search
2000 character limit reached

SPTNet: An Efficient Alternative Framework for Generalized Category Discovery with Spatial Prompt Tuning

Published 20 Mar 2024 in cs.CV and cs.AI | (2403.13684v3)

Abstract: Generalized Category Discovery (GCD) aims to classify unlabelled images from both seen' andunseen' classes by transferring knowledge from a set of labelled `seen' class images. A key theme in existing GCD approaches is adapting large-scale pre-trained models for the GCD task. An alternate perspective, however, is to adapt the data representation itself for better alignment with the pre-trained model. As such, in this paper, we introduce a two-stage adaptation approach termed SPTNet, which iteratively optimizes model parameters (i.e., model-finetuning) and data parameters (i.e., prompt learning). Furthermore, we propose a novel spatial prompt tuning method (SPT) which considers the spatial property of image data, enabling the method to better focus on object parts, which can transfer between seen and unseen classes. We thoroughly evaluate our SPTNet on standard benchmarks and demonstrate that our method outperforms existing GCD methods. Notably, we find our method achieves an average accuracy of 61.4% on the SSB, surpassing prior state-of-the-art methods by approximately 10%. The improvement is particularly remarkable as our method yields extra parameters amounting to only 0.117% of those in the backbone architecture. Project page: https://visual-ai.github.io/sptnet.

Authors (3)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. k-means++: The advantages of careful seeding. Technical report, Stanford, 2006.
  2. Masked siamese networks for label-efficient learning. In ECCV, 2022.
  3. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv: 2203.17274, 2022.
  4. Mixmatch: A holistic approach to semi-supervised learning. In NeurIPS, 2019.
  5. Benders subproblem decomposition for bilevel problems with convex follower. INFORMS Journal on Computing, 2022.
  6. Open-world semi-supervised learning. In ICLR, 2022.
  7. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
  8. A simple framework for contrastive learning of visual representations. In ICML, 2020a.
  9. Big self-supervised models are strong semi-supervised learners. In NeurIPS, 2020b.
  10. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020c.
  11. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
  12. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodological), 1977.
  13. Lpt: Long-tailed prompt tuning for image classification. In ICLR, 2022.
  14. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
  15. Decomposition of nonconvex optimization via bi-level distributed aladin. IEEE Transactions on Control of Network Systems, 2020.
  16. A unified objective for novel class discovery. In ICCV, 2021.
  17. Dynamic few-shot visual learning without forgetting. In CVPR, 2018.
  18. Class-relation knowledge distillation for novel class discovery. In ICCV, 2023.
  19. Learning to discover novel visual categories via deep transfer clustering. In ICCV, 2019.
  20. Automatically discovering and learning new visual categories with ranking statistics. In ICLR, 2020.
  21. Autonovel: Automatically discovering and learning novel visual categories. IEEE TPAMI, 2021.
  22. Cipr: An efficient framework with cross-instance positive relations for generalized category discovery. TMLR, 2024.
  23. Provable guarantees for self-supervised deep learning with spectral contrastive loss. In NeurIPS, 2021.
  24. Deep residual learning for image recognition. In CVPR, 2016.
  25. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
  26. Visual prompt tuning. In ECCV, 2022.
  27. Joint representation learning and novel category discovery on single- and multi-modal data. In ICCV, 2021.
  28. Maple: Multi-modal prompt learning. In CVPR, 2023.
  29. Supervised contrastive learning. In NeurIPS, 2020.
  30. 3d object representations for fine-grained categorization. In ICCV workshop, 2013.
  31. Learning multiple layers of features from tiny images. 2009.
  32. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 2017.
  33. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.
  34. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
  35. DINOv2: Learning robust visual features without supervision. TMLR, 2024.
  36. Moment matching for multi-source domain adaptation. In ICCV, 2019.
  37. Dynamic conceptional contrastive learning for generalized category discovery. In CVPR, 2023.
  38. Openldn: Learning to discover novel classes for open-world semi-supervised learning. In ECCV, 2022.
  39. What does clip know about a red circle? visual prompt engineering for vlms. In ICCV, 2023.
  40. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NeurIPS, 2020.
  41. A graph-theoretic framework for understanding open-world semi-supervised learning. In NeurIPS, 2024.
  42. The herbarium challenge 2019 dataset. arXiv preprint arXiv: 1906.05372, 2019.
  43. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NeurIPS, 2017.
  44. Contrastive multiview coding. In ECCV, 2020.
  45. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008.
  46. Open-set recognition: A good closed-set classifier is all you need. In ICLR, 2021.
  47. Generalized category discovery. In CVPR, 2022.
  48. No representation rules them all in category discovery. In NeurIPS, 2023.
  49. S-prompts learning with pre-trained transformers: An occam’s razor for domain incremental learning. In NeurIPS, 2022a.
  50. Usb: A unified semi-supervised learning benchmark for classification. In NeurIPS, 2022b.
  51. Discover and align taxonomic context priors for open-world semi-supervised learning. In NeurIPS, 2024.
  52. Caltech-ucsd birds 200. 2010.
  53. Parametric classification for generalized category discovery: A baseline study. ICCV, 2023.
  54. Promptcal: Contrastive affinity learning via auxiliary prompts for generalized novel category discovery. In CVPR, 2023.
  55. Novel visual category discovery with dual ranking statistics and mutual knowledge distillation. In NeurIPS, 2021.
  56. Learning semi-supervised gaussian mixture models for generalized category discovery. In ICCV, 2023.
  57. Neighborhood contrastive learning for novel class discovery. In CVPR, 2021a.
  58. Openmix: Reviving known knowledge for discovering novel visual categories in an open world. In CVPR, 2021b.
Citations (13)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Tweets

Sign up for free to view the 1 tweet with 8 likes about this paper.

Sign Up for Free