Papers
Topics
Authors
Recent
Search
2000 character limit reached

Prototypical Hash Encoding for On-the-Fly Fine-Grained Category Discovery

Published 24 Oct 2024 in cs.CV | (2410.19213v2)

Abstract: In this paper, we study a practical yet challenging task, On-the-fly Category Discovery (OCD), aiming to online discover the newly-coming stream data that belong to both known and unknown classes, by leveraging only known category knowledge contained in labeled data. Previous OCD methods employ the hash-based technique to represent old/new categories by hash codes for instance-wise inference. However, directly mapping features into low-dimensional hash space not only inevitably damages the ability to distinguish classes and but also causes "high sensitivity" issue, especially for fine-grained classes, leading to inferior performance. To address these issues, we propose a novel Prototypical Hash Encoding (PHE) framework consisting of Category-aware Prototype Generation (CPG) and Discriminative Category Encoding (DCE) to mitigate the sensitivity of hash code while preserving rich discriminative information contained in high-dimension feature space, in a two-stage projection fashion. CPG enables the model to fully capture the intra-category diversity by representing each category with multiple prototypes. DCE boosts the discrimination ability of hash code with the guidance of the generated category prototypes and the constraint of minimum separation distance. By jointly optimizing CPG and DCE, we demonstrate that these two components are mutually beneficial towards an effective OCD. Extensive experiments show the significant superiority of our PHE over previous methods, e.g., obtaining an improvement of +5.3% in ALL ACC averaged on all datasets. Moreover, due to the nature of the interpretable prototypes, we visually analyze the underlying mechanism of how PHE helps group certain samples into either known or unknown categories. Code is available at https://github.com/HaiyangZheng/PHE.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84–90, 2017.
  2. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.
  3. Densely connected convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.
  4. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
  5. Learning to discover novel visual categories via deep transfer clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8401–8409, 2019.
  6. Neighborhood contrastive learning for novel class discovery. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10867–10875, 2021.
  7. A unified objective for novel class discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9284–9292, 2021.
  8. Generalized category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7492–7501, 2022.
  9. Dynamic conceptional contrastive learning for generalized category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  10. Federated generalized category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28741–28750, 2024.
  11. Textual knowledge matters: Cross-modality co-teaching for generalized visual class discovery. arXiv preprint arXiv:2403.07369, 2024.
  12. On-the-fly category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11691–11700, 2023.
  13. Parametric classification for generalized category discovery: A baseline study. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
  14. Rom Rubenovich Varshamov. Estimate of the number of signals in error correcting codes. Docklady Akad. Nauk, SSSR, 117:739–741, 1957.
  15. Parametric information maximization for generalized category discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
  16. Generalized category discovery with decoupled prototypical network. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
  17. Hashnet: Deep learning to hash by continuation. In Proceedings of the IEEE international conference on computer vision, pages 5608–5617, 2017.
  18. Feature learning based deep supervised hashing with pairwise labels. In Subbarao Kambhampati, editor, Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016, pages 1711–1717. IJCAI/AAAI Press, 2016.
  19. Deep supervised hashing for fast image retrieval. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2064–2072, 2016.
  20. Deep polarized network for supervised learning of accurate binary hashing codes. In IJCAI, pages 825–831, 2020.
  21. One loss for all: Deep hashing with a single cosine similarity based learning objective. Advances in Neural Information Processing Systems, 34:24286–24298, 2021.
  22. Central similarity quantization for efficient image and video retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3083–3092, 2020.
  23. Deep hashing with minimal-distance-separated hash centers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 23455–23464, 2023.
  24. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019.
  25. Concept-level debugging of part-prototype networks. arXiv preprint arXiv:2205.15769, 2022.
  26. Deformable protopnet: An interpretable image classifier using deformable prototypes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10265–10275, 2022.
  27. Proto2proto: Can you recognize the car, the way i do? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10233–10243, 2022.
  28. Neural prototype trees for interpretable fine-grained image recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14933–14943, 2021.
  29. Interpretable image recognition by constructing transparent embedding space. In Proceedings of the IEEE/CVF international conference on computer vision, pages 895–904, 2021.
  30. Protopformer: Concentrating on prototypical parts in vision transformers for interpretable image recognition. arXiv preprint arXiv:2208.10431, 2022.
  31. The caltech-ucsd birds-200-2011 dataset. Computation & Neural Systems Technical Report, 2011.
  32. 3d object representations for fine-grained categorization. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 2013.
  33. Cats and dogs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2012.
  34. Food-101–mining discriminative components with random forests. In European Conference on Computer Vision, 2014.
  35. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8769–8778, 2018.
  36. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
  37. John A Hartigan. Clustering algorithms. John Wiley & Sons, Inc., 1975.
  38. Autonovel: Automatically discovering and learning novel visual categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6767–6781, 2021.
  39. Joint representation learning and novel category discovery on single-and multi-modal data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 610–619, 2021.
  40. A memorizing and generalizing framework for lifelong person re-identification. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–18, 2023.
  41. Research on improved u-net based remote sensing image segmentation algorithm. arXiv preprint arXiv:2408.12672, 2024.
  42. Machine learning-based research on the adaptability of adolescents to online education. arXiv preprint arXiv:2408.16849, 2024.
  43. A mixed-heuristic quantum-inspired simplified swarm optimization algorithm for scheduling of real-time tasks in the multiprocessor system. Applied Soft Computing, 131:109807, 2022.
  44. Dynamic fraud detection: Integrating reinforcement learning into graph neural networks. arXiv preprint arXiv:2409.09892, 2024.
  45. Dral: Deep reinforcement adaptive learning for multi-uavs navigation in unknown indoor environment. arXiv preprint arXiv:2409.03930, 2024.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up