Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Zero-shot Classification using Hyperdimensional Computing (2401.16876v1)

Published 30 Jan 2024 in cs.CV and cs.LG

Abstract: Classification based on Zero-shot Learning (ZSL) is the ability of a model to classify inputs into novel classes on which the model has not previously seen any training examples. Providing an auxiliary descriptor in the form of a set of attributes describing the new classes involved in the ZSL-based classification is one of the favored approaches to solving this challenging task. In this work, inspired by Hyperdimensional Computing (HDC), we propose the use of stationary binary codebooks of symbol-like distributed representations inside an attribute encoder to compactly represent a computationally simple end-to-end trainable model, which we name Hyperdimensional Computing Zero-shot Classifier~(HDC-ZSC). It consists of a trainable image encoder, an attribute encoder based on HDC, and a similarity kernel. We show that HDC-ZSC can be used to first perform zero-shot attribute extraction tasks and, can later be repurposed for Zero-shot Classification tasks with minimal architectural changes and minimal model retraining. HDC-ZSC achieves Pareto optimal results with a 63.8% top-1 classification accuracy on the CUB-200 dataset by having only 26.6 million trainable parameters. Compared to two other state-of-the-art non-generative approaches, HDC-ZSC achieves 4.3% and 9.9% better accuracy, while they require more than 1.85x and 1.72x parameters compared to HDC-ZSC, respectively.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. O. Vinyals, C. Blundell, T. Lillicrap et al., “Matching networks for one shot learning,” Advances in Neural Information Processing Systems (NeurIPS), vol. 29, 2016.
  2. J. Snell, K. Swersky, and R. Zemel, “Prototypical networks for few-shot learning,” Advances in Neural Information Processing Systems (NeurIPS), vol. 30, 2017.
  3. V. Sunder, A. Srinivasan, L. Vig et al., “One-shot information extraction from document images using neuro-deductive program synthesis,” International Workshop on Neural-Symbolic Learning and Reasoning (NeSy), 2019.
  4. B. Romera-Paredes and P. Torr, “An embarrassingly simple approach to zero-shot learning,” in Proceedings of the 32nd International Conference on Machine Learning (ICML), vol. 37.   PMLR, 2015, pp. 2152–2161.
  5. E. Altszyler, P. Brusco, N. Basiou et al., “Zero-shot multi-domain dialog state tracking using descriptive rules,” International Workshop on Neural-Symbolic Learning and Reasoning (NeSy), 2020.
  6. M. Norouzi, T. Mikolov, S. Bengio et al., “Zero-shot learning by convex combination of semantic embeddings,” arXiv preprint arXiv:1312.5650, 2013.
  7. E. Kodirov, T. Xiang, and S. Gong, “Semantic autoencoder for zero-shot learning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 3174–3183.
  8. Z. Zhang and V. Saligrama, “Zero-shot learning via semantic similarity embedding,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
  9. D. Huynh and E. Elhamifar, “Compositional fine-grained low-shot learning,” arXiv preprint arXiv:2105.10438, 2021.
  10. S. Narayan, A. Gupta, F. S. Khan et al., “Latent embedding feedback and discriminative features for zero-shot classification,” in European Conference on Computer Vision (ECCV).   Springer, 2020, pp. 479–495.
  11. A. Radford, J. W. Kim, C. Hallacy et al., “Learning transferable visual models from natural language supervision,” in International Conference on Machine Learning (ICML).   PMLR, 2021, pp. 8748–8763.
  12. J. Li, M. Jing, K. Lu et al., “Leveraging the invariant side of generative zero-shot learning,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 7394–7403.
  13. P. Kanerva, “Hyperdimensional computing: An introduction to computing in distributed representation with high-dimensional random vectors,” Cognitive Computation, vol. 1, no. 2, pp. 139–159, 2009.
  14. G. Karunaratne, M. Schmuck, M. Le Gallo et al., “Robust high-dimensional memory-augmented neural networks,” Nature Communications, vol. 12, no. 1, pp. 1–12, 2021.
  15. M. Hersche, G. Karunaratne, G. Cherubini et al., “Constrained Few-shot Class-incremental Learning,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  16. G. Karunaratne, M. Hersche, J. Langeneager et al., “In-memory realization of in-situ few-shot continual learning with a dynamically evolving explicit memory,” in IEEE 48th European Solid State Circuits Conference (ESSCIRC), 2022, pp. 105–108.
  17. H. Jiang, R. Wang, S. Shan et al., “Transferable contrastive network for generalized zero-shot learning,” CoRR, vol. abs/1908.05832, 2019.
  18. P. D. Schiavone, D. Rossi, A. Pullini et al., “Quentin: an ultra-low-power pulpissimo soc in 22nm fdx,” in 2018 IEEE SOI-3D-Subthreshold Microelectronics Technology Unified Conference (S3S), 2018, pp. 1–3.
  19. Y. Xian, C. H. Lampert, B. Schiele et al., “Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 9, pp. 2251–2265, 2018.
  20. Z. Akata, F. Perronnin, Z. Harchaoui et al., “Label-embedding for image classification,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 7, pp. 1425–1438, 2015.
  21. Y. Xian, Z. Akata, G. Sharma et al., “Latent embeddings for zero-shot classification,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 69–77.
  22. R. Socher, M. Ganjoo, C. D. Manning et al., “Zero-shot learning through cross-modal transfer,” Advances in Neural Information Processing Systems (NeurIPS), vol. 26, 2013.
  23. C. H. Lampert, H. Nickisch, and S. Harmeling, “Attribute-based classification for zero-shot visual object categorization,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 36, pp. 453–465, 2014.
  24. S. Changpinyo, W.-L. Chao, B. Gong et al., “Synthesized classifiers for zero-shot learning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 5327–5336.
  25. V. K. Verma and P. Rai, “A simple exponential family framework for zero-shot learning,” in Machine Learning and Knowledge Discovery in Databases.   Springer International Publishing, 2017, pp. 792–808.
  26. Y. Xian, S. Sharma, B. Schiele et al., “F-VAEGAN-D2: A feature generating framework for any-shot learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  27. R. Felix, I. Reid, G. Carneiro et al., “Multi-modal cycle-consistent generalized zero-shot learning,” in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 21–37.
  28. Y. Xian, T. Lorenz, B. Schiele et al., “Feature generating networks for zero-shot learning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 5542–5551.
  29. A. Rahimi, P. Kanerva, and J. M. Rabaey, “A robust and energy-efficient classifier using brain-inspired hyperdimensional computing,” in Proceedings of the 2016 International Symposium on Low Power Electronics and Design (ISLPED), 2016.
  30. M. Schmuck, L. Benini, and A. Rahimi, “Hardware optimizations of dense binary hyperdimensional computing: Rematerialization of hypervectors, binarized bundling, and combinational associative memory,” ACM Journal on Emerging Technologies in Computing Systems (JETC), vol. 15, no. 4, pp. 1–25, 2019.
  31. C. Wah, S. Branson, P. Welinder et al., “Cub-200 2011 dataset,” California Institute of Technology, Tech. Rep. CNS-TR-2011-001, 2011.
  32. I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in International Conference on Learning Representations (ICLR), 2019.
  33. ——, “SGDR: Stochastic gradient descent with warm restarts,” arXiv preprint arXiv:1608.03983, 2016.
  34. R. Zakizadeh, M. Sasdelli, Y. Qian et al., “Finetag: Multi-attribute classification at fine-grained level in images,” arXiv preprint arXiv:1806.07124, 2018.
  35. K. Han, J. Guo, C. Zhang et al., “Attribute-aware attention model for fine-grained representation learning,” in Proceedings of the 26th ACM international conference on Multimedia, 2018, pp. 2040–2048.
  36. A. Garofalo, G. Ottavi, F. Conti et al., “A heterogeneous in-memory computing cluster for flexible end-to-end inference of real-world deep neural networks,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems (JETCAS), vol. 12, no. 2, pp. 422–435, 2022.
  37. R. Khaddam-Aljameh, M. Stanisavljevic, J. F. Mas et al., “Hermes-core—a 1.59-tops/mm 2 pcm on 14-nm cmos in-memory compute core using 300-ps/lsb linearized cco-based adcs,” IEEE Journal of Solid-State Circuits, vol. 57, no. 4, pp. 1027–1038, 2022.
  38. M. Eggimann, A. Rahimi, and L. Benini, “A 5 μ𝜇\muitalic_μw standard cell memory-based configurable hyperdimensional computing accelerator for always-on smart sensing,” IEEE Transactions on Circuits and Systems I: Regular Papers, vol. 68, no. 10, pp. 4116–4128, 2021.
Citations (1)
View on Semantic Scholar

Summary

  • The paper presents HDC-ZSC, a novel non-generative method leveraging hyperdimensional computing for efficient zero-shot learning.
  • It uses a three-phase training process with an image encoder and a stationary HDC-based attribute encoder, achieving 63.8% top-1 accuracy on the CUB-200 dataset.
  • The approach outperforms similar methods by up to 9.9% while significantly reducing parameter counts, making it ideal for resource-limited systems.

Hyperdimensional Computing and Zero-shot Learning

Introduction to Zero-shot Learning and HDC

Zero-shot learning (ZSL) is an unorthodox approach in machine learning where a model attempts to classify objects it has never explicitly seen during training. It relies on descriptors that semantically link seen and unseen classes. This paper presents an innovative non-generative approach named Hyperdimensional Computing Zero-shot Classifier (HDC-ZSC), which utilizes Hyperdimensional Computing (HDC) to create efficient attribute encoders for ZSL tasks. HDC, drawing inspiration from the brain's processing capabilities, employs high-dimensional binary vectors (or hypervectors) to represent information which allows for simple, powerful computations.

Model Architecture and Training

The HDC-ZSC model comprises three key components: a trainable image encoder that processes input images, a stationary HDC-based attribute encoder utilizing binary codebooks to encode class attributes, and a similarity kernel that compares embeddings from the encoder to those represented by the attribute encoder. Interestingly, the attribute encoder leverages randomly initialized binary hypervectors that are quasi-orthogonal, enabling HDC's potential for resource-efficient computing.

Training involves a three-phase process: the model initially focuses on standard image classification tasks, shifts towards the attribute extraction phase, and finally converges to the ZSC task. These steps equip the image encoder with rich semantics needed for zero-shot classification. Essentially, the HDC-ZSC model first learns to recognize attributes from images and subsequently uses the learned representations to interpolate to unseen classes.

Empirical Results and Comparisons

The model's prowess is quantified on the CUB-200 dataset, where HDC-ZSC reaches an impressive 63.8% top-1 classification accuracy with a minimal parametric footprint of only 26.6 million trainable parameters. In stark contrast to other non-generative methods, it outperforms them by up to 9.9% more accurately and requires significantly fewer parameters (by factors of 1.85× and 1.72×). Additionally, against computationally expensive generative methods, HDC-ZSC maintains competitive accuracy while greatly reducing the parameter count, thereby underscoring its efficiency.

Concluding Remarks

The efficacy of HDC-ZSC makes a compelling case for its application in environments where computational resources are constrained, such as embedded systems. The paper suggests that future iterations of the model could explore hardware-accelerated versions, enabling on-the-edge training and inference, which would be a significant leap for AI deployment in resource-limited scenarios. Acknowledgments were given to Junyuan Cui for his preliminary technical studies, which paved the way for the HDC-ZSC's development.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets