Papers
Topics
Authors
Recent
Search
2000 character limit reached

Unsupervised Feature Learning with Emergent Data-Driven Prototypicality

Published 4 Jul 2023 in cs.CV and cs.AI | (2307.01421v1)

Abstract: Given an image set without any labels, our goal is to train a model that maps each image to a point in a feature space such that, not only proximity indicates visual similarity, but where it is located directly encodes how prototypical the image is according to the dataset. Our key insight is to perform unsupervised feature learning in hyperbolic instead of Euclidean space, where the distance between points still reflect image similarity, and yet we gain additional capacity for representing prototypicality with the location of the point: The closer it is to the origin, the more prototypical it is. The latter property is simply emergent from optimizing the usual metric learning objective: The image similar to many training instances is best placed at the center of corresponding points in Euclidean space, but closer to the origin in hyperbolic space. We propose an unsupervised feature learning algorithm in Hyperbolic space with sphere pACKing. HACK first generates uniformly packed particles in the Poincar\'e ball of hyperbolic space and then assigns each image uniquely to each particle. Images after congealing are regarded more typical of the dataset it belongs to. With our feature mapper simply trained to spread out training instances in hyperbolic space, we observe that images move closer to the origin with congealing, validating our idea of unsupervised prototypicality discovery. We demonstrate that our data-driven prototypicality provides an easy and superior unsupervised instance selection to reduce sample complexity, increase model generalization with atypical instances and robustness with typical ones.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. James W Anderson. Hyperbolic geometry. Springer Science & Business Media, 2006.
  2. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48, 2009.
  3. Prototype selection for interpretable classification. The Annals of Applied Statistics, 5(4):2403–2424, 2011.
  4. Károly Böröczky. Packing of spheres in spaces of constant curvature. Acta Mathematica Hungarica, 32(3-4):243–261, 1978.
  5. Prototypical examples in deep learning: Metrics, characteristics, and utility. [Online]. Available: https://openreview. net/forum?id=r1xyx3R9tQ, 2018.
  6. From trees to continuous embeddings and back: Hyperbolic hierarchical clustering. Advances in Neural Information Processing Systems, 33:15065–15076, 2020.
  7. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  8. Henry Cohn. A conceptual breakthrough in sphere packing. arXiv preprint arXiv:1611.01685, 2016.
  9. John Horton Conway and Neil James Alexander Sloane. Sphere packings, lattices and groups, volume 290. Springer Science & Business Media, 2013.
  10. Discriminatory analysis. nonparametric discrimination: Consistency properties. International Statistical Review/Revue Internationale de Statistique, 57(3):238–247, 1989.
  11. Hyperbolic neural networks. arXiv preprint arXiv:1805.09112, 2018.
  12. Alan Gibbons. Algorithmic graph theory. Cambridge university press, 1985.
  13. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
  14. Co-sne: Dimensionality reduction and visualization for hyperbolic data. arXiv preprint arXiv:2111.15037, 2021.
  15. Free hyperbolic neural networks with limited radii. arXiv preprint arXiv:2107.11472, 2021.
  16. Clipped hyperbolic classifiers are super-hyperbolic classifiers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11–20, 2022.
  17. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020.
  18. Momentum contrast for unsupervised visual representation learning. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9726–9735, 2019.
  19. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  20. Learning hyperbolic representations for unsupervised 3d segmentation. arXiv preprint arXiv:2012.01644, 2020.
  21. Examples are not enough, learn to criticize! criticism for interpretability. Advances in neural information processing systems, 29, 2016.
  22. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  23. Learning multiple layers of features from tiny images. Technical Report., 2009.
  24. Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/, 1998.
  25. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
  26. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  27. Less is more: Culling the training set to improve robustness of deep neural networks. In International Conference on Decision and Game Theory for Security, pages 102–114. Springer, 2018.
  28. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016.
  29. Continuous hierarchical representations with poincar\\\backslash\’e variational auto-encoders. arXiv preprint arXiv:1901.06033, 2019.
  30. Hyperbolic deep learning in computer vision: A survey. arXiv preprint arXiv:2305.06611, 2023.
  31. Learning from one example through shared densities on transforms. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No. PR00662), volume 1, pages 464–471. IEEE, 2000.
  32. James Munkres. Algorithms for the assignment and transportation problems. Journal of the society for industrial and applied mathematics, 5(1):32–38, 1957.
  33. A wrapped normal distribution on hyperbolic space for gradient-based learning. In International Conference on Machine Learning, pages 4693–4702. PMLR, 2019.
  34. Poincar\\\backslash\’e embeddings for learning hierarchical representations. arXiv preprint arXiv:1705.08039, 2017.
  35. Poincaré embeddings for learning hierarchical representations. Advances in neural information processing systems, 30, 2017.
  36. Rik Sarkar. Low distortion delaunay embedding of trees in hyperbolic plane. In International Symposium on Graph Drawing, pages 355–366. Springer, 2011.
  37. Claude Elwood Shannon. A mathematical theory of communication. ACM SIGMOBILE mobile computing and communications review, 5(1):3–55, 2001.
  38. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
  39. Convnets and imagenet beyond accuracy: Understanding mistakes and uncovering biases. In Proceedings of the European Conference on Computer Vision (ECCV), pages 498–512, 2018.
  40. Richard Szeliski et al. Image alignment and stitching: A tutorial. Foundations and Trends® in Computer Graphics and Vision, 2(1):1–104, 2007.
  41. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
  42. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pages 9929–9939. PMLR, 2020.
  43. Unsupervised discovery of the long-tail in instance segmentation using hierarchical self-supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2603–2612, 2021.
  44. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3733–3742, 2018.
  45. Jianping Zhang. Selecting typical instances in instance-based learning. In Machine learning proceedings 1992, pages 470–479. Elsevier, 1992.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up