Papers
Topics
Authors
Recent
2000 character limit reached

Poincaré ResNet (2303.14027v3)

Published 24 Mar 2023 in cs.CV

Abstract: This paper introduces an end-to-end residual network that operates entirely on the Poincar\'e ball model of hyperbolic space. Hyperbolic learning has recently shown great potential for visual understanding, but is currently only performed in the penultimate layer(s) of deep networks. All visual representations are still learned through standard Euclidean networks. In this paper we investigate how to learn hyperbolic representations of visual data directly from the pixel-level. We propose Poincar\'e ResNet, a hyperbolic counterpart of the celebrated residual network, starting from Poincar\'e 2D convolutions up to Poincar\'e residual connections. We identify three roadblocks for training convolutional networks entirely in hyperbolic space and propose a solution for each: (i) Current hyperbolic network initializations collapse to the origin, limiting their applicability in deeper networks. We provide an identity-based initialization that preserves norms over many layers. (ii) Residual networks rely heavily on batch normalization, which comes with expensive Fr\'echet mean calculations in hyperbolic space. We introduce Poincar\'e midpoint batch normalization as a faster and equally effective alternative. (iii) Due to the many intermediate operations in Poincar\'e layers, we lastly find that the computation graphs of deep learning libraries blow up, limiting our ability to train on deep hyperbolic networks. We provide manual backward derivations of core hyperbolic operations to maintain manageable computation graphs.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Multi-relational poincaré graph embeddings. Advances in Neural Information Processing Systems, 2019.
  2. Silvere Bonnabel. Stochastic gradient descent on riemannian manifolds. IEEE Transactions on Automatic Control, 58:2217–2229, 2013.
  3. Latent variable modelling with hyperbolic normalizing flows. In International Conference on Machine Learning, 2020.
  4. From trees to continuous embeddings and back: Hyperbolic hierarchical clustering. In Advances in Neural Information Processing Systems, 2020.
  5. Hyperbolic graph convolutional neural networks. In Advances in Neural Information Processing Systems, 2019.
  6. Hyperbolic uncertainty aware semantic segmentation. Transactions on Intelligent Transportation Systems, 2022.
  7. Fully hyperbolic neural networks. arXiv preprint arXiv:2105.14686, 2021.
  8. Describing textures in the wild. In Computer Vision and Pattern Recognition, 2014.
  9. A hyperbolic-to-hyperbolic graph convolutional network. In Computer Vision and Pattern Recognition, 2021.
  10. Hierarchical image classification using entailment cone embeddings. In Computer Vision and Pattern Recognition Workshops, 2020.
  11. Embedding text in hyperbolic spaces. In Workshop on Graph-Based Methods for Natural Language Processing, 2018.
  12. Hyperbolic vision transformers: Combining improvements in metric learning. In Computer Vision and Pattern Recognition, 2022.
  13. Kernel methods in hyperbolic spaces. In International Conference on Computer Vision, 2021.
  14. Hyperbolic self-paced learning for self-supervised skeleton-based action representations. In International Conference on Learning Representations, 2023.
  15. Hyperbolic entailment cones for learning hierarchical embeddings. In International Conference on Machine Learning, 2018.
  16. Hyperbolic neural networks. In Advances in Neural Information Processing Systems, 2018.
  17. Curvature generation in curved spaces for few-shot learning. In International Conference on Computer Vision, 2021.
  18. Hyperbolic busemann learning with ideal prototypes. In Advances in Neural Information Processing Systems, 2021.
  19. Hyperbolic image segmentation. In Computer Vision and Pattern Recognition, 2022.
  20. Clipped hyperbolic classifiers are super-hyperbolic classifiers. In Computer Vision and Pattern Recognition, 2022.
  21. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In International Conference on Computer Vision, 2015.
  22. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition, 2016.
  23. Capturing implicit hierarchical structure in 3d biomedical images with self-supervised hyperbolic representations. Advances in Neural Information Processing Systems, 2021.
  24. Christian Szegedy Ian Goodfellow, Jonathan Shlens. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
  25. Hyperbolic image embeddings. In Computer Vision and Pattern Recognition, 2020.
  26. Poincaré maps for analyzing complex hierarchies in single-cell data. Nature communications, 11, 2020.
  27. Lorentzian distance learning for hyperbolic representations. In International Conference on Machine Learning, 2019.
  28. Hgan: Hyperbolic generative adversarial network. IEEE Access, 9:96309–96320, 2021.
  29. Hyperbolic graph neural networks. In Advances in neural information processing systems, 2019.
  30. Hyperbolic visual embedding learning for zero-shot recognition. In Computer Vision and Pattern Recognition, 2020.
  31. Energy-based out-of-distribution detection. In Advances in neural information processing systems, 2020.
  32. Searching for actions on the hyperbole. In Computer Vision and Pattern Recognition, 2020.
  33. Teng Long and Nanne van Noord. Cross-modal scalable hyperbolic hierarchical clustering. In ICCV, 2023.
  34. Differentiating through the fréchet mean. In International Conference on Machine Learning, 2020.
  35. Adaptive poincaré point to set distance for few-shot classification. In AAAI Conference on Artificial Intelligence, 2022.
  36. Continuous hierarchical representations with poincaré variational auto-encoders. In Advances in neural information processing systems, 2019.
  37. Riemannian continuous normalizing flows. Advances in Neural Information Processing Systems, 2020.
  38. Hyperbolic deep learning in computer vision: A survey. arXiv preprint arXiv:2305.06611, 2023.
  39. Gradient-based hierarchical clustering using continuous representations of trees in hyperbolic space. In International Conference on Knowledge Discovery & Data Mining, 2019.
  40. A wrapped normal distribution on hyperbolic space for gradient-based learning. In International Conference on Machine Learning, 2019.
  41. Reading digits in natural images with unsupervised feature learning. In NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
  42. Poincaré embeddings for learning hierarchical representations. Advances in neural information processing systems, 2017.
  43. Hyperbolic deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44:10023 – 10044, 2022.
  44. Representation tradeoffs for hyperbolic embeddings. In International conference on machine learning, 2018.
  45. Rik Sarkar. Low distortion delaunay embedding of trees in hyperbolic plane. In International Conference on Graph Drawing, 2011.
  46. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, 2017.
  47. Hyperbolic neural networks++. In International Conference on Learning Representations, 2021.
  48. Learning the predictability of the future. In Computer Vision and Pattern Recognition, 2021.
  49. Poincar\\\backslash\’e glove: Hyperbolic word embeddings. In International Conference on Learning Representations, 2019.
  50. Abraham Albert Ungar. A Gyrovector Space Approach to Hyperbolic Geometry. Springer Cham, 2009.
  51. Abhinav Valada. On hyperbolic embeddings in object detection. In German Conference on Pattern Recognition, 2022.
  52. HypLL: The hyperbolic learning library. In ACM MM 2023 Open Source Competition, 2023.
  53. Unsupervised discovery of the long-tail in instance segmentation using hierarchical self-supervision. In Computer Vision and Pattern Recognition, 2021.
  54. Meta hyperbolic networks for zero-shot learning. Neurocomputing, 491:57–66, 2022.
  55. Unsupervised hyperbolic metric learning. In Computer Vision and Pattern Recognition, 2021.
  56. Skin lesion recognition with class-hierarchy regularized hyperbolic embeddings. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2022.
  57. Hyperbolic knowledge transfer with class hierarchy for few-shot learning. In International Joint Conference on Artificial Intelligence, 2022.
  58. Hippocampal spatial representations exhibit a hyperbolic geometry that expands with experience. Nature Neuroscience, 26:131–139, 2022.
  59. Learning better visual data similarities via new grouplet non-euclidean embedding. In International Conference on Computer Vision, 2021.
  60. Hyperbolic graph attention network. IEEE Transactions on Big Data, 8:1690–1701, 2021.
  61. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40:1452–1464, 2017.
Citations (9)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Paper to Video (Beta)

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up