Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
133 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

The Numerical Stability of Hyperbolic Representation Learning (2211.00181v4)

Published 31 Oct 2022 in cs.LG, cs.NA, and math.NA

Abstract: Given the exponential growth of the volume of the ball w.r.t. its radius, the hyperbolic space is capable of embedding trees with arbitrarily small distortion and hence has received wide attention for representing hierarchical datasets. However, this exponential growth property comes at a price of numerical instability such that training hyperbolic learning models will sometimes lead to catastrophic NaN problems, encountering unrepresentable values in floating point arithmetic. In this work, we carefully analyze the limitation of two popular models for the hyperbolic space, namely, the Poincar\'e ball and the Lorentz model. We first show that, under the 64 bit arithmetic system, the Poincar\'e ball has a relatively larger capacity than the Lorentz model for correctly representing points. Then, we theoretically validate the superiority of the Lorentz model over the Poincar\'e ball from the perspective of optimization. Given the numerical limitations of both models, we identify one Euclidean parametrization of the hyperbolic space which can alleviate these limitations. We further extend this Euclidean parametrization to hyperbolic hyperplanes and exhibits its ability in improving the performance of hyperbolic SVM.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Riemannian adaptive optimization methods. In International Conference on Learning Representations, 2018.
  2. Bonnabel, S. Stochastic gradient descent on Riemannian manifolds. IEEE Transactions on Automatic Control, 58(9):2217–2229, 2013.
  3. Scalable hyperbolic recommender systems. arXiv preprint arXiv:1902.08648, 2019.
  4. Fully hyperbolic neural networks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.  5672–5686, 2022.
  5. Highly scalable and provably accurate classification in Poincaré balls. In 2021 IEEE International Conference on Data Mining (ICDM), pp.  61–70. IEEE, 2021.
  6. Large-margin classification in hyperbolic space. In The 22nd international conference on artificial intelligence and statistics, pp.  1832–1840. PMLR, 2019.
  7. Hyperbolic neural networks. Advances in neural information processing systems, 31, 2018.
  8. Poincaré embedding reveals edge-based functional networks of the brain. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2020, pp.  448–457. Springer, 2020.
  9. Graham, R. L. An efficient algorithm for determining the convex hull of a finite planar set. Info. Pro. Lett., 1:132–133, 1972.
  10. Clipped hyperbolic classifiers are super-hyperbolic classifiers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  11–20, 2022.
  11. A no-go theorem for robust acceleration in the hyperbolic plane. Advances in Neural Information Processing Systems, 34:3914–3924, 2021.
  12. Hyperbolic image embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  6418–6428, 2020.
  13. Geoopt: Riemannian optimization in pytorch. arXiv preprint arXiv:2005.02819, 2020.
  14. Learning multiple layers of features from tiny images. 2009.
  15. Hierarchical differentiation of myeloid progenitors is encoded in the transcription factor network. PloS one, 6(8):e22649, 2011.
  16. Lorentzian distance learning for hyperbolic representations. In International Conference on Machine Learning, pp. 3672–3681. PMLR, 2019.
  17. A fully hyperbolic neural model for hierarchical multi-class classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp.  460–475, 2020.
  18. Fine-grained entity typing in hyperbolic space. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pp.  169–180, 2019.
  19. Continuous hierarchical representations with Poincaré variational auto-encoders. Advances in neural information processing systems, 32, 2019.
  20. Decoding the regulatory network of early blood development from single-cell gene expression measurements. Nature biotechnology, 33(3):269–276, 2015.
  21. Poincaré embeddings for learning hierarchical representations. Advances in neural information processing systems, 30, 2017.
  22. Learning continuous hierarchies in the Lorentz model of hyperbolic geometry. In International Conference on Machine Learning, pp. 3779–3788. PMLR, 2018.
  23. Single-cell analysis of mixed-lineage states leading to a binary cell fate choice. Nature, 537(7622):698–702, 2016.
  24. Automatic differentiation in pytorch. 2017.
  25. Transcriptional heterogeneity and lineage commitment in myeloid progenitors. Cell, 163(7):1663–1677, 2015.
  26. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011.
  27. Hyperbolic deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
  28. Platt, J. et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61–74, 1999.
  29. Representation tradeoffs for hyperbolic embeddings. In International conference on machine learning, pp. 4460–4469. PMLR, 2018.
  30. Sarkar, R. Low distortion Delaunay embedding of trees in hyperbolic plane. In International Symposium on Graph Drawing, pp.  355–366. Springer, 2011.
  31. Hyperbolic neural networks++. In International Conference on Learning Representations, 2020.
  32. Mixed-curvature variational autoencoders. In International Conference on Learning Representations, 2019.
  33. Ungar, A. A. Hyperbolic trigonometry and its application in the Poincaré ball model of hyperbolic geometry. Computers & Mathematics with Applications, 41(1-2):135–147, 2001.
  34. Gradient descent in hyperbolic space. arXiv preprint arXiv:1805.08207, 2018.
  35. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
  36. Numerically accurate hyperbolic embeddings using tiling-based models. Advances in Neural Information Processing Systems, 32, 2019.
  37. Hypertext: Endowing fasttext with hyperbolic geometry. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp.  1166–1171, 2020.
Citations (18)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now