Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Matrix Manifold Neural Networks++ (2405.19206v1)

Published 29 May 2024 in stat.ML and cs.LG

Abstract: Deep neural networks (DNNs) on Riemannian manifolds have garnered increasing interest in various applied areas. For instance, DNNs on spherical and hyperbolic manifolds have been designed to solve a wide range of computer vision and nature language processing tasks. One of the key factors that contribute to the success of these networks is that spherical and hyperbolic manifolds have the rich algebraic structures of gyrogroups and gyrovector spaces. This enables principled and effective generalizations of the most successful DNNs to these manifolds. Recently, some works have shown that many concepts in the theory of gyrogroups and gyrovector spaces can also be generalized to matrix manifolds such as Symmetric Positive Definite (SPD) and Grassmann manifolds. As a result, some building blocks for SPD and Grassmann neural networks, e.g., isometric models and multinomial logistic regression (MLR) can be derived in a way that is fully analogous to their spherical and hyperbolic counterparts. Building upon these works, we design fully-connected (FC) and convolutional layers for SPD neural networks. We also develop MLR on Symmetric Positive Semi-definite (SPSD) manifolds, and propose a method for performing backpropagation with the Grassmann logarithmic map in the projector perspective. We demonstrate the effectiveness of the proposed approach in the human action recognition and node classification tasks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2007.
  2. Fast and Simple Computations on Tensors with Log-Euclidean Metrics. Technical Report RR-5584, INRIA, 2005.
  3. Geometric Mean and Geodesic Regression on Grassmannians. Linear Algebra and its Applications, 466:83–101, 2015.
  4. A Grassmann Manifold Handbook: Basic Geometry and Computational Aspects. CoRR, abs/2011.13699, 2020. URL https://arxiv.org/abs/2011.13699.
  5. Rank-preserving Geometric Means of Positive Semi-definite Matrices. Linear Algebra and its Applications, 438:3202–3216, 2013.
  6. Riemannian Batch Normalization for SPD Neural Networks. In NeurIPS, pp.  15463–15474, 2019.
  7. ManifoldNet: A Deep Neural Network for Manifold-valued Data with Applications. TPAMI, 44(2):799–810, 2020.
  8. Hyperbolic Graph Convolutional Neural Networks. CoRR, abs/1910.12933, 2019. URL https://arxiv.org/abs/1910.12933.
  9. Fully Hyperbolic Neural Networks. In ACL, pp.  5672–5686, 2022.
  10. Computationally Tractable Riemannian Manifolds for Graph Embeddings. In AAAI, pp.  7133–7141, 2021.
  11. A Hyperbolic-to-Hyperbolic Graph Convolutional Network. In CVPR, pp.  154–163, 2021.
  12. Deep Manifold Learning of Symmetric Positive Definite Matrices with Application to Face Recognition. In AAAI, pp.  4009–4015, 2017.
  13. The Geometry of Algorithms with Orthogonality Constraints. SIAM Journal on Matrix Analysis and Applications, 20(2):303–353, 1998.
  14. Hyperbolic neural networks. In NeurIPS, pp.  5350–5360, 2018.
  15. First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations. In CVPR, pp.  409–419, 2018.
  16. Inductive Representation Learning on Large Graphs. In NIPS, pp.  1025–1035, 2017.
  17. Dimensionality Reduction on SPD Manifolds: The Emergence of Geometry-Aware Methods. TPAMI, 40:48–62, 2018.
  18. A Riemannian Network for SPD Matrix Learning. In AAAI, pp.  2036–2042, 2017.
  19. Building Deep Networks on Grassmann Manifolds. In AAAI, pp.  3279–3286, 2018.
  20. Matrix Backpropagation for Deep Networks with Structured Layers. In ICCV, pp.  2965–2973, 2015.
  21. Ce Ju and Cuntai Guan. Graph Neural Networks on SPD Manifolds for Motor Imagery Classification: A Perspective From the Time-Frequency Analysis. IEEE Transactions on Neural Networks and Learning Systems, pp.  1–15, 2023.
  22. Node Embedding from Neural Hamiltonian Orbits in Graph Neural Networks. In ICML, pp.  15786–15808, 2023.
  23. Sejong Kim. Ordered Gyrovector Spaces. Symmetry, 12(6), 2020.
  24. Semi-Supervised Classification with Graph Convolutional Networks. CoRR, abs/1609.02907, 2017. URL https://arxiv.org/abs/1609.02907.
  25. SPD Domain-specific Batch Normalization to Crack Interpretable Unsupervised Domain Adaptation in EEG. In NeurIPS, pp.  6219–6235, 2022.
  26. Hyperplane Margin Classifiers on the Multinomial Manifold. In ICML, pp.  66, 2004.
  27. Zhenhua Lin. Riemannian Geometry of Symmetric Positive Definite Matrices via Cholesky Decomposition. SIAM Journal on Matrix Analysis and Applications, 40(4):1353–1370, 2019.
  28. Hyperbolic Graph Neural Networks. In NeurIPS, pp.  8228–8239, 2019.
  29. Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition. In CVPR, pp.  143–152, 2020.
  30. Vector-valued Distance and Gyrocalculus on the Space of Symmetric Positive Definite Matrices. In NeurIPS, pp.  18350–18366, 2021.
  31. Documentation Mocap Database HDM05. Technical Report CG-2007-2, Universität Bonn, June 2007.
  32. Rectified Linear Units Improve Restricted Boltzmann Machines. In ICML, pp.  807–814, 2010.
  33. Query-driven Active Surveying for Collective Classification. In 10th International Workshop on Mining and Learning with Graphs, volume 8, pp.  1, 2012a.
  34. Query-driven Active Surveying for Collective Classification. In Workshop on Mining and Learning with Graphs, 2012b.
  35. Xuan Son Nguyen. GeomNet: A Neural Network Based on Riemannian Geometries of SPD Matrix Space and Cholesky Space for 3D Skeleton-Based Interaction Recognition. In ICCV, pp.  13379–13389, 2021.
  36. Xuan Son Nguyen. A Gyrovector Space Approach for Symmetric Positive Semi-definite Matrix Learning. In ECCV, pp.  52–68, 2022a.
  37. Xuan Son Nguyen. The Gyro-Structure of Some Matrix Manifolds. In NeurIPS, pp.  26618–26630, 2022b.
  38. Building Neural Networks on Matrix Manifolds: A Gyrovector Space Approach. CoRR, abs/2305.04560, 2023. URL https://arxiv.org/abs/2305.04560.
  39. A Neural Network Based on SPD Manifold Learning for Skeleton-based Hand Gesture Recognition. In CVPR, pp.  12036–12045, 2019.
  40. A Riemannian Framework for Tensor Computing. Technical Report RR-5255, INRIA, 2004.
  41. Riemannian Geometric Statistics in Medical Image Analysis. Academic Press, 2020.
  42. Skeleton-based Action Recognition via Spatial and Temporal Transformer Networks. Computer Vision and Image Understanding, 208:103219, 2021.
  43. Collective Classification in Network Data. AI Magazine, 29(3):93–106, 2008.
  44. NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis. In CVPR, pp.  1010–1019, 2016.
  45. Hyperbolic Neural Networks++. CoRR, abs/2006.08210, 2021. URL https://arxiv.org/abs/2006.08210.
  46. Mixed-curvature Variational Autoencoders. CoRR, abs/1911.08411, 2020. URL https://arxiv.org/abs/1911.08411.
  47. An Interface between Grassmann Manifolds and Vector Spaces. In CVPRW, pp.  3695–3704, 2020.
  48. Neural Architecture Search of SPD Manifold Networks. In IJCAI, pp.  3002–3009, 2021.
  49. Abraham Albert Ungar. Beyond the Einstein Addition Law and Its Gyroscopic Thomas Precession: The Theory of Gyrogroups and Gyrovector Spaces. Fundamental Theories of Physics, vol. 117, Springer, Netherlands, 2002.
  50. Abraham Albert Ungar. Analytic Hyperbolic Geometry: Mathematical Foundations and Applications. World Scientific Publishing Co. Pte. Ltd., Hackensack, NJ, 2005.
  51. Abraham Albert Ungar. Analytic Hyperbolic Geometry in N Dimensions: An Introduction. CRC Press, 2014.
  52. Graph Attention Networks. CoRR, abs/1710.10903, 2018. URL https://arxiv.org/abs/1710.10903.
  53. GrasNet: A Simple Grassmannian Network for Image Set Classification. Neural Processing Letters, 52(1):693–711, 2020.
  54. SymNet: A Simple Symmetric Positive Definite Manifold Deep Learning Method for Image Set Classification. IEEE Transactions on Neural Networks and Learning Systems, pp.  1–15, 2021.
  55. Link Prediction Based on Graph Neural Networks. CoRR, abs/1802.09691, 2018. URL https://arxiv.org/abs/1802.09691.
  56. Lorentzian Graph Convolutional Networks. In Proceedings of the Web Conference 2021, pp.  1249–1261, 2021.
  57. Hyperbolic Graph Attention Network. IEEE Transactions on Big Data, 8(6):1690–1701, 2022.
  58. Modeling Graphs Beyond Hyperbolic: Graph Neural Networks in Symmetric Positive Definite Matrices. CoRR, abs/2306.14064, 2023. URL https://arxiv.org/abs/2306.14064.
  59. Embedding Graphs on Grassmann Manifold. Neural Networks, 152:322–331, 2022.
  60. Learning Discriminative Representations for Skeleton Based Action Recognition. In CVPR, pp.  10608–10617, 2023.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets