Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
153 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Galois theorem for machine learning: Functions on symmetric matrices and point clouds via lightweight invariant features (2405.08097v3)

Published 13 May 2024 in cs.LG and math.AC

Abstract: In this work, we present a mathematical formulation for machine learning of (1) functions on symmetric matrices that are invariant with respect to the action of permutations by conjugation, and (2) functions on point clouds that are invariant with respect to rotations, reflections, and permutations of the points. To achieve this, we provide a general construction of generically separating invariant features using ideas inspired by Galois theory. We construct $O(n2)$ invariant features derived from generators for the field of rational functions on $n\times n$ symmetric matrices that are invariant under joint permutations of rows and columns. We show that these invariant features can separate all distinct orbits of symmetric matrices except for a measure zero set; such features can be used to universally approximate invariant functions on almost all weighted graphs. For point clouds in a fixed dimension, we prove that the number of invariant features can be reduced, generically without losing expressivity, to $O(n)$, where $n$ is the number of points. We combine these invariant features with DeepSets to learn functions on symmetric matrices and point clouds with varying sizes. We empirically demonstrate the feasibility of our approach on molecule property regression and point cloud distance prediction.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (130)
  1. The generalized method of moments for multi-reference alignment. IEEE Transactions on Signal Processing, 70:1377–1388, 2022.
  2. Invariants of s 4 and the shape of sets of vectors. Applicable Algebra in Engineering, Communication and Computing, 7:53–57, 1996.
  3. The hardness of approximation of euclidean k-means. In 31st International Symposium on Computational Geometry (SoCG 2015). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2015.
  4. Neural injective functions for multisets, measures and graphs via a finite witness theorem. Advances in Neural Information Processing Systems, 36, 2024.
  5. Sample complexity of the Boolean multireference alignment problem. In 2017 IEEE International Symposium on Information Theory (ISIT), pages 1316–1320. IEEE, 2017.
  6. Group invariant machine learning by fundamental domain projections. In NeurIPS Workshop on Symmetry and Geometry in Neural Representations, pages 181–218. PMLR, 2023.
  7. k-means++: The advantages of careful seeding. In Soda, volume 7, pages 1027–1035, 2007.
  8. László Babai. Graph isomorphism in quasipolynomial time. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing, pages 684–697, 2016.
  9. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478, 2021.
  10. Single-particle cryo-electron microscopy: Mathematical theory, computational challenges, and opportunities. IEEE signal processing magazine, 37(2):58–76, 2020.
  11. Estimation under group actions: recovering orbits from invariants. Applied and Computational Harmonic Analysis, 66:236–319, 2023.
  12. Mrec: a fast and versatile framework for aligning and matching point clouds with applications to single cell molecular data. arXiv preprint arXiv:2001.01666, 2020.
  13. Multireference alignment using semidefinite programming. In Proceedings of the 5th Conference on Innovations in Theoretical Computer Science, pages 459–470. ACM, 2014.
  14. The sample complexity of sparse multireference alignment and single-particle cryo-electron microscopy. SIAM Journal on Mathematics of Data Science, 6(2):254–282, 2024.
  15. Dihedral multi-reference alignment. IEEE Transactions on Information Theory, 68(5):3489–3499, 2022.
  16. Equivariant subgraph aggregation networks. arXiv preprint arXiv:2110.02910, 2021.
  17. Improving graph neural network expressivity via subgraph isomorphism counting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  18. Improving approximate optimal transport distances using quantization. In Uncertainty in artificial intelligence, pages 290–300. PMLR, 2021.
  19. Permutation invariant representations with applications to graph deep learning. arXiv preprint arXiv:2203.07546, 2022.
  20. On reconstructing n-point configurations from the distribution of distances or areas. Advances in Applied Mathematics, 32(4):709–735, 2004.
  21. Supervised training of conditional monge maps. Advances in Neural Information Processing Systems, 35:6859–6872, 2022.
  22. Sparse multi-reference alignment: Sample complexity and computational hardness. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8977–8981. IEEE, 2022.
  23. Optimal rates of estimation for multi-reference alignment. Mathematical Statistics and Learning, 2(1):25–75, 2020.
  24. 970 million druglike small molecules for virtual screening in the chemical universe database GDB-13. J. Am. Chem. Soc., 131:8732, 2009.
  25. Glen E Bredon. Introduction to compact transformation groups. Academic press, 1972.
  26. Probabilistic symmetries and invariant neural networks. Journal of Machine Learning Research, 21(90):1–61, 2020.
  27. Degree bounds for fields of rational invariants of ℤ/p⁢ℤℤ𝑝ℤ\mathbb{Z}/p\mathbb{Z}blackboard_Z / italic_p blackboard_Z and other finite groups. Journal of Pure and Applied Algebra, 228(10):107693, 2024.
  28. Machine learning and invariant theory. Notices of the American Mathematical Society, 70(8):1205–1213, 2022.
  29. G-invariant representations using coorbits: Bi-lipschitz properties. arXiv preprint arXiv:2308.11784, 2023.
  30. G-invariant representations using coorbits: Injectivity properties. arXiv preprint arXiv:2310.16365, 2023.
  31. Complete set of translation invariant measurements with lipschitz bounds. Applied and Computational Harmonic Analysis, 49(2):521–539, 2020.
  32. Stable separation of orbits for finite abelian group actions. Journal of Fourier Analysis and Applications, 30(1):12, 2024.
  33. Can graph neural networks count substructures? Advances in neural information processing systems, 33:10383–10395, 2020.
  34. A general theory of equivariant CNNs on homogeneous spaces. Advances in neural information processing systems, 32, 2019.
  35. Bilipschitz group invariants. arXiv preprint arXiv:2305.17241, 2023.
  36. Group-invariant max filtering. arXiv preprint arXiv:2205.14039, 2022.
  37. Equivariant point network for 3d point cloud analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14514–14523, 2021.
  38. Aldo Conca. Gröbner bases of ideals of minors of a symmetric matrix. Journal of Algebra, 166:406–421, 1994.
  39. SE⁢(3)SE3\mathrm{SE}(3)roman_SE ( 3 )-equivariant attention networks for shape reconstruction in function space. In The Eleventh International Conference on Learning Representations, 2023.
  40. Group equivariant convolutional networks. In International conference on machine learning, pages 2990–2999. PMLR, 2016.
  41. George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303–314, 1989.
  42. Linear regression model with histogram-valued variables. Statistical Analysis and Data Mining: The ASA Data Science Journal, 8(2):75–113, 2015.
  43. Low-dimensional invariant embeddings for universal geometric learning. Foundations of Computational Mathematics, pages 1–41, 2024.
  44. Computational invariant theory. Springer, 2015.
  45. Vector neurons: A general framework for so (3)-equivariant networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12200–12209, 2021.
  46. On the universality of rotation equivariant point cloud networks. arXiv preprint arXiv:2010.02449, 2020.
  47. Emilie Dufresne. Separating invariants. PhD thesis, Queen’s University, 2008.
  48. Orbit recovery for band-limited functions. arXiv preprint arXiv:2306.00155, 2023.
  49. Provably strict generalisation benefit for equivariant models. In International Conference on Machine Learning, pages 2959–2969. PMLR, 2021.
  50. Understanding and extending subgraph gnns by rethinking their symmetries. arXiv preprint arXiv:2206.11140, 2022.
  51. Maximum likelihood for high-noise group orbit estimation and single-particle cryo-EM. arXiv preprint arXiv:2107.01305, 2021.
  52. Se (3)-transformers: 3d roto-translation equivariant attention networks. Advances in neural information processing systems, 33:1970–1981, 2020.
  53. Lorentz-and permutation-invariants of particles. Journal of Physics A: Mathematical and Theoretical, 54(15):155201, 2021.
  54. e3nn: Euclidean neural networks. arXiv preprint arXiv:2207.09453, 2022.
  55. Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263–1272. PMLR, 2017.
  56. Ward Haddadin. Invariant polynomials and machine learning. arXiv preprint arXiv:2104.12733, 2021.
  57. Weisfeiler leman for euclidean equivariant machine learning, 2024.
  58. Complete neural networks for euclidean graphs. arXiv preprint arXiv:2301.13821, 2023.
  59. William L Hamilton. Graph representation learning. Morgan & Claypool Publishers, 2020.
  60. Perspectives on CUR decompositions. Applied and Computational Harmonic Analysis, 48(3):1088–1099, 2020.
  61. Rational invariants of a group action. construction and rewriting. Journal of Symbolic Computation, 42(1-2):203–217, 2007.
  62. Rational invariants of scalings from hermite normal forms. In Proceedings of the 37th International Symposium on Symbolic and Algebraic Computation, pages 219–226, 2012.
  63. Computation of invariants of finite abelian groups. Mathematics of Computation, 85(302):3029–3050, 2016.
  64. Equivariant diffusion for molecule generation in 3d. In International conference on machine learning, pages 8867–8887. PMLR, 2022.
  65. A short tutorial on the Weisfeiler-Lehman test and its variants. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8533–8537. IEEE, 2021.
  66. From local to global: Spectral-inspired graph neural networks. In NeurIPS 2022 Workshop: New Frontiers in Graph Learning, 2022.
  67. Lsst: from science drivers to reference design and anticipated data products. The Astrophysical Journal, 873(2):111, 2019.
  68. Probably certifiably correct k-means clustering. Mathematical Programming, 165:605–642, 2017.
  69. Linear regression for numeric symbolic variables: a least squares approach based on wasserstein distance. Advances in Data Analysis and Classification, 9(1):81–106, 2015.
  70. Gregor Kemper. The computation of invariant fields and a constructive version of a theorem by rosenlicht. Transformation Groups, 12:657–670, 2007.
  71. Gregor Kemper. Using extended derksen ideals in computational invariant theory. Journal of symbolic computation, 72:161–181, 2016.
  72. Gregor Kemper. Invariant theory: a third lease of life. arXiv preprint arXiv:2403.12709, 2024.
  73. Algorithmic invariant theory of nonreductive groups. Qualitative Theory of Dynamical Systems, 11(1):79–110, 2012.
  74. Separating invariants over finite fields. Journal of Pure and Applied Algebra, 226(4):106904, 2022.
  75. Clebsch–gordan nets: a fully fourier space spherical convolutional neural network. Advances in Neural Information Processing Systems, 31, 2018.
  76. Equivariance with learned canonicalization functions. In International Conference on Machine Learning, pages 15546–15566. PMLR, 2023.
  77. Universal invariant and equivariant graph neural networks. Advances in Neural Information Processing Systems, 32, 2019.
  78. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International conference on machine learning, pages 2747–2755. PMLR, 2018.
  79. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.
  80. Sign and basis invariant networks for spectral graph representation learning. In The Eleventh International Conference on Learning Representations, 2022.
  81. Calculating generators for invariant fields of linear algebraic groups. In Applied Algebra, Algebraic Algorithms and Error-Correcting Codes: 13th International Symposium, AAECC-13 Honolulu, Hawaii, USA, November 15–19, 1999 Proceedings 13, pages 392–403. Springer, 1999.
  82. Invariant and equivariant graph networks. In International Conference on Learning Representations, 2018.
  83. Provably powerful graph networks. Advances in neural information processing systems, 32, 2019.
  84. Facundo Mémoli. Gromov–wasserstein distances and the metric approach to object matching. Foundations of computational mathematics, 11:417–487, 2011.
  85. On the universality of invariant networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 4363–4371. PMLR, 09–15 Jun 2019.
  86. Weisfeiler and leman go machine learning: The story so far. Journal of Machine Learning Research, 24(333):1–59, 2023.
  87. Max filtering with reflection groups. Advances in Computational Mathematics, 49(6):82, 2023.
  88. Injectivity, stability, and positive definiteness of max filtering. arXiv preprint arXiv:2212.11156, 2022.
  89. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 4602–4609, 2019.
  90. Tetrasphere: A neural descriptor for o⁢(3)𝑜3o(3)italic_o ( 3 )-invariant point cloud analysis. arXiv preprint arXiv:2211.14456, 2022.
  91. Machine learning of molecular electronic properties in chemical compound space. New Journal of Physics, 15(9):095003, 2013.
  92. Clustering subgaussian mixtures by semidefinite programming. Information and Inference: A Journal of the IMA, 6(4):389–415, 2017.
  93. Peter J Olver. Invariants of finite and discrete group actions via moving frames. Bulletin of the Iranian Mathematical Society, 49(2):1–12, 2023.
  94. Lie groups and algebraic groups. Springer Science & Business Media, 2012.
  95. Frame averaging for invariant and equivariant network design. arXiv preprint arXiv:2110.03336, 2021.
  96. Incompleteness of graph neural networks for points clouds in three dimensions, 2022.
  97. Smooth, exact rotational symmetrization for deep learning on point clouds. Advances in Neural Information Processing Systems, 36, 2024.
  98. Equivariant polynomials for graph neural networks. In International Conference on Machine Learning, pages 28191–28222. PMLR, 2023.
  99. Effective rotation-invariant point cnn with spherical harmonics kernels. In 2019 International Conference on 3D Vision (3DV), pages 47–56. IEEE, 2019.
  100. Invariant theory. Algebraic Geometry IV: Linear Algebraic Groups Invariant Theory, pages 123–278, 1994.
  101. The sample complexity of multireference alignment. SIAM Journal on Mathematics of Data Science, 1(3):497–517, 2019.
  102. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.
  103. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
  104. Three iterations of (1−d)1𝑑(1-d)( 1 - italic_d )-wl test distinguish non isometric clouds of d𝑑ditalic_d-dimensional points. arXiv preprint arXiv:2303.12853, 2023.
  105. Maxwell Rosenlicht. Some basic theorems on algebraic groups. American Journal of Mathematics, 78(2):401–443, 1956.
  106. Equivariance through parameter-sharing. In International conference on machine learning, pages 2892–2901. PMLR, 2017.
  107. Fast and accurate modeling of molecular atomization energies with machine learning. Physical review letters, 108(5):058301, 2012.
  108. Kate Storey-Fisher. Tools for Cosmology with Galaxy Clustering. PhD thesis, New York University, 2023.
  109. E(n) equivariant graph neural networks. In International conference on machine learning, pages 9323–9332. PMLR, 2021.
  110. Fred J Sigworth. Principles of cryo-EM single-particle image processing. Microscopy, 65(1):57–67, 2016.
  111. Amit Singer. Mathematics for cryo-electron microscopy. In Proceedings of the International Congress of Mathematicians: Rio de Janeiro 2018, pages 3995–4014. World Scientific, 2018.
  112. Three-dimensional structure determination from common lines in cryo-EM by eigenvectors and semidefinite programming. SIAM Journal on Imaging Sciences, 4(2):543–572, 2011.
  113. Nicolas M Thiéry. Algebraic invariants of graphs; a study based on computer exploration. ACM SIGSAM Bulletin, 34(3):9–20, 2000.
  114. Universal representation of permutation-invariant functions on vectors and tensors. In 35th Intl. Conf. Algorithmic Learning Theory (ALT), 2024.
  115. Autobahn: Automorphism-based graph neural nets, 2021.
  116. Graph attention networks. stat, 1050(20):10–48550, 2017.
  117. Learning cosmology and clustering with cosmic graphs. The Astrophysical Journal, 937(2):115, 2022.
  118. Scalars are universal: Equivariant machine learning, structured like classical physics. Advances in Neural Information Processing Systems, 34:28848–28863, 2021.
  119. Towards fully covariant machine learning. Transactions on Machine Learning Research, 2023.
  120. Dimensionless machine learning: Imposing exact units equivariance. Journal of Machine Learning Research, 24(109):1–32, 2023.
  121. Hermann Weyl. The classical groups: their invariants and representations. Princeton University Press, 1946.
  122. Universal approximation of functions on sets. Journal of Machine Learning Research, 23(151):1–56, 2022.
  123. Coordinate independent convolutional networks–isometry and gauge equivariant convolutions on riemannian manifolds. arXiv preprint arXiv:2106.06020, 2021.
  124. Recognizing rigid patterns of unlabeled point clouds by complete and continuous isometry invariants with no false negatives and no false positives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1275–1284, 2023.
  125. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9(2):513–530, 2018.
  126. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912–1920, 2015.
  127. How powerful are graph neural networks? In International Conference on Learning Representations, 2018.
  128. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv preprint arXiv:2203.02923, 2022.
  129. Deep sets. Advances in neural information processing systems, 30, 2017.
  130. Artificial intelligence for science in quantum, atomistic, and continuum systems. arXiv preprint arXiv:2307.08423, 2023.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now