2000 character limit reached
The Galerkin method beats Graph-Based Approaches for Spectral Algorithms (2306.00742v3)
Published 1 Jun 2023 in cs.LG, cs.AI, and stat.ML
Abstract: Historically, the machine learning community has derived spectral decompositions from graph-based approaches. We break with this approach and prove the statistical and computational superiority of the Galerkin method, which consists in restricting the study to a small set of test functions. In particular, we introduce implementation tricks to deal with differential operators in large dimensions with structured kernels. Finally, we extend on the core principles beyond our approach to apply them to non-linear spaces of functions, such as the ones parameterized by deep neural networks, through loss-based optimization procedures.
- Analysis and Geometry of Markov Diffusion Operators. Springer, 2014.
- Orthogonal polynomials and diffusion operators. Annales de la Faculté des Sciences de Toulouse, 30(5):985–1073, 2021.
- Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. In NeurIPS, 2022.
- VICReg: Variance-invariance-covariance regularization for self-supervised learning. In ICLR, 2022.
- Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation, 15(6):1373–1396, 2003.
- Laplacian eigenmaps and spectral techniques for embedding and clustering. In NeurIPS, 2004.
- Label propagation and quadratic criterion. In Semi-Supervised Learning. MIT Press, 2006.
- Reproducing Kernel Hilbert spaces in Probability and Statistics. Springer Science & Business Media, 2011.
- Machine learning in acoustics: Theory and applications featured. The Journal of the Acoustical Society of America, 146(5):3590–3628, 2019.
- Measure based regularization. In NeurIPS, 2003.
- Sébastien Bubeck. Convex optimization: Algorithms and complexity. Foundations and Trends in Machine Learning, 8(3-4):231–357, 2015.
- How many samples are needed to leverage smoothness? In NeurIPS, 2023.
- Overcoming the curse of dimensionality with Laplacian regularization in semi-supervised learning. In NeurIPS, 2021.
- On minimal variations for unsupervised representation learning. In ICASSP, 2023a.
- The SSL interplay: Augmentations, inductive bias, and generalization. In ICML, 2023b.
- Françoise Chatelin. Spectral Approximation of Linear Operators. Academic Press, 1983.
- Riemannian flow matching on general geometries, 2023. ArXiv preprint 2302.03660.
- A simple framework for contrastive learning of visual representations. In ICLR, 2020.
- Fan Chung. Spectral Graph Theory. American Mathematical Society, 1997.
- Diffusion maps. Applied and Computational Harmonic Analysis, 21(1):5–30, 2006.
- The Theory of Atomic Spectra. Cambridge University Press, 1935.
- Heinz Otto Cordes. Spectral Theory of Linear Differential Operators and Comparison Algebras. Cambridge University Press, 1987.
- Neural eigenfunctions are structured representation learners, 2022. ArXiv preprint 2210.12637.
- Score-based generative modeling with critically-damped Langevin diffusion. In ICLR, 2022.
- Whitening for self-supervised representation learning. In ICML, 2021.
- Fluid Dynamics. Georgii Ivanovich Petrov (on his 100th birthday). Fluid Dynamics, 47:289–291, 2012.
- Spherical Harmonics in p Dimensions. World Scientific Publishing Company, 2014.
- Unsupervised learning methods for molecular simulation data. Chemical Reviews, 121(16):9722–9758, 2021.
- Singular value decomposition and least squares solutions. Numerische Mathematik, 14:403–420, 1970.
- Michael Greenacre. Theory and Applications of Correspondence Analysis. Academic Press, 1984.
- Representations of knowledge in complex systems. Journal of the Royal Statistical Society. Series B (Methodological), 56(4):549–603, 1994.
- A kernel view of the dimensionality reduction of manifolds. In ICML, 2004.
- Solving high-dimensional eigenvalue problems using deep neural networks: A diffusion Monte Carlo like approach. Journal of Computational Physics, 423:109792, 2020.
- Provable guarantees for self-supervised deep learning with spectral contrastive loss. In NeurIPS, 2021.
- Array programming with NumPy. Nature, 585(7825):357–362, 2020.
- Graph Laplacians and their convergence on random neighborhood graphs. Journal of Machine Learning Research, 8:1325–1368, 2007.
- Charles Hermite. Sur la formule d’interpolation de Lagrange. (Extrait d’une lettre de M. Hermite à M. Borchardt). Journal für die reine und angewandte Mathematik, 84:70–79, 1877.
- Contrastive learning can find an optimal basis for approximately view-invariant functions. In ICLR, 2023.
- Ian Jolliffe. Principal Component Analysis. Springer, 2002.
- Vladimir Kondrachov. On certain properties of functions in the spaces Lpsubscript𝐿𝑝{L}_{p}italic_L start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT. Doklady Akademii Nauk SSSR, 48(8):563–565, 1945.
- Predicting what you already know helps: Provable self-supervised learning. In NeurIPS, 2021.
- Stein variational gradient descent: A general purpose Bayesian inference algorithm. In NeurIPS, 2016.
- Kernel methods through the roof: Handling billions of points efficiently. In NeurIPS, 2020.
- Stanislav Minsker. On some extensions of Bernstein’s inequality for self-adjoint operators. Statistics & Probability Letters, 127:111–119, 2017.
- Leon Mirsky. Symmetric gauge functions and unitarily invariant norms. The Quarterly Journal of Mathematics, 11(1):50–59, 1960.
- Kernelized diffusion maps. In COLT, 2023.
- Random features for large-scale kernel machines. In NeurIPS, 2007.
- Generalization properties of learning with random features. In NeurIPS, 2017.
- Less is more: Nyström computational regularization. In NeurIPS, 2015.
- Learning with Kernels: Support Vector Machines, Regularization, Optimization, and beyond. MIT Press, 2001.
- A generalized representer theorem. In COLT, 2001.
- The relationship of DBSCAN to matrix factorization and spectral clustering. In Lernen, Wissen, Daten, Analysen, 2018.
- Mikhail Shubin. Pseudodifferential Operators and Spectral Theory. Springer, 1987.
- Tangent prop - a formalism for specifying selected invariances in an adaptive network. In NeurIPS, 1991.
- Amit Singer. From graph to manifold laplacian: The convergence rate. Applied and Computational Harmonic Analysis, 21(1):128–134, 2006.
- Josef Singer. On the equivalence of the Galerkin and Rayleigh-Ritz methods. The Aeronautical Journal, 66(621):592, 1962.
- Joel Tropp. An introduction to matrix concentration inequalities. Foundations and Trends in Machine Learning, 8(1-2):1–230, 2015.
- Recovering gene interactions from single-cell data using data diffusion. Cell, 174(3):716–729, 2018.
- Charles van Loan. Generalizing the singular value decomposition. Journal on Numerical Analysis, 13(1):76–83, 1976.
- Hermann Weyl. Das asymptotische verteilungsgesetz der eigenwerte linearer partieller differentialgleichungen (mit einer anwendung auf die theorie der hohlraumstrahlung). Journal on Numerical Analysis, 74(4):441–479, 1912.
- Using the Nyström method to speed up kernel machines. In NeurIPS, 2000.
- Barlow twins: Self-supervised learning via redundancy reduction. In ICML, 2021.
- Solving eigenvalue PDEs of metastable diffusion processes using artificial neural networks. Journal of Computational Physics, 465:111377, 2022.
- Ding-Xuan Zhou. Derivative reproducing properties for kernel methods in learning theory. Journal of Computational and Applied Mathematics, 220(1):456–463, 2008.
- Generalized Laplacian eigenmaps. In NeurIPS, 2022.
- Contrastive Laplacian eigenmaps. In NeurIPS, 2021.
- Semi-supervised learning using Gaussian fields and harmonic functions. In ICML, 2003.
Sponsored by Paperpile, the PDF & BibTeX manager trusted by top AI labs.
Get 30 days freePaper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.