Papers
Topics
Authors
Recent
Search
2000 character limit reached

Sequential subspace methods on Stiefel manifold optimization problems

Published 20 Apr 2024 in math.OC | (2404.13301v1)

Abstract: We study the minimization of a quadratic over Stiefel manifolds (the set of all orthogonal $r$-frames in \IRn), which has applications in high-dimensional semi-supervised classification tasks. To reduce the computational complexity, sequential subspace methods(SSM) are employed to convert the high-dimensional minimization problems to low-dimensional ones. In this paper, we are interested in attaining an optimal solution of good quality, i.e., a qualified" critical point. Qualified critical points are those critical points, at which the associated multiplier matrix meets some upper bound condition. These critical points enjoy the global optimality in special quadratic problems. For a general quadratic, SSM computes a sequence ofqualified critical points" in its low-dimensional ``surrogate regularized models". The convergence to a qualified critical point is ensured, whenever each SSM subspace is constructed by the following vectors: (i) a set of orthogonal unit vectors associated with the current iterate, (ii) a set of vectors corresponding to the gradient of the objective, and (iii) a set of eigenvectors associated with the smallest $r$ eigenvalues of the system matrix. In addition, when Newton direction vectors are included in subspaces, the convergence of SSM can be accelerated significantly.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Faster ica under orthogonal constraint. 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4464–4468, 2017.
  2. Projection-like retractions on matrix manifolds. SIAM J. Optim., 22(1):135–158, 2012-01.
  3. Optimization Algorithms on Matrix Manifolds. 2008.
  4. Unitary evolution recurrent neural networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML’16, page 1120–1128. JMLR.org, 2016.
  5. Infeasible deterministic, stochastic, and variance-reduction algorithms for optimization under orthogonality constraints. ArXiv, abs/2303.16510, 2023.
  6. Can we gain more from orthogonality regularizations in training deep cnns? In Neural Information Processing Systems, 2018.
  7. Dimitri P. Bertsekas. On the goldstein-levitin-polyak gradient projection method. IEEE Transactions on Automatic Control, 21(2):174–184, 1976.
  8. D.P. Bertsekas. Nonlinear Programming. Athena scientific optimization and computation series. Athena Scientific, 2016.
  9. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation, 15(6):1373–1396, 2003.
  10. Semi-supervised learning on riemannian manifolds. Machine Learning, 56:209–239, 2004.
  11. Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of Machine Learning Research, 7(85):2399–2434, 2006.
  12. Nicolas Boumal. An introduction to optimization on smooth manifolds. 2022-04. Published: To appear with Cambridge University Press.
  13. Trust Region Methods. MPS-SIAM Series on Optimization. Society for Industrial and Applied Mathematics (SIAM, 3600 Market Street, Floor 6, Philadelphia, PA 19104), 2000.
  14. F. R. K. Chung. Spectral Graph Theory. American Mathematical Society, 1997.
  15. Fan Chung. Random walks and local cuts in graphs. Linear Algebra and its Applications, 423(1):22–32, 2007. Special Issue devoted to papers presented at the Aveiro Workshop on Graph Spectra.
  16. Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion maps. Proceedings of the National Academy of Sciences, 102(21):7426–7431, 2005.
  17. Pierre Comon. Independent component analysis, a new concept? Signal Processing, 36(3):287–314, 1994. Higher Order Statistics.
  18. Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29(6):141–142, 2012.
  19. The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. & Appl., 20(2):303–353, 1998-01.
  20. A procrustes problem on the stiefel manifold. Numerische Mathematik, 82(4):599–619, 1999.
  21. A. A. Goldstein. Convex programming in Hilbert space. Bulletin of the American Mathematical Society, 70(5):709 – 710, 1964.
  22. William W. Hager. Minimizing a quadratic over a sphere. SIAM J. Optim., 12:188–208, 2001.
  23. Matrix Analysis. Cambridge University Press, 1985.
  24. Global convergence of SSM for minimizing a quadratic over a sphere. Math. Comput., 74:1413–1423, 2005.
  25. Principal component analysis: a review and recent developments. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 374(2065):20150202, 2016.
  26. Andrew V. Knyazev. Toward the optimal preconditioned eigensolver: Locally optimal block preconditioned conjugate gradient method. SIAM Journal on Scientific Computing, 23(2):517–541, 2001.
  27. Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
  28. Constrained minimization methods. USSR Computational Mathematics and Mathematical Physics, 6:1–50, 12 1966.
  29. Linear and Nonlinear Programming. Springer International Publishing, 2016.
  30. J.H. Manton. Optimization algorithms exploiting unitary constraints. IEEE Transactions on Signal Processing, 50(3):635–650, 2002.
  31. A metric on shape space with explicit geodesics. arXiv: Differential Geometry, 2007.
  32. Y. Nishimori. Learning algorithm for independent component analysis by geodesic flows on orthogonal group. IJCNN’99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).
  33. Numerical Optimization. Springer, 2006.
  34. Hiroyuki Sato. Riemannian conjugate gradient methods: General framework and specific algorithms with convergence analyses. SIAM Journal on Optimization, 32(4):2690–2717, 2022.
  35. Peter H. Schönemann. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31:1–10, 1966.
  36. Ohad Shamir. A stochastic pca and svd algorithm with an exponential convergence rate. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 144–152, Lille, France, 07–09 Jul 2015. PMLR.
  37. Danny C. Sorensen. Newton’s method with a model trust region modification. SIAM Journal on Numerical Analysis, 19:409–426, 1982.
  38. Nearly linear time algorithms for preconditioning and solving symmetric, diagonally dominant linear systems. SIAM Journal on Matrix Analysis and Applications, 35(3):835–885, 2014.
  39. A global geometric framework for nonlinear dimensionality reduction. Science, 290(5500):2319–2323, 2000.
  40. A feasible method for optimization with orthogonality constraints. Mathematical Programming, 142(1):397–434, 2013.
  41. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
  42. Learning with local and global consistency. In S. Thrun, L. Saul, and B. Schölkopf, editors, Advances in Neural Information Processing Systems, volume 16. MIT Press, 2003.
  43. Conditions for optimal solutions of unbalanced procrustes problem on stiefel manifold. Journal of Computational Mathematics, 25(6):661–671, 2007.
  44. Aet vs. aed: Unsupervised representation learning by auto-encoding transformations rather than data. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2542–2550, 2019.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 3 tweets with 0 likes about this paper.

Sign Up for Free