Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Riemannian Proximal Newton Method (2304.04032v3)

Published 8 Apr 2023 in math.OC

Abstract: In recent years, the proximal gradient method and its variants have been generalized to Riemannian manifolds for solving optimization problems with an additively separable structure, i.e., $f + h$, where $f$ is continuously differentiable, and $h$ may be nonsmooth but convex with computationally reasonable proximal mapping. In this paper, we generalize the proximal Newton method to embedded submanifolds for solving the type of problem with $h(x) = \mu |x|_1$. The generalization relies on the Weingarten and semismooth analysis. It is shown that the Riemannian proximal Newton method has a local quadratic convergence rate under certain reasonable assumptions. Moreover, a hybrid version is given by concatenating a Riemannian proximal gradient method and the Riemannian proximal Newton method. It is shown that if the switch parameter is chosen appropriately, then the hybrid method converges globally and also has a local quadratic convergence rate. Numerical experiments on random and synthetic data are used to demonstrate the performance of the proposed methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. Optimization algorithms on matrix manifolds. Princeton University Press, Princeton, NJ, 2008.
  2. An extrinsic look at the Riemannian Hessian. In International conference on geometric science of information, pages 361–368. Springer, 2013.
  3. The rate of convergence of Nesterov’s accelerated forward-backward method is actually faster than 1/k21superscript𝑘21/k^{2}1 / italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT. SIAM Journal on Optimization, 26:1824–1834, 2016.
  4. Amir Beck. First-order methods in optimization. SIAM, 2017.
  5. Large-scale sparse inverse covariance matrix estimation. SIAM Journal on Scientific Computing, 41:A380–A401, 2019.
  6. Nicolas Boumal. An introduction to optimization on smooth manifolds. Cambridge University Press, 2023.
  7. Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems. IEEE Transactions on Image Processing, 18:2419–2434, 2009.
  8. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences, 2:183–202, 2009.
  9. Frank H Clarke. Optimization and nonsmooth analysis. SIAM, 1990.
  10. Proximal gradient method for nonsmooth optimization over the Stiefel manifold. SIAM Journal on Optimization, 30:210–239, 2020.
  11. Solving partial least squares regression via manifold optimization approaches. IEEE Transactions on Neural Networks and Learning Systems, 30:588–600, 2 2019.
  12. A direct formulation for sparse PCA using semidefinite programming. In Advances in Neural Information Processing Systems, volume 17. MIT Press, 2004.
  13. Implicit functions and solution mappings: a view from variational analysis, volume 11. Springer, 2009.
  14. ε𝜀\varepsilonitalic_ε-subgradient algorithms for locally Lipschitz functions on Riemannian manifolds. Advances in Computational Mathematics, 42:333–360, 2016.
  15. M. Seetharama Gowda. Inverse and implicit function theorems for H-differentiable and semismooth functions. Optimization Methods and Software, 19:443–461, 10 2004.
  16. Intrinsic representation of tangent vectors and vector transports on matrix manifolds. Numerische Mathematik, 136:523–543, 6 2017.
  17. A proximal bundle algorithm for nonsmooth optimization on Riemannian manifolds. IMA Journal of Numerical Analysis, 43(1):293–325, 12 2021.
  18. S. Hosseini and A. Uschmajew. A Riemannian gradient sampling algorithm for nonsmooth optimization on manifolds. SIAM Journal on Optimization, 27(1):173–189, 2017.
  19. Wen Huang. Optimization algorithms on Riemannian manifolds with applications. PhD thesis, The Florida State University, 2013.
  20. Wen Huang and Ke Wei. An extension of fast iterative shrinkage-thresholding algorithm to Riemannian optimization for sparse principal component analysis. Numerical Linear Algebra with Applications, 29, 1 2022.
  21. Wen Huang and Ke Wei. Riemannian proximal gradient methods. Mathematical Programming, 194:371–413, 7 2022.
  22. Wen Huang and Ke Wei. An inexact Riemannian proximal gradient method. Computational Optimization and Applications, 2023.
  23. A Riemannian optimization approach to clustering problems, 2022.
  24. Norman H Josephy. Newton’s method for generalized equations. Technical report, Wisconsin Univ-Madison Mathematics Research Center, 1979.
  25. Norman H Josephy. Quasi-Newton methods for generalized equations. Technical report, Wisconsin Univ-madison Mathematics Research Center, 1979.
  26. A modified principal component technique based on the lasso. Journal of Computational and Graphical Statistics, 12:531–547, 9 2003.
  27. The implicit function theorem: history, theory, and applications. Springer Science & Business Media, 2002.
  28. IMRO: A proximal quasi-Newton method for solving ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-regularized least squares problems. SIAM Journal on Optimization, 27:583–615, 2017.
  29. Weakly convex optimization over Stiefel manifold using Riemannian subgradient-type methods. SIAM Journal on Optimization, 31(3):1605–1634, 2021.
  30. John M. Lee. Introduction to Smooth Manifolds. Springer, New York, NY, 2012.
  31. John M Lee. Introduction to Riemannian manifolds, volume 2. Springer, 2018.
  32. Accelerated proximal gradient methods for nonconvex programming. In Advances in Neural Information Processing Systems, 2015.
  33. Proximal Newton-type methods for minimizing composite functions. SIAM Journal on Optimization, 24:1420–1443, 2014.
  34. On efficiently solving the subproblems of a level-set method for fused lasso problems. SIAM Journal on Optimization, 28:1842–1866, 2018.
  35. Convex sparse spectral clustering: single-view to multi-view. IEEE Transactions on Image Processing, 25:2833–2843, 6 2016.
  36. A globally convergent proximal Newton-type method in nonsmooth convex optimization. Mathematical Programming, 198:899–936, 3 2023.
  37. Yurii Nesterov. A method for solving the convex programming problem with convergence rate 𝒪⁢(1/k2)𝒪1superscript𝑘2\mathcal{O}(1/k^{2})caligraphic_O ( 1 / italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ). Proceedings of the USSR Academy of Sciences, 269:543–547, 1983.
  38. Yurii Nesterov. Lectures on convex optimization, volume 137. Springer, 2018.
  39. Compressed modes for variational problems in mathematics and physics. Proceedings of the National Academy of Sciences, 110:18368–18373, 11 2013.
  40. A sparse proximal Newton splitting method for constrained image deblurring. Neurocomputing, 122:245–257, 12 2013.
  41. Semismooth homeomorphisms and strong stability of semidefinite and Lorentz complementarity problems. Mathematics of Operations Research, 28(1):39–63, 2003.
  42. Spectral clustering based on learning similarity matrix. Bioinformatics, 34:2069–2076, 6 2018.
  43. A nonsmooth version of Newton’s method. Mathematical Programming, 58:353–367, 1993.
  44. SpaSM: A MATLAB toolbox for sparse statistical modeling. Journal of Statistical Software, 84, 2018.
  45. A proximal Newton-type method for equilibrium problems. Optimization Letters, 12:997–1009, 7 2018.
  46. Defeng Sun. A further result on an implicit function theorem for locally Lipschitz functions. Operations Research Letters, 28:193–198, 2001.
  47. Face recognition by sparse discriminant analysis via joint ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT-norm minimization. Pattern Recognition, 47:2447–2453, 2014.
  48. Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58:267–288, 1 1996.
  49. Michael Ulbrich. Semismooth Newton methods for variational inequalities and constrained optimization problems in function spaces. SIAM, 2011.
  50. Accelerated and inexact forward-backward algorithms. SIAM Journal on Optimization, 23:1607–1633, 2013.
  51. Multitask learning in computational biology. In Proceedings of ICML Workshop on Unsupervised and Transfer Learning, volume 27, pages 207–216. PMLR, 2012.
  52. Proximal quasi-Newton method for composite optimization over the Stiefel manifold. Journal of Scientific Computing, 95, 5 2023.
  53. A regularized semismooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing, 76:364–389, 7 2018.
  54. On the robust isolated calmness of a class of nonsmooth optimizations on Riemannian manifolds and its applications, 2022.
  55. A semismooth Newton based augmented Lagrangian method for nonsmooth optimization on matrix manifolds. Mathematical Programming, pages 1–61, 2022.
  56. Sparse principal component analysis. Journal of Computational and Graphical Statistics, 15:265–286, 6 2006.
  57. On the global geometry of sphere-constrained sparse blind deconvolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4894–4902, 2017.
  58. A selective overview of sparse principal component analysis. Proceedings of the IEEE, 106:1311–1320, 8 2018.
  59. Proximal quasi-Newton for computationally intensive ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-regularized M𝑀Mitalic_M-estimators. In Advances in Neural Information Processing Systems, 2014.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now