Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Scalable Approximate Optimal Diagonal Preconditioning (2312.15594v2)

Published 25 Dec 2023 in math.NA, cs.NA, and math.OC

Abstract: We consider the problem of finding the optimal diagonal preconditioner for a positive definite matrix. Although this problem has been shown to be solvable and various methods have been proposed, none of the existing approaches are scalable to matrices of large dimension, or when access is limited to black-box matrix-vector products, thereby significantly limiting their practical application. In view of these challenges, we propose practical algorithms applicable to finding approximate optimal diagonal preconditioners of large sparse systems. Our approach is based on the idea of dimension reduction, and combines techniques from semi-definite programming (SDP), random projection, semi-infinite programming (SIP), and column generation. Numerical experiments demonstrate that our method scales to sparse matrices of size greater than $107$. Notably, our approach is efficient and implementable using only black-box matrix-vector product operations, making it highly practical for a wide variety of applications.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. Disciplined quasiconvex programming. Optimization Letters, 14:1643–1657, 2020.
  2. A comparative study of sparse approximate inverse preconditioners. Applied Numerical Mathematics, 30(2-3):305–340, 1999.
  3. A robust incomplete factorization preconditioner for positive definite matrices. Numerical Linear Algebra with Applications, 10(5-6):385–400, 2003.
  4. Minimizing the euclidean condition number. SIAM Journal on Control and Optimization, 32(6):1763–1768, 1994.
  5. Andrew M Bradley. Algorithms for the equilibration of matrices and their application to limited-memory quasi-newton methods. Technical report, STANFORD UNIV CA, 2010.
  6. A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Mathematical programming, 95(2):329–357, 2003.
  7. A modified ssor preconditioner for sparse symmetric indefinite linear systems of equations. International Journal for Numerical Methods in Engineering, 65(6):785–807, 2006.
  8. Minimizing the condition number of a gram matrix. SIAM Journal on optimization, 21(1):127–148, 2011.
  9. An estimate for the condition number of a matrix. SIAM Journal on Numerical Analysis, 16(2):368–375, 1979.
  10. New developments of admm-based interior point methods for linear programming and conic programming. arXiv preprint arXiv:, 2209.01793, 2022.
  11. Precise expressions for random projections: Low-rank approximation and randomized newton. Advances in Neural Information Processing Systems, 33:18272–18283, 2020.
  12. Revisiting spectral bundle methods: Primal-dual (sub) linear convergence rates. SIAM Journal on Optimization, 33(2):1305–1332, 2023.
  13. On the simplicity and conditioning of low rank semidefinite programs. SIAM Journal on Optimization, 31(4):2614–2637, 2021.
  14. An optimal-storage approach to semidefinite programming using approximate complementarity. SIAM Journal on Optimization, 31(4):2695–2725, 2021.
  15. Adaptive lanczos methods for recursive condition estimation. Numerical Algorithms, 1(1):1–19, 1991.
  16. Randomized nyström preconditioning. SIAM Journal on Matrix Analysis and Applications, 44(2):718–752, 2023.
  17. Hdsdp: Software for semidefinite programming. arXiv preprint arXiv:2207.13862, 2022.
  18. Cardinal optimizer (copt) user guide. arXiv preprint arXiv:2208.14314, 2022.
  19. Diagonal scaling in douglas-rachford splitting and admm. In 53rd IEEE Conference on Decision and Control, pages 5033–5039. IEEE, 2014.
  20. Anne Greenbaum and GH Rodrigue. Optimal preconditioners of a given sparsity pattern. BIT Numerical Mathematics, 29(4):610–634, 1989.
  21. LLC Gurobi Optimization. Gurobi optimizer reference manual, 2021.
  22. A spectral bundle method for semidefinite programming. SIAM Journal on Optimization, 10(3):673–696, 2000.
  23. Carl GJ Jacobi. Ueber eine neue auflösungsart der bei der methode der kleinsten quadrate vorkommenden lineären gleichungen. Astronomische Nachrichten, 22(20):297–306, 1845.
  24. Structured semidefinite programming for recovering structured preconditioners. arXiv preprint arXiv:2310.18265, 2023.
  25. Fast and near-optimal diagonal preconditioning. arXiv preprint arXiv:, 2008.01722, 2020.
  26. A fast algorithm for matrix balancing. IMA Journal of Numerical Analysis, 33(3):1029–1047, 2013.
  27. A symmetry preserving algorithm for matrix scaling. SIAM journal on Matrix Analysis and Applications, 35(3):931–955, 2014.
  28. The suitesparse matrix collection website interface. Journal of Open Source Software, 4(35):1244, 2019.
  29. Properties of a cutting plane method for semidefinite programming. submitted for publication, 2003.
  30. Searching for optimal per-coordinate step-sizes with multidimensional backtracking. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  31. Low-rank semidefinite programming: Theory and applications. Foundations and Trends® in Optimization, 2(1-2):1–156, 2016.
  32. Random projections for conic programs. Linear Algebra and its Applications, 626:204–220, 2021.
  33. Minimizing condition number via convex programming. SIAM Journal on Matrix Analysis and Applications, 32(4):1193–1211, 2011.
  34. Selected topics in column generation. Operations research, 53(6):1007–1023, 2005.
  35. Optimizing condition numbers. SIAM Journal on Optimization, 20(2):935–947, 2009.
  36. Scott Menard. Applied logistic regression analysis. Number 106. Sage, 2002.
  37. Conic optimization via operator splitting and homogeneous self-dual embedding. Journal of Optimization Theory and Applications, 169:1042–1068, 2016.
  38. An interior point constraint generation algorithm for semi-infinite optimization with health-care application. Operations research, 59(5):1184–1197, 2011.
  39. A matrix generation approach for eigenvalue optimization. Mathematical programming, 109:155–179, 2007.
  40. Michael L Overton. Large-scale optimization of eigenvalues. SIAM Journal on Optimization, 2(1):88–120, 1992.
  41. Gábor Pataki. On the rank of extreme matrices in semidefinite programs and the multiplicity of optimal eigenvalues. Mathematics of operations research, 23(2):339–358, 1998.
  42. Diagonal preconditioning for first order primal-dual algorithms in convex optimization. In 2011 International Conference on Computer Vision, pages 1762–1769. IEEE, 2011.
  43. Optimal diagonal preconditioning: Theory and practice. arXiv preprint arXiv:2209.00809, 2022.
  44. Daniel Ruiz. A scaling algorithm to equilibrate both rows and columns norms in matrices. Technical report, CM-P00040415, 2001.
  45. Yousef Saad. Iterative methods for sparse linear systems, volume 82. SIAM, 2003.
  46. Alexander Shapiro. Optimally scaled matrices, necessary and sufficient conditions. Numerische Mathematik, 39:239–245, 1982.
  47. Kartik Krishnan Sivaramakrishnan. Linear programming approaches to semidefinite programming problems. Rensselaer Polytechnic Institute, 2002.
  48. Preconditioning via diagonal scaling. arXiv preprint arXiv:1610.03871, 2016.
  49. Practical first order methods for large scale semidefinite programming. Technical report, Technical report, Technical report, University of California, Berkeley, 2014.
  50. John C Urschel. Uniform error estimates for the lanczos method. SIAM Journal on Matrix Analysis and Applications, 42(3):1423–1450, 2021.
  51. Abraham Van der Sluis. Condition numbers and equilibration of matrices. Numerische Mathematik, 14(1):14–23, 1969.
  52. GA Watson. An algorithm for optimal ℓℓ\ellroman_ℓ 2 scaling of matrices. IMA journal of numerical analysis, 11(4):481–492, 1991.
  53. Handbook of semidefinite programming: theory, algorithms, and applications, volume 27. Springer Science & Business Media, 2012.
  54. Yinyu Ye. Interior point algorithms: theory and analysis. John Wiley & Sons, 2011.
  55. Scalable semidefinite programming. SIAM Journal on Mathematics of Data Science, 3(1):171–200, 2021.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now