2000 character limit reached
Optimizing ADMM and Over-Relaxed ADMM Parameters for Linear Quadratic Problems (2401.00657v1)
Published 1 Jan 2024 in math.OC, cs.CV, and math.SP
Abstract: The Alternating Direction Method of Multipliers (ADMM) has gained significant attention across a broad spectrum of machine learning applications. Incorporating the over-relaxation technique shows potential for enhancing the convergence rate of ADMM. However, determining optimal algorithmic parameters, including both the associated penalty and relaxation parameters, often relies on empirical approaches tailored to specific problem domains and contextual scenarios. Incorrect parameter selection can significantly hinder ADMM's convergence rate. To address this challenge, in this paper we first propose a general approach to optimize the value of penalty parameter, followed by a novel closed-form formula to compute the optimal relaxation parameter in the context of linear quadratic problems (LQPs). We then experimentally validate our parameter selection methods through random instantiations and diverse imaging applications, encompassing diffeomorphic image registration, image deblurring, and MRI reconstruction.
- Accelerated first order methods for variational imaging. arXiv preprint arXiv:2110.02813.
- Bartlett, M. S. 1951. An inverse matrix adjustment arising in discriminant analysis. The Annals of Mathematical Statistics, 22(1): 107–111.
- Computing large deformation metric mappings via geodesic flows of diffeomorphisms. International Journal of Computer Vision, 61: 139–157.
- Boley, D. 2013. Local linear convergence of the alternating direction method of multipliers on quadratic or linear programs. SIAM Journal on Optimization, 23(4): 2183–2207.
- Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine learning, 3(1): 1–122.
- Robust principal component analysis? Journal of the ACM (JACM), 58(3): 1–37.
- Fractional Fourier transform of the Gaussian and fractional domain signal support. IEE Proceedings-Vision, Image and Signal Processing, 150(2): 99–106.
- Plug-and-play ADMM for image restoration: fixed-point convergence and applications. IEEE Transactions on Computational Imaging, 3(1): 84–98.
- A relaxed version of Bregman’s method for convex programming. Journal of Optimization Theory and Applications, 51: 421–440.
- On the global and linear convergence of the generalized alternating direction method of multipliers. Journal of Scientific Computing, 66: 889–916.
- Arbitrary order total variation for deformable image registration. Pattern Recognition, 109318.
- Eckstein, J. 1994. Parallel alternating direction multiplier decomposition of convex programs. Journal of Optimization Theory and Applications, 80(1): 39–62.
- An explicit rate bound for over-relaxed ADMM. In 2016 IEEE International Symposium on Information Theory (ISIT), 2104–2108. IEEE.
- Optimal parameter selection for the alternating direction method of multipliers (ADMM): quadratic problems. IEEE Transactions on Automatic Control, 60(3): 644–658.
- The split Bregman method for L1-regularized problems. SIAM Journal on Imaging Sciences, 2(2): 323–343.
- Truncated residual based plug-and-play ADMM algorithm for MRI reconstruction. IEEE Transactions on Computational Imaging, 8: 96–108.
- Learning a model-driven variational network for deformable image registration. IEEE Transactions on Medical Imaging, 41(1): 199–212.
- Robust decentralized learning using ADMM with unreliable agents. IEEE Transactions on Signal Processing, 70: 2743–2757.
- Distributed network reconstruction based on binary compressed sensing via ADMM. IEEE Transactions on Network Science and Engineering.
- Implementation of high-order variational models made easy for image processing. Mathematical Methods in the Applied Sciences, 39(14): 4208–4233.
- Auto-tuned weighted-penalty parameter ADMM for distributed optimal power flow. IEEE Transactions on Power Systems, 36(2): 970–978.
- Adaptive ADMM for distributed AC optimal power flow. IEEE Transactions on Power Systems, 34(3): 2025–2035.
- Nesterov, Y. E. 1983. A method of solving a convex programming problem with convergence rate O(1k2)𝑂1superscript𝑘2O(\frac{1}{k^{2}})italic_O ( divide start_ARG 1 end_ARG start_ARG italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ). In Doklady Akademii Nauk, volume 269, 543–547. Russian Academy of Sciences.
- Adaptive restart for accelerated gradient schemes. Foundations of computational mathematics, 15: 715–732.
- OSQP: An operator splitting solver for quadratic programs. Mathematical Programming Computation, 12(4): 637–672.
- The ADMM algorithm for distributed quadratic problems: parameter selection and constraint preconditioning. IEEE Transactions on Signal Processing, 64(2): 290–305.
- Nesterov accelerated ADMM for fast diffeomorphic image registration. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part IV 24, 150–160.
- Admm for efficient deep learning with global convergence. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 111–119.
- Wohlberg, B. 2017. ADMM penalty parameter selection by residual balancing. arXiv preprint arXiv:1704.06209.
- Adaptive ADMM with spectral penalty parameter selection. In Artificial Intelligence and Statistics, 718–727. PMLR.
- Interpolation and denoising of graph signals using plug-and-play ADMM. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 5431–5435.
- Privacy-preserving decentralized power system economic dispatch considering carbon capture power plants and carbon emission trading scheme via over-relaxed ADMM. International Journal of Electrical Power & Energy Systems, 121: 106094.
- Federated learning via inexact ADMM. IEEE Transactions on Pattern Analysis and Machine Intelligence.