An over-relaxed ADMM for separable convex programming and its applications to statistical learning (2401.03651v2)
Abstract: The alternating direction method of multipliers (ADMM) has been applied successfully in a broad spectrum of areas. Moreover, it was shown in the literature that ADMM is closely related to the Douglas-Rachford operator-splitting method, and can be viewed as a special case of the proximal point algorithm (PPA). As is well known, the relaxed version of PPA not only maintains the convergence properties of PPA in theory, but also numerically outperforms the original PPA. Therefore, it is interesting to ask whether ADMM can be accelerated by using the over-relaxation technique. In this paper, we answer this question affirmatively by developing an over-relaxed ADMM integrated with a criterion to decide whether a relaxation step is implemented in the iteration. The global convergence and $O(1/t)$ convergence rate of the over-relaxed ADMM are established. We also implement our proposed algorithm to solve Lasso and sparse inverse covariance selection problems, and compare its performance with the relaxed customized ADMM in \cite{CGHY} and the classical ADMM. The results show that our algorithm substantially outperforms the other two methods.
- Arthur P. Dempster. Covariance selection. Biometrics, 157-175 (1972)
- M. Tao and J. Yang, “Alternating direction algorithms for total variation deconvolution in image reconstruction,” TR0918, 2009.
- P. L. Combettes and J. C. Pesquet, “A douglas-Rachford splitting approach to nonsmooth convex variational signal recovery,” IEEE Journal of Selected Topics in Signal Processing, vol. 1, no. 4, pp. 564–574, 2007.
- Combettes, Patrick L. and Pesquet, Jean Christophe, “Proximal splitting methods in signal processing,” Heinz H Bauschke, vol. 49, pp. págs. 185–212, 2009.
- M. K. Ng, P. Weiss, and X. Yuan, “Solving constrained total-variation image restoration and reconstruction problems via alternating direction methods,” vol. 32, no. 5, pp. 2710–2736, 2010.
- G.Y.Gu and J.F.Yang. On the optimal linear convergence factor of the relaxed proximal point algorithm for monotone inclusion problems. arXiv preprint arXiv:1905.04537, 2019.
- M.H.Xu. Proximal alternating directions method for structured variational inequalities. Journal of Optimization Theory and Applications, 134(1):107–117, 2007.
- F.Ma. Convergence study on the proximal alternating direction method with larger step size. Numerical Algorithms, 85(2):399–425, 2020.
- M. Tao and X. Yuan. On the optimal linear convergence rate of a generalized proximal point algorithm. Journal of Scientific Computing, 74(2):826–850, 2018.