Multi-Block Nonconvex Nonsmooth Proximal ADMM: Convergence and Rates under Kurdyka-Łojasiewicz Property (2009.04014v3)

Abstract: In this paper, we consider a multi-block generalized alternating direction method of multiplier (GADMM) algorithm for minimizing a linearly constrained separable nonconvex and possibly nonsmooth optimization problem. The GADMM generalizes the classical ADMM by including proximal terms in each primal updates and an over-relaxation parameter in the dual update. We prove that any limit point of the sequence is a critical point. By introducing a modified augmented Lagrangian we show that the sequence generated by the GADMM is bounded and the norm of the difference of consecutive terms approaches to zero. Under the powerful {K\L} properties we show that the GADMM sequence has a finite length and converges to a stationary point, and we drive its convergence rate. Given a proper lower-semicontinuous function $f:\mathbb R^n\to\mathbb R$ and a critical point $x^*\in\mathbb R^n$, the {K\L} property asserts that there exists a continuous concave monotonically increasing function $\psi$ such that around $x^*$ it holds $\psi'(f(x)-f(x^*))\cdot{\rm dist}(0,\partial f(x))\ge 1$ . When $\psi(s)=s^{1-\theta}$ with $\theta\in[0,1]$ this is equivalent to $|f(x)-f(x^*)|^{\theta}{\rm dist}(0,\partial f(x))^{-1}$ to remain bounded around $x^*$. We show that if $\theta=0$, the sequence generated by GADMM converges in a finite numbers of iterations. If $\theta\in(0,1/2]$, then the rate of convergence is $cQ^{k}$ where $c>0$, $Q\in(0,1)$, and $k\in\mathbb N$ is the iteration number. If $\theta\in(1/2,1]$ then the rate $\mathcal O(1/k^{r})$ where $r=(1-\theta)/(2\theta-1)$ will be achieved.