Decentralized Stochastic Nonconvex Optimization under the Relaxed Smoothness

Abstract: This paper studies decentralized optimization problem $$f(\mathbf{x})=\frac{1}{m}\sum_{i=1}^m f_i(\mathbf{x})$$, where each local function has the form of $$f_i(\mathbf{x}) = {\mathbb E}\left[F(\mathbf{x};{\xi}_i)\right]$$ which is $(L_0,L_1)$-smooth but possibly nonconvex and the random variable ${\xi}_i$ follows distribution ${\mathcal D}_i$. We propose a novel algorithm called decentralized normalized stochastic gradient descent (DNSGD), which can achieve the $\epsilon$-stationary point on each local agent. We present a new framework for analyzing decentralized first-order methods in the relaxed smooth setting, based on the Lyapunov function related to the product of the gradient norm and the consensus error. The analysis shows upper bounds on sample complexity of $${\mathcal O}(m^{{-1}(L_f\sigma^{2\Delta_f\epsilon^{-4}}} + \sigma^{2\epsilon^{-2}} + L_f^{{-2}L_1^{3\sigma^{2\Delta_f\epsilon^{-1}}}} + L_f<sup>{-2}L_1<sup>2\sigma<sup>2))$$ per agent and communication complexity of $$\tilde{\mathcal O}((L_f\epsilon^{-2} + L_1\epsilon<sup>{-1})\gamma<sup>{-1/2}\Delta_f)$$, where $L_f=L_0 +L_1\zeta$, $\sigma<sup>2$ is the variance of the stochastic gradient, $\Delta_f$ is the initial optimal function value gap, $\gamma$ is the spectral gap of the network, and $\zeta$ is the degree of the gradient dissimilarity. In the special case of $L_1=0$, the above results (nearly) match the lower bounds on decentralized nonconvex optimization in the standard smooth setting. We also conduct numerical experiments to show the empirical superiority of our method.