A Communication Efficient Quasi-Newton Method for Large-scale Distributed Multi-agent Optimization

Abstract: We propose a communication efficient quasi-Newton method for large-scale multi-agent convex composite optimization. We assume the setting of a network of agents that cooperatively solve a global minimization problem with strongly convex local cost functions augmented with a non-smooth convex regularizer. By introducing consensus variables, we obtain a block-diagonal Hessian and thus eliminate the need for additional communication when approximating the objective curvature information. Moreover, we reduce computational costs of existing primal-dual quasi-Newton methods from $\mathcal{O}(d^3)$ to $\mathcal{O}(cd)$ by storing $c$ pairs of vectors of dimension $d$. An asynchronous implementation is presented that removes the need for coordination. Global linear convergence rate in expectation is established, and we demonstrate the merit of our algorithm numerically with real datasets.