FedHybrid: A Hybrid Primal-Dual Algorithm Framework for Federated Optimization

Abstract: We consider a multi-agent consensus optimization problem over a server-client (federated) network, where all clients are connected to a central server. Current distributed algorithms fail to capture the heterogeneity in clients' local computation capacities. Motivated by the generalized Method of Multipliers in centralized optimization, we derive an approximate Newton-type primal-dual method with a practical distributed implementation by utilizing the server-client topology. Then we propose a new primal-dual algorithm framework FedHybrid that allows different clients to perform various types of updates. Specifically, each client can choose to perform either gradient-type or Newton-type updates. We propose a novel analysis framework for primal-dual methods and obtain a linear convergence rate of FedHybrid for strongly convex functions, regardless of clients' choices of gradient-type or Newton-type updates. Numerical studies are provided to demonstrate the efficacy of our method in practice. To the best of our knowledge, this is the first hybrid algorithmic framework allowing heterogeneous local updates for distributed consensus optimization with a provable convergence and rate guarantee.