Byzantine-Resilient Decentralized Stochastic Optimization with Robust Aggregation Rules (2206.04568v2)

Abstract: This paper focuses on decentralized stochastic optimization in the presence of Byzantine attacks. During the optimization process, an unknown number of malfunctioning or malicious workers, termed as Byzantine workers, disobey the algorithmic protocol and send arbitrarily wrong messages to their neighbors. Even though various Byzantine-resilient algorithms have been developed for distributed stochastic optimization with a central server, we show that there are two major issues in the existing robust aggregation rules when being applied to the decentralized scenario: disagreement and non-doubly stochastic virtual mixing matrix. This paper provides comprehensive analysis that discloses the negative effects of these two issues, and gives guidelines of designing favorable Byzantine-resilient decentralized stochastic optimization algorithms. Under these guidelines, we propose iterative outlier scissor (IOS), an iterative filtering-based robust aggregation rule with provable performance guarantees. Numerical experiments demonstrate the effectiveness of IOS. The code of simulation implementation is available at github.com/Zhaoxian-Wu/IOS.