Robust Aggregation for Federated Learning (1912.13445v2)

Abstract: Federated learning is the centralized training of statistical models from decentralized data on mobile devices while preserving the privacy of each device. We present a robust aggregation approach to make federated learning robust to settings when a fraction of the devices may be sending corrupted updates to the server. The approach relies on a robust aggregation oracle based on the geometric median, which returns a robust aggregate using a constant number of iterations of a regular non-robust averaging oracle. The robust aggregation oracle is privacy-preserving, similar to the non-robust secure average oracle it builds upon. We establish its convergence for least squares estimation of additive models. We provide experimental results with linear models and deep networks for three tasks in computer vision and natural language processing. The robust aggregation approach is agnostic to the level of corruption; it outperforms the classical aggregation approach in terms of robustness when the level of corruption is high, while being competitive in the regime of low corruption. Two variants, a faster one with one-step robust aggregation and another one with on-device personalization, round off the paper.

• The paper proposes a robust aggregation method using the geometric median, which can tolerate up to half of the device updates being corrupted.
• It develops a smoothed Weiszfeld algorithm that converges in just three iterations while ensuring data privacy through secure multiparty computation.
• Theoretical analysis and experiments confirm its effectiveness in high corruption scenarios, enhancing model reliability in federated learning.

Robust Aggregation for Federated Learning

The paper "Robust Aggregation for Federated Learning" addresses the challenge of maintaining model robustness in federated learning environments where some devices may send corrupted updates. Federated learning, a paradigm designed for privacy-preserving model training on decentralized data, requires robust aggregation techniques to withstand adversarial influences and hardware malfunctions. The authors propose using the geometric median as an aggregation method, offering improved robustness while preserving privacy.

Core Contributions and Methods

The paper makes several key contributions:

1. Robust Aggregation using Geometric Median: The geometric median is chosen due to its optimal breakdown point of 1/2, making it highly resistant to corruptions. This property ensures that the aggregate can withstand up to half of the data being corrupted.
2. Algorithmic Implementation with Privacy Preservation: The authors develop a smoothed Weiszfeld algorithm to compute the geometric median as an iterative secure aggregate. This allows them to maintain privacy through secure multi-party computation, and the algorithm empirically shows rapid convergence, necessitating only three iterations to achieve a robust aggregate.
3. Convergence Analysis: For least-squares estimation in additive models, the paper provides theoretical convergence guarantees. The algorithm proves robust to up to half of the corrupted devices in federated learning scenarios, even with bounded heterogeneity.
4. Experimental Results: Comprehensive experiments illustrate the robustness of the approach in tasks like character recognition and NLP. The proposed method excels in high corruption scenarios while maintaining performance in low corruption settings.
5. Extensions: The paper introduces a one-step robust aggregation variant to reduce communication costs and an on-device personalization technique to better handle data heterogeneity.

Implications

The authors' robust aggregation method has significant implications for real-world federated learning applications. By embedding this robust aggregation technique, systems can remain resilient to data poisoning attacks and faults from unreliable devices, maintaining model integrity. The privacy-preserving nature of the method ensures that it aligns well with the foundational principles of federated learning, where data privacy is paramount.

Future Directions

This work opens avenues for further exploration. Future research could focus on extending the convergence analysis to more general convex objectives or investigating other robust aggregation techniques that might offer improved computational efficiency. Exploring the integration with other privacy-preserving techniques could also enhance the model's robustness and applicability.

Overall, this paper contributes a substantial improvement in federated learning robustness, both theoretically and practically, providing a pathway for more reliable deployment in sensitive distributed systems.