2000 character limit reached
Provably Personalized and Robust Federated Learning
Published 14 Jun 2023 in cs.LG and cs.DC | (2306.08393v2)
Abstract: Identifying clients with similar objectives and learning a model-per-cluster is an intuitive and interpretable approach to personalization in federated learning. However, doing so with provable and optimal guarantees has remained an open challenge. We formalize this problem as a stochastic optimization problem, achieving optimal convergence rates for a large class of loss functions. We propose simple iterative algorithms which identify clusters of similar clients and train a personalized model-per-cluster, using local client gradients and flexible constraints on the clusters. The convergence rates of our algorithms asymptotically match those obtained if we knew the true underlying clustering of the clients and are provably robust in the Byzantine setting where some fraction of the clients are malicious.
- “Communication-efficient learning of deep networks from decentralized data” In Artificial intelligence and statistics, 2017, pp. 1273–1282 PMLR
- “Taskonomy: Disentangling task transfer learning” In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3712–3722
- “Towards personalized federated learning” In IEEE Transactions on Neural Networks and Learning Systems IEEE, 2022
- Viraj Kulkarni, Milind Kulkarni and Aniruddha Pant “Survey of personalization techniques for federated learning” In 2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4), 2020, pp. 794–797 IEEE
- Felix Sattler, Klaus-Robert Muller and Wojciech Samek “Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints” In IEEE Transactions on Neural Networks and Learning Systems 32(8), 2021
- “Three Approaches for Personalization with Applications to Federated Learning”, 2021 arXiv:2002.10619 [cs.LG]
- “An efficient framework for clustered federated learning” In Advances in Neural Information Processing Systems 33, 2020, pp. 19586–19597
- Mathieu Even, Laurent Massoulié and Kevin Scaman “On sample optimality in personalized collaborative and federated learning” In Advances in Neural Information Processing Systems 35, 2022, pp. 212–225
- “Federated Multi-Task Learning” In 31st Conference on Neural Information Processing Systems, 2017
- “Fair and Robust Federated Learning through Personalization” In 38th International Conference on Machine Learning, 2021
- L. Lamport, R. Shostak and M. Pease “The Byzantine Generals Problem” In In Concurrency: the Works of Leslie Lamport., 2019
- “Machine learning with adversaries: Byzantine tolerant gradient descent” In Advances in Neural Information Processing Systems 30, 2017
- “Byzantine-robust distributed learning: Towards optimal statistical rates” In International Conference on Machine Learning (PMLR), 2018
- “Aggregator: Byzantine machine learning via robust gradient aggregation” In Proceedings of Machine Learning and Systems 1, 2018, pp. 81–106
- “The hidden vulnerability of distributed learning in byzantium” In International Conference on Machine Learning (PMLR), 2018, pp. 35213530
- K. Pillutla, S. Kakade and Z. Harchaoui “Robust aggregation for federated learning” In IEEE Transactions on Signal Processing 70, 2022, pp. 1142–1154
- B. Baruch, M. Baruch and Y. Goldberg “A little is enough: Circumventing defenses for distributed learning” In Advances in Neural Information Processing Systems 32, 2019
- C. Xie, O. Koyejo and I. Gupta “Fall of empires: Breaking byzantine-tolerant sgd by inner product manipulation” In Uncertainty in Artificial Intelligence (PMLR), 2020, pp. 261–270
- Sai Praneeth Karimireddy, Lie He and Martin Jaggi “Learning from History for Byzantine Robust Optimization” In Proceedings of the 38th International Conference on Machine Learning 129, 2021, pp. 5311–5319
- Sai Praneeth Karimireddy, Lie He and Martin Jaggi “Byzantine-Robust Learning on Heterogeneous Datasets via Bucketing” In International Conference on Learning Representations, 2022
- “Personalized Federated Learning via Heterogeneous Modular Networks” In IEEE International Conference on Data Mining, 2022, pp. 1197–1202
- “Personalized Federated Learning under Mixture of Distributions” In Proceedings of the 40th International Conference on Machine Learning (PMLR), 2023
- “Federated multi-task learning under a mixture of distributions” In Proceedings of the 35th International Conference on Machine Learning 34, 2021
- Sharan Vaswani, Francis Bach and Mark Schmidt “Fast and Faster Convergence of SGD for Over-Parameterized models and an Accelerated Perceptron” In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS) 89, 2019
- “Lower Bounds for Non-Convex Stochastic Optimization” In Mathematical Programming 199, 2023, pp. 165–214
- Yann LeCun, Corinna Cortes and CJ Burges “MNIST handwritten digit database” In ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist 2, 2010
- Alex Krizhevsky “Learning Multiple Layers of Features from Tiny Images”, 2009 URL: https://api.semanticscholar.org/CorpusID:18268744
- “Automatic differentiation in PyTorch” In NIPS-W, 2017
- “Very deep convolutional networks for large-scale image recognition” In International Conference on Learning Representations, 2015
- “Scikit-learn: Machine Learning in Python” In Journal of Machine Learning Research 12, 2011, pp. 2825–2830
- “Optimality and stability in federated learning: A game-theoretic approach” In Advances in Neural Information Processing Systems 34, 2021, pp. 1287–1298
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
