Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Tackling Computational Heterogeneity in FL: A Few Theoretical Insights (2307.06283v1)

Published 12 Jul 2023 in cs.LG and cs.DC

Abstract: The future of machine learning lies in moving data collection along with training to the edge. Federated Learning, for short FL, has been recently proposed to achieve this goal. The principle of this approach is to aggregate models learned over a large number of distributed clients, i.e., resource-constrained mobile devices that collect data from their environment, to obtain a new more general model. The latter is subsequently redistributed to clients for further training. A key feature that distinguishes federated learning from data-center-based distributed training is the inherent heterogeneity. In this work, we introduce and analyse a novel aggregation framework that allows for formalizing and tackling computational heterogeneity in federated optimization, in terms of both heterogeneous data and local updates. Proposed aggregation algorithms are extensively analyzed from a theoretical, and an experimental prospective.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pages 2938–2948. PMLR, 2020.
  2. Limited-memory bfgs with displacement aggregation. Mathematical Programming, 194(1-2):121–157, 2022.
  3. Personalized federated learning with communication compression. arXiv preprint arXiv:2209.05148, 2022.
  4. Federated learning with autotuned communication-efficient secure aggregation. In 2019 53rd Asilomar Conference on Signals, Systems, and Computers, pages 1222–1226. IEEE, 2019.
  5. Federated select: A primitive for communication-and memory-efficient federated learning. arXiv preprint arXiv:2208.09432, 2022.
  6. Federated meta-learning with fast convergence and efficient communication. arXiv preprint arXiv:1802.07876, 2018.
  7. Optimal client sampling for federated learning. Transactions on Machine Learning Research, 2017.
  8. Communication-efficient federated deep learning with layerwise asynchronous model update and temporally weighted aggregation. IEEE transactions on neural networks and learning systems, 31(10):4229–4238, 2019.
  9. Client selection in federated learning: Convergence analysis and power-of-choice selection strategies. arXiv preprint arXiv:2010.01243, 2020.
  10. Personalized federated learning for heterogeneous clients with clustered knowledge transfer. arXiv preprint arXiv:2109.08119, 2021.
  11. Heterogeneous ensemble knowledge transfer for training large models in federated learning. arXiv preprint arXiv:2204.12703, 2022a.
  12. Towards understanding biased client selection in federated learning. In International Conference on Artificial Intelligence and Statistics, pages 10351–10375. PMLR, 2022b.
  13. Variational federated multi-task learning. arXiv preprint arXiv:1906.06268, 2019.
  14. High-performance distributed ml at scale through parameter server consistency models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
  15. Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29(6):141–142, 2012.
  16. Estimation, optimization, and parallelism when data is sparse. Advances in Neural Information Processing Systems, 26, 2013.
  17. Semi-cyclic stochastic gradient descent. In International Conference on Machine Learning, pages 1764–1773. PMLR, 2019.
  18. Sgd: General analysis and improved rates. In International conference on machine learning, pages 5200–5209. PMLR, 2019.
  19. Fedshuffle: Recipes for better use of local work in federated learning. arXiv preprint arXiv:2204.13169, 2022.
  20. Stochastic client selection for federated learning with volatile clients. IEEE Internet of Things Journal, 9(20):20055–20070, 2022.
  21. Fedvarp: Tackling the variance due to partial client participation in federated learning. In Uncertainty in Artificial Intelligence, pages 906–916. PMLR, 2022.
  22. A linear speedup analysis of distributed deep learning with sparse and quantized communication. Advances in Neural Information Processing Systems, 31, 2018.
  23. Accelerated federated learning with decoupled adaptive optimization. In International Conference on Machine Learning, pages 10298–10322. PMLR, 2022.
  24. Mechanisms that incentivize data sharing in federated learning. arXiv preprint arXiv:2207.04557, 2022.
  25. Adaptive gradient-based meta-learning methods. Advances in Neural Information Processing Systems, 32, 2019.
  26. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492, 2016.
  27. Fair resource allocation in federated learning. arXiv preprint arXiv:1905.10497, 2019.
  28. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 37(3):50–60, 2020a.
  29. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 37(3):50–60, 2020b.
  30. Federated learning in mobile edge networks: A comprehensive survey. IEEE Communications Surveys & Tutorials, 22(3):2031–2063, 2020.
  31. Accelerating federated learning via momentum gradient descent. IEEE Transactions on Parallel and Distributed Systems, 31(8):1754–1766, 2020.
  32. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017a.
  33. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017b.
  34. Local learning matters: Rethinking data heterogeneity in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8397–8406, 2022.
  35. Agnostic federated learning. In International Conference on Machine Learning, pages 4615–4625. PMLR, 2019.
  36. Client selection for federated learning with heterogeneous resources in mobile edge. In ICC 2019-2019 IEEE international conference on communications (ICC), pages 1–7. IEEE, 2019.
  37. Adaptive federated optimization. arXiv preprint arXiv:2003.00295, 2020.
  38. A federated learning aggregation algorithm for pervasive computing: Evaluation and comparison. In 2021 IEEE International Conference on Pervasive Computing and Communications (PerCom), pages 1–10. IEEE, 2021.
  39. Federated learning on heterogeneous and long-tailed data via classifier re-training with federated features. IJCAI-ECAI 2022, 2022.
  40. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems, 2022.
  41. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611–7623, 2020.
  42. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
  43. Characterizing impacts of heterogeneity in federated learning upon large-scale smartphone data, 2021.
  44. Bayesian nonparametric federated learning of neural networks. In International conference on machine learning, pages 7252–7261. PMLR, 2019.
  45. Fedpage: A fast local stochastic gradient method for communication-efficient federated learning. arXiv preprint arXiv:2108.04755, 2021.
  46. Efficient privacy-preserving data merging and skyline computation over multi-source encrypted data. Information Sciences, 498:91–105, 2019.
  47. Federated learning with online adaptive heterogeneous local models. In Workshop on Federated Learning: Recent Advances and New Challenges (in Conjunction with NeurIPS 2022).
  48. On the convergence of heterogeneous federated learning with arbitrary adaptive online model pruning. arXiv preprint arXiv:2201.11803, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Adnan Ben Mansour (3 papers)
  2. Gaia Carenini (10 papers)
  3. Alexandre Duplessis (6 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now