Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Revisiting Weighted Aggregation in Federated Learning with Neural Networks (2302.10911v4)

Published 14 Feb 2023 in cs.LG

Abstract: In federated learning (FL), weighted aggregation of local models is conducted to generate a global model, and the aggregation weights are normalized (the sum of weights is 1) and proportional to the local data sizes. In this paper, we revisit the weighted aggregation process and gain new insights into the training dynamics of FL. First, we find that the sum of weights can be smaller than 1, causing global weight shrinking effect (analogous to weight decay) and improving generalization. We explore how the optimal shrinking factor is affected by clients' data heterogeneity and local epochs. Second, we dive into the relative aggregation weights among clients to depict the clients' importance. We develop client coherence to study the learning dynamics and find a critical point that exists. Before entering the critical point, more coherent clients play more essential roles in generalization. Based on the above insights, we propose an effective method for Federated Learning with Learnable Aggregation Weights, named as FedLAW. Extensive experiments verify that our method can improve the generalization of the global model by a large margin on different datasets and models.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. Federated learning based on dynamic regularization. In International Conference on Learning Representations, 2020.
  2. A convergence theory for deep learning via over-parameterization. In International Conference on Machine Learning, pp. 242–252. PMLR, 2019.
  3. Improving generalization in federated learning by seeking flat minima. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIII, pp.  654–672. Springer, 2022.
  4. On large-cohort training for federated learning. Advances in neural information processing systems, 34:20461–20475, 2021.
  5. Chatterjee, S. Coherent gradients: An approach to understanding generalization in gradient descent-based optimization. In International Conference on Learning Representations, 2019.
  6. Making coherence out of nothing at all: measuring the evolution of gradient alignment. arXiv preprint arXiv:2008.01217, 2020.
  7. Fedbe: Making bayesian model ensemble applicable to federated learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id=dgtpE6gKjHn.
  8. Distributionally robust federated averaging. Advances in neural information processing systems, 33:15111–15122, 2020.
  9. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pp. 1019–1028. PMLR, 2017.
  10. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pp. 1309–1318. PMLR, 2018.
  11. Efficient sharpness-aware minimization for improved training of neural networks. arXiv preprint arXiv:2110.03141, 2021.
  12. The role of permutation invariance in linear mode connectivity of neural networks. In International Conference on Learning Representations, 2022.
  13. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020.
  14. Large scale structure of neural network loss landscapes. Advances in Neural Information Processing Systems, 32, 2019.
  15. Stiffness: A new perspective on generalization in neural networks. arXiv preprint arXiv:1901.09491, 2019.
  16. Forward and reverse gradient-based hyperparameter optimization. In International Conference on Machine Learning, pp. 1165–1173. PMLR, 2017.
  17. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018.
  18. Auto-fedrl: Federated hyperparameter optimization for multi-institutional medical image segmentation. arXiv preprint arXiv:2203.06338, 2022.
  19. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  4700–4708, 2017.
  20. Personalized cross-silo federated learning on non-iid data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.  7865–7873, 2021.
  21. Averaging weights leads to wider optima and better generalization. In 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, pp.  876–885, 2018.
  22. On the relation between the sharpest directions of dnn loss and the sgd step length. arXiv preprint arXiv:1807.05031, 2018.
  23. The break-even point on optimization trajectories of deep neural networks. In International Conference on Learning Representations, 2019.
  24. The break-even point on optimization trajectories of deep neural networks. arXiv preprint arXiv:2002.09572, 2020.
  25. Hyp-rl: Hyperparameter optimization by reinforcement learning. arXiv preprint arXiv:1906.11527, 2019.
  26. Kantorovich, L. V. On the translocation of masses. Journal of mathematical sciences, 133(4):1381–1382, 2006.
  27. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pp. 5132–5143. PMLR, 2020.
  28. On large-batch training for deep learning: Generalization gap and sharp minima. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
  29. Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning, pp. 5905–5914. PMLR, 2021.
  30. On the training dynamics of deep networks with l⁢_⁢2𝑙_2l\_2italic_l _ 2 regularization. Advances in Neural Information Processing Systems, 33:4790–4799, 2020.
  31. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
  32. Learning to collaborate in decentralized learning of personalized models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  9766–9775, 2022a.
  33. Federated learning: Challenges, methods, and future directions. IEEE Signal Process. Mag., 37(3):50–60, 2020a. doi: 10.1109/MSP.2020.2975749. URL https://doi.org/10.1109/MSP.2020.2975749.
  34. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429–450, 2020b.
  35. Understanding the disharmony between weight normalization family and weight decay. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.  4715–4722, 2020c.
  36. Towards effective clustered federated learning: A peer-to-peer framework with adaptive neighbor matching. IEEE Transactions on Big Data, 2022b.
  37. Mining latent relationships among clients: Peer-to-peer federated learning with adaptive neighbor matching. arXiv preprint arXiv:2203.12285, 2022c.
  38. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351–2363, 2020.
  39. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
  40. Understanding the generalization benefit of normalization layers: Sharpness reduction. arXiv preprint arXiv:2206.07085, 2022.
  41. Gradient-based hyperparameter optimization through reversible learning. In International conference on machine learning, pp. 2113–2122. PMLR, 2015.
  42. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.  1273–1282. PMLR, 2017.
  43. Mostafa, H. Robust federated learning through representation matching and adaptive hyper-parameters. arXiv preprint arXiv:1912.13075, 2019.
  44. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33:22045–22055, 2020.
  45. What can linear interpolation of neural network loss landscapes tell us? arXiv preprint arXiv:2106.16004, 2021.
  46. Spherical motion dynamics: Learning dynamics of normalized neural network using sgd and weight decay. Advances in Neural Information Processing Systems, 34:6380–6391, 2021.
  47. Federated learning with matched averaging. arXiv preprint arXiv:2002.06440, 2020a.
  48. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611–7623, 2020b.
  49. A field guide to federated optimization. arXiv preprint arXiv:2107.06917, 2021.
  50. Drflm: Distributionally robust federated learning with inter-client noise via local mixup. arXiv preprint arXiv:2204.07742, 2022.
  51. Auto-fedavg: learnable federated averaging for multi-institutional medical image segmentation. arXiv preprint arXiv:2104.10195, 2021.
  52. Understanding and scheduling weight decay. arXiv preprint arXiv:2011.11152, 2020.
  53. Critical learning periods in federated learning. arXiv preprint arXiv:2109.05613, 2021.
  54. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data), pp.  581–590. IEEE, 2020.
  55. Gradient diversity: a key ingredient for scalable distributed learning. In International Conference on Artificial Intelligence and Statistics, pp.  1998–2007. PMLR, 2018.
  56. Three mechanisms of weight decay regularization. In International Conference on Learning Representations, 2018.
  57. Weak and strong gradient directions: Explaining memorization, generalization, and hardness of examples at scale. arXiv preprint arXiv:2003.07422, 2020.
  58. An improved analysis of training over-parameterized deep neural networks. Advances in neural information processing systems, 32, 2019.
Citations (41)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now