Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
117 tokens/sec
GPT-4o
8 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Technical Report: On the Convergence of Gossip Learning in the Presence of Node Inaccessibility (2401.09498v2)

Published 17 Jan 2024 in cs.LG and cs.AI

Abstract: Gossip learning (GL), as a decentralized alternative to federated learning (FL), is more suitable for resource-constrained wireless networks, such as Flying Ad-Hoc Networks (FANETs) that are formed by unmanned aerial vehicles (UAVs). GL can significantly enhance the efficiency and extend the battery life of UAV networks. Despite the advantages, the performance of GL is strongly affected by data distribution, communication speed, and network connectivity. However, how these factors influence the GL convergence is still unclear. Existing work studied the convergence of GL based on a virtual quantity for the sake of convenience, which failed to reflect the real state of the network when some nodes are inaccessible. In this paper, we formulate and investigate the impact of inaccessible nodes to GL under a dynamic network topology. We first decompose the weight divergence by whether the node is accessible or not. Then, we investigate the GL convergence under the dynamic of node accessibility and theoretically provide how the number of inaccessible nodes, data non-i.i.d.-ness, and duration of inaccessibility affect the convergence. Extensive experiments are carried out in practical settings to comprehensively verify the correctness of our theoretical findings.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (26)
  1. C. Qu, J. Boubin, D. Gafurov, J. Zhou, N. Aloysius, H. Nguyen, and P. Calyam, “Uav swarms in smart agriculture: Experiences and opportunities,” in 2022 IEEE 18th International Conference on e-Science (e-Science).   IEEE, 2022, pp. 148–158.
  2. D. Albani, T. Manoni, A. Arik, D. Nardi, and V. Trianni, “Field coverage for weed mapping: toward experiments with a uav swarm,” in Bio-inspired Information and Communication Technologies: 11th EAI International Conference, BICT 2019, Pittsburgh, PA, USA, March 13–14, 2019, Proceedings 11.   Springer, 2019, pp. 132–146.
  3. S. Yu, J. Zhu, J. Zhou, J. Cheng, X. Bian, J. Shen, and P. Wang, “Key technology progress of plant-protection uavs applied to mountain orchards: A review,” Agronomy, vol. 12, no. 11, p. 2828, 2022.
  4. A. Davis, P. S. Wills, J. E. Garvey, W. Fairman, M. A. Karim, and B. Ouyang, “Developing and field testing path planning for robotic aquaculture water quality monitoring,” Applied Sciences, vol. 13, no. 5, p. 2805, 2023.
  5. M. Mohan, G. Richardson, G. Gopan, M. M. Aghai, S. Bajaj, G. P. Galgamuwa, M. Vastaranta, P. S. P. Arachchige, L. Amorós, A. P. D. Corte et al., “Uav-supported forest regeneration: Current trends, challenges and implications,” Remote Sensing, vol. 13, no. 13, p. 2596, 2021.
  6. I. Hegedűs, G. Danner, and M. Jelasity, “Gossip learning as a decentralized alternative to federated learning,” in Distributed Applications and Interoperable Systems: 19th IFIP WG 6.1 International Conference, DAIS 2019, Held as Part of the 14th International Federated Conference on Distributed Computing Techniques, DisCoTec 2019, Kongens Lyngby, Denmark, June 17–21, 2019, Proceedings 19.   Springer, 2019, pp. 74–90.
  7. I. Hegedus, G. Danner, and M. Jelasity, “Decentralized learning works: An empirical comparison of gossip learning and federated learning,” Journal of Parallel and Distributed Computing, vol. 148, pp. 109–124, 2021.
  8. X. Lian, C. Zhang, H. Zhang, C.-J. Hsieh, W. Zhang, and J. Liu, “Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent,” Advances in neural information processing systems, vol. 30, 2017.
  9. L. Giaretta and Š. Girdzijauskas, “Gossip learning: Off the beaten path,” in 2019 IEEE International Conference on Big Data (Big Data).   IEEE, 2019, pp. 1117–1124.
  10. A. Koloskova, S. Stich, and M. Jaggi, “Decentralized stochastic optimization and gossip algorithms with compressed communication,” in International Conference on Machine Learning.   PMLR, 2019, pp. 3478–3487.
  11. Z. Tang, S. Shi, B. Li, and X. Chu, “Gossipfl: A decentralized federated learning framework with sparsified and adaptive communication,” IEEE Transactions on Parallel and Distributed Systems, vol. 34, no. 3, pp. 909–922, 2022.
  12. A. Hashemi, A. Acharya, R. Das, H. Vikalo, S. Sanghavi, and I. Dhillon, “On the benefits of multiple gossip steps in communication-constrained decentralized federated learning,” IEEE Transactions on Parallel and Distributed Systems, vol. 33, no. 11, pp. 2727–2739, 2021.
  13. S. Boyd, A. Ghosh, B. Prabhakar, and D. Shah, “Randomized gossip algorithms,” IEEE transactions on information theory, vol. 52, no. 6, pp. 2508–2530, 2006.
  14. R. Ormándi, I. Hegedűs, and M. Jelasity, “Gossip learning with linear models on fully distributed data,” Concurrency and Computation: Practice and Experience, vol. 25, no. 4, pp. 556–571, 2013.
  15. M. Blot, D. Picard, M. Cord, and N. Thome, “Gossip training for deep learning,” arXiv preprint arXiv:1611.09726, 2016.
  16. A. Mathkar and V. S. Borkar, “Distributed reinforcement learning via gossip,” IEEE Transactions on Automatic Control, vol. 62, no. 3, pp. 1465–1470, 2016.
  17. Y. Belal, A. Bellet, S. B. Mokhtar, and V. Nitu, “Pepper: Empowering user-centric recommender systems over gossip learning,” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 6, no. 3, pp. 1–27, 2022.
  18. J. Daily, A. Vishnu, C. Siegel, T. Warfel, and V. Amatya, “Gossipgrad: Scalable deep learning using gossip communication based asynchronous gradient descent,” arXiv preprint arXiv:1803.05880, 2018.
  19. A. Hashemi, A. Acharya, R. Das, H. Vikalo, S. Sanghavi, and I. Dhillon, “On the benefits of multiple gossip steps in communication-constrained decentralized federated learning,” IEEE Transactions on Parallel and Distributed Systems, vol. 33, no. 11, pp. 2727–2739, 2022.
  20. K. Yuan, Q. Ling, and W. Yin, “On the convergence of decentralized gradient descent,” SIAM Journal on Optimization, vol. 26, no. 3, pp. 1835–1854, 2016.
  21. J. Chen, S. Wang, L. Carin, and C. Tao, “Finite-time consensus learning for decentralized optimization with nonlinear gossiping,” arXiv preprint arXiv:2111.02949, 2021.
  22. X. Gu, K. Huang, J. Zhang, and L. Huang, “Fast federated learning in the presence of arbitrary device unavailability,” Advances in Neural Information Processing Systems, vol. 34, pp. 12 052–12 064, 2021.
  23. T. Zhu, F. He, L. Zhang, Z. Niu, M. Song, and D. Tao, “Topology-aware generalization of decentralized sgd,” in International Conference on Machine Learning.   PMLR, 2022, pp. 27 479–27 503.
  24. K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in deep residual networks,” in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14.   Springer, 2016, pp. 630–645.
  25. A. Krizhevsky, G. Hinton et al., “Learning multiple layers of features from tiny images,” 2009.
  26. S. Mao, “Chapter 8 - fundamentals of communication networks,” in Cognitive Radio Communications and Networks, A. M. Wyglinski, M. Nekovee, and Y. T. Hou, Eds.   Oxford: Academic Press, 2010, pp. 201–234. [Online]. Available: https://www.sciencedirect.com/science/article/pii/B9780123747150000083

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets