Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Green, Quantized Federated Learning over Wireless Networks: An Energy-Efficient Design (2207.09387v3)

Published 19 Jul 2022 in cs.LG

Abstract: In this paper, a green-quantized FL framework, which represents data with a finite precision level in both local training and uplink transmission, is proposed. Here, the finite precision level is captured through the use of quantized neural networks (QNNs) that quantize weights and activations in fixed-precision format. In the considered FL model, each device trains its QNN and transmits a quantized training result to the base station. Energy models for the local training and the transmission with quantization are rigorously derived. To minimize the energy consumption and the number of communication rounds simultaneously, a multi-objective optimization problem is formulated with respect to the number of local iterations, the number of selected devices, and the precision levels for both local training and transmission while ensuring convergence under a target accuracy constraint. To solve this problem, the convergence rate of the proposed FL system is analytically derived with respect to the system control variables. Then, the Pareto boundary of the problem is characterized to provide efficient solutions using the normal boundary inspection method. Design insights on balancing the tradeoff between the two objectives while achieving a target accuracy are drawn from using the Nash bargaining solution and analyzing the derived convergence rate. Simulation results show that the proposed FL framework can reduce energy consumption until convergence by up to 70\% compared to a baseline FL algorithm that represents data with full precision without damaging the convergence rate.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (34)
  1. M. Kim, W. Saad, M. Mozaffari, and M. Debbah, “On the tradeoff between energy, precision, and accuracy in federated quantized neural networks,” in Proc. of IEEE Int. Conf. Commun., Seoul, South Korea, May 2022.
  2. M. Chen, Z. Yang, W. Saad, C. Yin, H. V. Poor, and S. Cui, “A joint learning and communications framework for federated learning over wireless networks,” IEEE Trans. Wireless Commun., vol. 20, no. 1, pp. 269–283, Jan. 2021.
  3. R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni, “Green AI,” arXiv preprint arXiv:1907.10597, 2019.
  4. H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. Arcas, “Communication-efficient learning of deep networks from decentralized data,” arXiv preprint arXiv:1602.05629, 2017.
  5. B. Moons, K. Goetschalckx, N. Van Berckelaer, and M. Verhelst, “Minimum energy quantized neural networks,” in Proc. of Asilomar Conf. on Signals, Systems, and Computers, Pacific Grove, CA, USA, Apr. 2017.
  6. S. Savazzi, V. Rampa, S. Kianoush, and M. Bennis, “An energy and carbon footprint analysis of distributed and federated learning,” arXiv preprint arXiv:2206.10380, 2022.
  7. N. H. Tran, W. Bao, A. Zomaya, M. N. H. Nguyen, and C. S. Hong, “Federated learning over wireless networks: Optimization model design and analysis,” in Proc. of IEEE Conf. on Computer Commun., Paris, France, May 2019.
  8. Z. Yang, M. Chen, W. Saad, C. S. Hong, and M. Shikh-Bahaei, “Energy efficient federated learning over wireless communication networks,” IEEE Trans. Wireless Commun., vol. 20, no. 3, pp. 1935–1949, Mar. 2021.
  9. Q. Zeng, Y. Du, K. Huang, and K. K. Leung, “Energy-efficient resource management for federated edge learning with cpu-gpu heterogeneous computing,” IEEE Trans. Wireless Commun., vol. 20, no. 12, pp. 7947–7962, Dec. 2021.
  10. B. Luo, X. Li, S. Wang, J. Huangy, and L. Tassiulas, “Cost-effective federated learning design,” in Proc. of IEEE Conf. on Computer Commun., Vancouver, BC, Canada, May 2021.
  11. R. Balakrishnan, M. Akdeniz, S. Dhakal, and N. Himayat, “Resource management and fairness for federated learning over wireless edge networks,” in Proc. of IEEE Workshop on Signal Process. Advances in Wireless Commun., Atlanta, GA, USA, May 2020.
  12. G. Zhu, Y. Du, D. Gündüz, and K. Huang, “One-bit over-the-air aggregation for communication-efficient federated edge learning: Design and convergence analysis,” IEEE Trans. Wireless Commun., vol. 20, no. 3, pp. 2120–2135, Mar. 2021.
  13. P. Liu, J. Jiang, G. Zhu, L. Cheng, W. Jiang, W. Luo, Y. Du, and Z. Wang, “Training time minimization for federated edge learning with optimized gradient quantization and bandwidth allocation,” Frontiers of Information Technology & Electronic Engineering, vol. 23, no. 8, pp. 1247–1263, 2022 .
  14. C. Feng, Z. Zhao, Y. Wang, T. Q. Quek, and M. Peng, “On the design of federated learning in the mobile edge computing systems,” IEEE Trans. Commun., vol. 69, no. 9, pp. 5902–5916, Sep. 2021.
  15. R. Chen, L. Li, K. Xue, C. Zhang, M. Pan, and Y. Fang, “Energy efficient federated learning over heterogeneous mobile devices via joint design of weight quantization and wireless transmission,” IEEE Trans. Mobile Comput., pp. 1–13, Oct. 2022.
  16. I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio, “Quantized neural networks: Training neural networks with low precision weights and activations.” arXiv preprint arXiv:1609.07061, 2016.
  17. S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan, “Deep learning with limited numerical precision,” in Proc. of International Conference on Machine Learning (ICML), Lille, France, Jul. 2015.
  18. S. Zhou, Y. Wu, Z. Ni, X. Zhou, H. Wen, and Y. Zou, “DoReFa-Net: Training low bitwidth convolutional neural networks with low bitwidth gradients,” arXiv preprint arXiv:1606.06160, 2018.
  19. S. Zheng, C. Shen, and X. Chen, “Design and analysis of uplink and downlink communications for federated learning,” IEEE J. Sel. Areas Commun., vol. 39, no. 7, Jul. 2021.
  20. U. Evci, T. Gale, J. Menick, P. S. Castro, and E. Elsen, “Rigging the lottery: Making all tickets winners,” in Proc. of International Conference on Machine Learning (ICML), Vienna, Austria, Apr. 2020, pp. 2943–2952.
  21. E. Bjornson, E. A. Jorswieck, M. Debbah, and B. Ottersten, “Multiobjective signal processing optimization: The way to balance conflicting metrics in 5g systems,” IEEE Signal Processing Magazine, vol. 31, no. 6, pp. 14–23, Nov. 2014.
  22. X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang, “On the convergence of fedavg on non-iid data,” in Proc. of International Conference on Learning Representations (ICLR), May 2020.
  23. Z. Yuchen, D. J. C., and W. M. J., “Communication-efficient algorithms for statistical optimization,” J. Mach. Learn. Res., vol. 14, no. 1, p. 3321–3363, Jan. 2013.
  24. I. Das and J. E. Dennis, “Normal-boundary intersection: A new method for generating the pareto surface in nonlinear multicriteria optimization problems,” SIAM journal on optimization, vol. 8, no. 3, pp. 631–657, Aug. 1998.
  25. R. Wituła and D. Słota, “Cardano’s formula, square roots, chebyshev polynomials and radicals,” Journal of Mathematical Analysis and Applications, vol. 363, no. 2, pp. 639–647, Feb. 2010.
  26. D. P. Bertsekas, “Nonlinear programming,” Journal of the Operational Research Society, vol. 48, no. 3, pp. 334–334, 1997.
  27. E. Larsson and E. Jorswieck, “Competition versus cooperation on the miso interference channel,” IEEE J. Sel. Areas Commun., vol. 26, no. 7, pp. 1059–1069, Sep. 2008.
  28. P. Hyunggon and M. van der Schaar, “Bargaining strategies for networked multimedia resource management,” IEEE Trans. Signal Process., vol. 55, no. 7, pp. 3496–3511, Jul. 2007.
  29. R. Yedida, S. Saha, and T. Prashanth, “Lipschitzlr: Using theoretically computed adaptive learning rates for fast convergence,” Applied Intelligence, vol. 51, Mar. 2021.
  30. T. Hofmann, A. Lucchi, S. Lacoste-Julien, and B. McWilliams, “Variance reduced stochastic gradient descent with neighbors,” in Proc. of Neural Information Processing Systems (NeurIPS), Montreal, Canada, Dec. 2015.
  31. Q. Jin and A. Mokhtari, “Exploiting local convergence of quasi-newton methods globally: Adaptive sample size approach,” in Proc. of Neural Information Processing Systems (NeurIPS), Virtual, Dec. 2021.
  32. A. Øland and B. Raj, “Reducing communication overhead in distributed learning by an order of magnitude (almost),” in Proc. of IEEE Int. Conf. Acoustics, Speech, and Signal Processing, South Brisbane, QLD, Australia, 2015, pp. 2219–2223.
  33. Y. Sarikaya and O. Ercetin, “Motivating workers in federated learning: A stackelberg game perspective,” IEEE Net. Lett., vol. 2, no. 1, pp. 23–27, Oct. 2020.
  34. A. Reisizadeh, A. Mokhtari, H. Hassani, A. Jadbabaie, and R. Pedarsani, “FedPAQ: A communication-efficient federated learning method with periodic averaging and quantization,” in Proc. of International Conference on Artificial Intelligence and Statistics (AISTATS), Virtual Conference, Jun. 2020.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Minsu Kim (115 papers)
  2. Walid Saad (378 papers)
  3. Mohammad Mozaffari (37 papers)
  4. Merouane Debbah (269 papers)
Citations (23)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now