Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
117 tokens/sec
GPT-4o
8 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Federated Learning for distribution skewed data using sample weights (2401.02586v1)

Published 5 Jan 2024 in cs.LG and cs.AI

Abstract: One of the most challenging issues in federated learning is that the data is often not independent and identically distributed (nonIID). Clients are expected to contribute the same type of data and drawn from one global distribution. However, data are often collected in different ways from different resources. Thus, the data distributions among clients might be different from the underlying global distribution. This creates a weight divergence issue and reduces federated learning performance. This work focuses on improving federated learning performance for skewed data distribution across clients. The main idea is to adjust the client distribution closer to the global distribution using sample weights. Thus, the machine learning model converges faster with higher accuracy. We start from the fundamental concept of empirical risk minimization and theoretically derive a solution for adjusting the distribution skewness using sample weights. To determine sample weights, we implicitly exchange density information by leveraging a neural network-based density estimation model, MADE. The clients data distribution can then be adjusted without exposing their raw data. Our experiment results on three real-world datasets show that the proposed method not only improves federated learning accuracy but also significantly reduces communication costs compared to the other experimental methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y. Arcas, “Communication-Efficient Learning of Deep Networks from Decentralized Data,” in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, ser. Proceedings of Machine Learning Research, A. Singh and J. Zhu, Eds., vol. 54.   PMLR, 20–22 Apr 2017, pp. 1273–1282. [Online]. Available: https://proceedings.mlr.press/v54/mcmahan17a.html
  2. Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V. Chandra, “Federated learning with non-iid data,” ArXiv, vol. abs/1806.00582, 2018.
  3. A. Krizhevsky, V. Nair, and G. Hinton, “Cifar-10 (canadian institute for advanced research).” [Online]. Available: http://www.cs.toronto.edu/ kriz/cifar.html
  4. P. Warden, “Speech commands: A dataset for limited-vocabulary speech recognition,” 2018.
  5. A. Hard, K. Rao, R. Mathews, F. Beaufays, S. Augenstein, H. Eichner, C. Kiddon, and D. Ramage, “Federated learning for mobile keyboard prediction,” CoRR, vol. abs/1811.03604, 2018. [Online]. Available: http://arxiv.org/abs/1811.03604
  6. I. Feki, S. Ammar, Y. Kessentini, and K. Muhammad, “Federated learning for COVID-19 screening from Chest X-ray images,” Applied Soft Computing, vol. 106, p. 107330, Jul. 2021. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S1568494621002532
  7. S. Zhai, X. Jin, L. Wei, H. Luo, and M. Cao, “Dynamic Federated Learning for GMEC With Time-Varying Wireless Link,” IEEE Access, vol. 9, pp. 10 400–10 412, 2021. [Online]. Available: https://ieeexplore.ieee.org/document/9317862/
  8. A. K. Singh, D. Saxena, J. Kumar, and V. Gupta, “A quantum approach towards the adaptive prediction of cloud workloads,” IEEE Transactions on Parallel and Distributed Systems, vol. 32, no. 12, pp. 2893–2905, 2021.
  9. W. Y. B. Lim, J. S. Ng, Z. Xiong, J. Jin, Y. Zhang, D. Niyato, C. Leung, and C. Miao, “Decentralized edge intelligence: A dynamic resource allocation framework for hierarchical federated learning,” IEEE Transactions on Parallel and Distributed Systems, vol. 33, no. 3, pp. 536–550, 2022.
  10. H. Zhu, J. Xu, S. Liu, and Y. Jin, “Federated learning on non-iid data: A survey,” Neurocomputing, vol. 465, pp. 371–390, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0925231221013254
  11. A. K. Sahu, T. Li, M. Sanjabi, M. Zaheer, A. S. Talwalkar, and V. Smith, “On the convergence of federated optimization in heterogeneous networks,” ArXiv, vol. abs/1812.06127, 2018.
  12. S. Itahara, T. Nishio, Y. Koda, M. Morikura, and K. Yamamoto, “Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-iid private data,” IEEE Transactions on Mobile Computing, no. 01, pp. 1–1, mar 2021.
  13. L. Liu, J. Zhang, S. Song, and K. B. Letaief, “Edge-assisted hierarchical federated learning with non-iid data,” CoRR, vol. abs/1905.06641, 2019.
  14. T. Shen, J. Zhang, X. Jia, F. Zhang, G. Huang, P. Zhou, F. Wu, and C. Wu, “Federated mutual learning,” ArXiv, vol. abs/2006.16765, 2020.
  15. H. Wang, Z. Kaplan, D. Niu, and B. Li, “Optimizing federated learning on non-iid data with reinforcement learning,” in IEEE INFOCOM 2020 - IEEE Conference on Computer Communications, 2020, pp. 1698–1707.
  16. Q. Li, Y. Diao, Q. Chen, and B. He, “Federated learning on non-iid data silos: An experimental study,” CoRR, vol. abs/2102.02079, 2021. [Online]. Available: https://arxiv.org/abs/2102.02079
  17. X. Zhang, M. Hong, S. V. Dhople, W. Yin, and Y. Liu, “Fedpd: A federated learning framework with optimal rates and adaptivity to non-iid data,” CoRR, vol. abs/2005.11418, 2020. [Online]. Available: https://arxiv.org/abs/2005.11418
  18. M. Luo, F. Chen, D. Hu, Y. Zhang, J. Liang, and J. Feng, “No fear of heterogeneity: Classifier calibration for federated learning with non-iid data,” in Advances in Neural Information Processing Systems, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., 2021.
  19. Z. Zhu, J. Hong, and J. Zhou, “Data-free knowledge distillation for heterogeneous federated learning,” 2021.
  20. X. Li, M. JIANG, X. Zhang, M. Kamp, and Q. Dou, “FedBN: Federated learning on non-IID features via local batch normalization,” in International Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/forum?id=6YEQUn0QICG
  21. A. K. Sahu, T. Li, M. Sanjabi, M. Zaheer, A. Talwalkar, and V. Smith, “On the convergence of federated optimization in heterogeneous networks,” CoRR, vol. abs/1812.06127, 2018. [Online]. Available: http://arxiv.org/abs/1812.06127
  22. J. Wang, Q. Liu, H. Liang, G. Joshi, and H. Vincent Poor, “”tackling the objective inconsistency problem in heterogeneous federated optimization”,” ”Advances in Neural Information Processing Systems”, vol. ”2020-December”, ”2020”.
  23. H. Wang, M. Yurochkin, Y. Sun, D. Papailiopoulos, and Y. Khazaeni, “Federated learning with matched averaging,” in International Conference on Learning Representations, 2020. [Online]. Available: https://openreview.net/forum?id=BkluqlSFDS
  24. M. Mohri, G. Sivek, and A. T. Suresh, “Agnostic federated learning,” in Proceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97.   PMLR, 09–15 Jun 2019, pp. 4615–4625. [Online]. Available: https://proceedings.mlr.press/v97/mohri19a.html
  25. M. Yurochkin, M. Agarwal, S. Ghosh, K. Greenewald, N. Hoang, and Y. Khazaeni, “Bayesian nonparametric federated learning of neural networks,” in Proceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97.   PMLR, 09–15 Jun 2019, pp. 7252–7261. [Online]. Available: https://proceedings.mlr.press/v97/yurochkin19a.html
  26. H.-Y. Chen and W.-L. Chao, “On bridging generic and personalized federated learning for image classification,” in International Conference on Learning Representations, 2022. [Online]. Available: https://openreview.net/forum?id=I1hQbx10Kxn
  27. Y. Tan, G. Long, J. Ma, L. Liu, T. Zhou, and J. Jiang, “Federated learning from pre-trained models: A contrastive learning approach,” 2022.
  28. M. Wang, J. Guo, and W. Jia, “Fedcl: Federated multi-phase curriculum learning to synchronously correlate user heterogeneity,” 2023.
  29. T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” in MLSys, 2020. [Online]. Available: https://proceedings.mlsys.org/book/316.pdf
  30. M. Germain, K. Gregor, I. Murray, and H. Larochelle, “Made: Masked autoencoder for distribution estimation,” in Proceedings of the 32nd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, F. Bach and D. Blei, Eds., vol. 37.   Lille, France: PMLR, 07–09 Jul 2015, pp. 881–889. [Online]. Available: https://proceedings.mlr.press/v37/germain15.html
  31. C. Jin, L. T. Liu, R. Ge, and M. I. Jordan, “On the local minima of the empirical risk,” in Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31.   Curran Associates, Inc., 2018.
  32. A. Menon and C. S. Ong, “Linking losses for density ratio and class-probability estimation,” in Proceedings of The 33rd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, M. F. Balcan and K. Q. Weinberger, Eds., vol. 48.   New York, New York, USA: PMLR, 20–22 Jun 2016, pp. 304–313. [Online]. Available: https://proceedings.mlr.press/v48/menon16.html
  33. L. Deng, “The mnist database of handwritten digit images for machine learning research,” IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 141–142, 2012.
  34. M. E. H. Chowdhury, T. Rahman, A. Khandakar, R. Mazhar, M. A. Kadir, Z. B. Mahbub, K. R. Islam, M. S. Khan, A. Iqbal, N. Al-Emadi, and M. B. I. Reaz, “Can AI help in screening viral and COVID-19 pneumonia?” CoRR, vol. abs/2003.13145, 2020. [Online]. Available: https://arxiv.org/abs/2003.13145
  35. S. Jaeger, S. Candemir, S. Antani, Y.-X. Wáng, P.-X. Lu, and G. Thoma, “Two public chest x-ray datasets for computer-aided screening of pulmonary diseases,” Quantitative imaging in medicine and surgery, vol. 4, pp. 475–7, 12 2014.
  36. D. S. Kermany, K. Zhang, and M. H. Goldbaum, “Labeled optical coherence tomography (oct) and chest x-ray images for classification,” 2018.
  37. Z. Wang, J. Qiu, Y. Zhou, Y. Shi, L. Fu, W. Chen, and K. B. Letaief, “Federated learning via intelligent reflecting surface,” IEEE Transactions on Wireless Communications, vol. 21, no. 2, pp. 808–822, 2022.
Citations (3)

Summary

We haven't generated a summary for this paper yet.