Federated Learning for distribution skewed data using sample weights (2401.02586v1)
Abstract: One of the most challenging issues in federated learning is that the data is often not independent and identically distributed (nonIID). Clients are expected to contribute the same type of data and drawn from one global distribution. However, data are often collected in different ways from different resources. Thus, the data distributions among clients might be different from the underlying global distribution. This creates a weight divergence issue and reduces federated learning performance. This work focuses on improving federated learning performance for skewed data distribution across clients. The main idea is to adjust the client distribution closer to the global distribution using sample weights. Thus, the machine learning model converges faster with higher accuracy. We start from the fundamental concept of empirical risk minimization and theoretically derive a solution for adjusting the distribution skewness using sample weights. To determine sample weights, we implicitly exchange density information by leveraging a neural network-based density estimation model, MADE. The clients data distribution can then be adjusted without exposing their raw data. Our experiment results on three real-world datasets show that the proposed method not only improves federated learning accuracy but also significantly reduces communication costs compared to the other experimental methods.
- B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y. Arcas, “Communication-Efficient Learning of Deep Networks from Decentralized Data,” in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, ser. Proceedings of Machine Learning Research, A. Singh and J. Zhu, Eds., vol. 54. PMLR, 20–22 Apr 2017, pp. 1273–1282. [Online]. Available: https://proceedings.mlr.press/v54/mcmahan17a.html
- Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V. Chandra, “Federated learning with non-iid data,” ArXiv, vol. abs/1806.00582, 2018.
- A. Krizhevsky, V. Nair, and G. Hinton, “Cifar-10 (canadian institute for advanced research).” [Online]. Available: http://www.cs.toronto.edu/ kriz/cifar.html
- P. Warden, “Speech commands: A dataset for limited-vocabulary speech recognition,” 2018.
- A. Hard, K. Rao, R. Mathews, F. Beaufays, S. Augenstein, H. Eichner, C. Kiddon, and D. Ramage, “Federated learning for mobile keyboard prediction,” CoRR, vol. abs/1811.03604, 2018. [Online]. Available: http://arxiv.org/abs/1811.03604
- I. Feki, S. Ammar, Y. Kessentini, and K. Muhammad, “Federated learning for COVID-19 screening from Chest X-ray images,” Applied Soft Computing, vol. 106, p. 107330, Jul. 2021. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S1568494621002532
- S. Zhai, X. Jin, L. Wei, H. Luo, and M. Cao, “Dynamic Federated Learning for GMEC With Time-Varying Wireless Link,” IEEE Access, vol. 9, pp. 10 400–10 412, 2021. [Online]. Available: https://ieeexplore.ieee.org/document/9317862/
- A. K. Singh, D. Saxena, J. Kumar, and V. Gupta, “A quantum approach towards the adaptive prediction of cloud workloads,” IEEE Transactions on Parallel and Distributed Systems, vol. 32, no. 12, pp. 2893–2905, 2021.
- W. Y. B. Lim, J. S. Ng, Z. Xiong, J. Jin, Y. Zhang, D. Niyato, C. Leung, and C. Miao, “Decentralized edge intelligence: A dynamic resource allocation framework for hierarchical federated learning,” IEEE Transactions on Parallel and Distributed Systems, vol. 33, no. 3, pp. 536–550, 2022.
- H. Zhu, J. Xu, S. Liu, and Y. Jin, “Federated learning on non-iid data: A survey,” Neurocomputing, vol. 465, pp. 371–390, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0925231221013254
- A. K. Sahu, T. Li, M. Sanjabi, M. Zaheer, A. S. Talwalkar, and V. Smith, “On the convergence of federated optimization in heterogeneous networks,” ArXiv, vol. abs/1812.06127, 2018.
- S. Itahara, T. Nishio, Y. Koda, M. Morikura, and K. Yamamoto, “Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-iid private data,” IEEE Transactions on Mobile Computing, no. 01, pp. 1–1, mar 2021.
- L. Liu, J. Zhang, S. Song, and K. B. Letaief, “Edge-assisted hierarchical federated learning with non-iid data,” CoRR, vol. abs/1905.06641, 2019.
- T. Shen, J. Zhang, X. Jia, F. Zhang, G. Huang, P. Zhou, F. Wu, and C. Wu, “Federated mutual learning,” ArXiv, vol. abs/2006.16765, 2020.
- H. Wang, Z. Kaplan, D. Niu, and B. Li, “Optimizing federated learning on non-iid data with reinforcement learning,” in IEEE INFOCOM 2020 - IEEE Conference on Computer Communications, 2020, pp. 1698–1707.
- Q. Li, Y. Diao, Q. Chen, and B. He, “Federated learning on non-iid data silos: An experimental study,” CoRR, vol. abs/2102.02079, 2021. [Online]. Available: https://arxiv.org/abs/2102.02079
- X. Zhang, M. Hong, S. V. Dhople, W. Yin, and Y. Liu, “Fedpd: A federated learning framework with optimal rates and adaptivity to non-iid data,” CoRR, vol. abs/2005.11418, 2020. [Online]. Available: https://arxiv.org/abs/2005.11418
- M. Luo, F. Chen, D. Hu, Y. Zhang, J. Liang, and J. Feng, “No fear of heterogeneity: Classifier calibration for federated learning with non-iid data,” in Advances in Neural Information Processing Systems, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., 2021.
- Z. Zhu, J. Hong, and J. Zhou, “Data-free knowledge distillation for heterogeneous federated learning,” 2021.
- X. Li, M. JIANG, X. Zhang, M. Kamp, and Q. Dou, “FedBN: Federated learning on non-IID features via local batch normalization,” in International Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/forum?id=6YEQUn0QICG
- A. K. Sahu, T. Li, M. Sanjabi, M. Zaheer, A. Talwalkar, and V. Smith, “On the convergence of federated optimization in heterogeneous networks,” CoRR, vol. abs/1812.06127, 2018. [Online]. Available: http://arxiv.org/abs/1812.06127
- J. Wang, Q. Liu, H. Liang, G. Joshi, and H. Vincent Poor, “”tackling the objective inconsistency problem in heterogeneous federated optimization”,” ”Advances in Neural Information Processing Systems”, vol. ”2020-December”, ”2020”.
- H. Wang, M. Yurochkin, Y. Sun, D. Papailiopoulos, and Y. Khazaeni, “Federated learning with matched averaging,” in International Conference on Learning Representations, 2020. [Online]. Available: https://openreview.net/forum?id=BkluqlSFDS
- M. Mohri, G. Sivek, and A. T. Suresh, “Agnostic federated learning,” in Proceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15 Jun 2019, pp. 4615–4625. [Online]. Available: https://proceedings.mlr.press/v97/mohri19a.html
- M. Yurochkin, M. Agarwal, S. Ghosh, K. Greenewald, N. Hoang, and Y. Khazaeni, “Bayesian nonparametric federated learning of neural networks,” in Proceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15 Jun 2019, pp. 7252–7261. [Online]. Available: https://proceedings.mlr.press/v97/yurochkin19a.html
- H.-Y. Chen and W.-L. Chao, “On bridging generic and personalized federated learning for image classification,” in International Conference on Learning Representations, 2022. [Online]. Available: https://openreview.net/forum?id=I1hQbx10Kxn
- Y. Tan, G. Long, J. Ma, L. Liu, T. Zhou, and J. Jiang, “Federated learning from pre-trained models: A contrastive learning approach,” 2022.
- M. Wang, J. Guo, and W. Jia, “Fedcl: Federated multi-phase curriculum learning to synchronously correlate user heterogeneity,” 2023.
- T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” in MLSys, 2020. [Online]. Available: https://proceedings.mlsys.org/book/316.pdf
- M. Germain, K. Gregor, I. Murray, and H. Larochelle, “Made: Masked autoencoder for distribution estimation,” in Proceedings of the 32nd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, F. Bach and D. Blei, Eds., vol. 37. Lille, France: PMLR, 07–09 Jul 2015, pp. 881–889. [Online]. Available: https://proceedings.mlr.press/v37/germain15.html
- C. Jin, L. T. Liu, R. Ge, and M. I. Jordan, “On the local minima of the empirical risk,” in Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31. Curran Associates, Inc., 2018.
- A. Menon and C. S. Ong, “Linking losses for density ratio and class-probability estimation,” in Proceedings of The 33rd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, M. F. Balcan and K. Q. Weinberger, Eds., vol. 48. New York, New York, USA: PMLR, 20–22 Jun 2016, pp. 304–313. [Online]. Available: https://proceedings.mlr.press/v48/menon16.html
- L. Deng, “The mnist database of handwritten digit images for machine learning research,” IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 141–142, 2012.
- M. E. H. Chowdhury, T. Rahman, A. Khandakar, R. Mazhar, M. A. Kadir, Z. B. Mahbub, K. R. Islam, M. S. Khan, A. Iqbal, N. Al-Emadi, and M. B. I. Reaz, “Can AI help in screening viral and COVID-19 pneumonia?” CoRR, vol. abs/2003.13145, 2020. [Online]. Available: https://arxiv.org/abs/2003.13145
- S. Jaeger, S. Candemir, S. Antani, Y.-X. Wáng, P.-X. Lu, and G. Thoma, “Two public chest x-ray datasets for computer-aided screening of pulmonary diseases,” Quantitative imaging in medicine and surgery, vol. 4, pp. 475–7, 12 2014.
- D. S. Kermany, K. Zhang, and M. H. Goldbaum, “Labeled optical coherence tomography (oct) and chest x-ray images for classification,” 2018.
- Z. Wang, J. Qiu, Y. Zhou, Y. Shi, L. Fu, W. Chen, and K. B. Letaief, “Federated learning via intelligent reflecting surface,” IEEE Transactions on Wireless Communications, vol. 21, no. 2, pp. 808–822, 2022.