Papers
Topics
Authors
Recent
Search
2000 character limit reached

Bayesian Personalized Federated Learning with Shared and Personalized Uncertainty Representations

Published 27 Sep 2023 in cs.LG | (2309.15499v2)

Abstract: Bayesian personalized federated learning (BPFL) addresses challenges in existing personalized FL (PFL). BPFL aims to quantify the uncertainty and heterogeneity within and across clients towards uncertainty representations by addressing the statistical heterogeneity of client data. In PFL, some recent preliminary work proposes to decompose hidden neural representations into shared and local components and demonstrates interesting results. However, most of them do not address client uncertainty and heterogeneity in FL systems, while appropriately decoupling neural representations is challenging and often ad hoc. In this paper, we make the first attempt to introduce a general BPFL framework to decompose and jointly learn shared and personalized uncertainty representations on statistically heterogeneous client data over time. A Bayesian federated neural network BPFed instantiates BPFL by jointly learning cross-client shared uncertainty and client-specific personalized uncertainty over statistically heterogeneous and randomly participating clients. We further involve continual updating of prior distribution in BPFed to speed up the convergence and avoid catastrophic forgetting. Theoretical analysis and guarantees are provided in addition to the experimental evaluation of BPFed against the diversified baselines.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in AISTATS, pp. 1273–1282, PMLR, 2017.
  2. F. Sattler, T. Korjakow, R. Rischke, and W. Samek, “Fedaux: Leveraging unlabeled auxiliary data in federated learning,” IEEE Trans. Neural Networks Learn. Syst., vol. 34, no. 9, pp. 5531–5543, 2023.
  3. M. Cao, L. Zhang, and B. Cao, “Toward on-device federated learning: A direct acyclic graph-based blockchain approach,” IEEE Trans. Neural Networks Learn. Syst., vol. 34, no. 4, pp. 2028–2042, 2023.
  4. Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V. Chandra, “Federated learning with non-iid data,” arXiv preprint arXiv:1806.00582, 2018.
  5. L. Cao, “Beyond i.i.d.: Non-iid thinking, informatics, and learning,” IEEE Intell. Syst., vol. 37, no. 4, pp. 5–17, 2022.
  6. X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang, “On the convergence of fedavg on non-iid data,” in ICLR, 2020.
  7. A. Fallah, A. Mokhtari, and A. Ozdaglar, “Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach,” NeurIPS, vol. 33, pp. 3557–3568, 2020.
  8. L. Collins, H. Hassani, A. Mokhtari, and S. Shakkottai, “Exploiting shared representations for personalized federated learning,” in ICML, pp. 2089–2099, PMLR, 2021.
  9. X. Zhang, Y. Li, W. Li, K. Guo, and Y. Shao, “Personalized federated learning via variational bayesian inference,” in ICML, pp. 26293–26310, PMLR, 2022.
  10. V. Smith, C. Chiang, M. Sanjabi, and A. Talwalkar, “Federated multi-task learning,” in NeurIPS, pp. 4424–4434, 2017.
  11. M. Al-Shedivat, L. Li, E. Xing, and A. Talwalkar, “On data efficiency of meta-learning,” in AISTATS, pp. 1369–1377, PMLR, 2021.
  12. Y. Mansour, M. Mohri, J. Ro, and A. T. Suresh, “Three approaches for personalization with applications to federated learning,” arXiv preprint arXiv:2002.10619, 2020.
  13. Y. Huang, L. Chu, Z. Zhou, L. Wang, J. Liu, J. Pei, and Y. Zhang, “Personalized cross-silo federated learning on non-iid data,” in AAAI, vol. 35, pp. 7865–7873, 2021.
  14. C. T Dinh, N. Tran, and J. Nguyen, “Personalized federated learning with moreau envelopes,” NeurIPS, vol. 33, pp. 21394–21405, 2020.
  15. K. Pillutla, K. Malik, A.-R. Mohamed, M. Rabbat, M. Sanjabi, and L. Xiao, “Federated learning with partial model personalization,” in ICML, pp. 17716–17758, PMLR, 2022.
  16. M. G. Arivazhagan, V. Aggarwal, A. K. Singh, and S. Choudhary, “Federated learning with personalization layers,” arXiv preprint arXiv:1912.00818, 2019.
  17. N. Kotelevskii, M. Vono, E. Moulines, and A. Durmus, “Fedpop: A bayesian approach for personalised federated learning,” NeurIPS, 2022.
  18. J. Snell and R. Zemel, “Bayesian few-shot classification with one-vs-each p\\\backslash\’olya-gamma augmented gaussian processes,” ICLR, 2021.
  19. L. V. Jospin, H. Laga, F. Boussaid, W. Buntine, and M. Bennamoun, “Hands-on bayesian neural networks—a tutorial for deep learning users,” IEEE Computational Intelligence Magazine, vol. 17, no. 2, pp. 29–48, 2022.
  20. L. Cao, H. Chen, X. Fan, J. Gama, Y.-S. Ong, and V. Kumar, “Bayesian federated learning: A survey,” IJCAI, 2023.
  21. S. Bhatt, A. Gupta, and P. Rai, “Bayesian federated learning via predictive distribution distillation,” arXiv preprint arXiv:2206.07562, 2022.
  22. Z. Dai, B. K. H. Low, and P. Jaillet, “Federated bayesian optimization via thompson sampling,” NeurIPS, vol. 33, pp. 9687–9699, 2020.
  23. L. Zang, Y. Qin, and R. Li, “Traffic flow prediction based on federated learning with joint pca compression and bayesian optimization,” in SMC, pp. 3330–3335, IEEE, 2022.
  24. M. Tang, X. Ning, Y. Wang, J. Sun, Y. Wang, H. Li, and Y. Chen, “Fedcor: Correlation-based active client selection strategy for heterogeneous federated learning,” in CVPR, pp. 10102–10111, 2022.
  25. I. Achituve, A. Shamsian, A. Navon, G. Chechik, and E. Fetaya, “Personalized federated learning with gaussian processes,” NeurIPS, vol. 34, pp. 8392–8406, 2021.
  26. M. Yurochkin, M. Agarwal, S. Ghosh, K. Greenewald, N. Hoang, and Y. Khazaeni, “Bayesian nonparametric federated learning of neural networks,” in ICML, pp. 7252–7261, PMLR, 2019.
  27. H. Wang, M. Yurochkin, Y. Sun, D. Papailiopoulos, and Y. Khazaeni, “Federated learning with matched averaging,” arXiv preprint arXiv:2002.06440, 2020.
  28. H.-Y. Chen and W.-L. Chao, “Fedbe: Making bayesian model ensemble applicable to federated learning,” ICLR, 2021.
  29. M. Al-Shedivat, J. Gillenwater, E. Xing, and A. Rostamizadeh, “Federated learning via posterior averaging: A new perspective and practical algorithms,” ICLR, 2021.
  30. M. Vono, V. Plassier, A. Durmus, A. Dieuleveut, and E. Moulines, “Qlsd: Quantised langevin stochastic dynamics for bayesian federated learning,” in AISTATS, pp. 6459–6500, PMLR, 2022.
  31. P. P. Liang, T. Liu, L. Ziyin, N. B. Allen, R. P. Auerbach, D. Brent, R. Salakhutdinov, and L.-P. Morency, “Think locally, act globally: Federated learning with local and global representations,” NeurIPS Workshop on Federated Learning, 2019.
  32. H.-Y. Chen and W.-L. Chao, “On bridging generic and personalized federated learning for image classification,” ICLR, 2022.
  33. R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa, “Natural language processing (almost) from scratch,” JMLR, vol. 12, no. ARTICLE, pp. 2493–2537, 2011.
  34. D. M. Blei, A. Kucukelbir, and J. D. McAuliffe, “Variational inference: A review for statisticians,” Journal of the American statistical Association, vol. 112, no. 518, pp. 859–877, 2017.
  35. H. Lan, Z. Liu, J. H. Hsiao, D. Yu, and A. B. Chan, “Clustering hidden markov models with variational bayesian hierarchical EM,” IEEE Trans. Neural Networks Learn. Syst., vol. 34, no. 3, pp. 1537–1551, 2023.
  36. F. Ye and A. G. Bors, “Lifelong mixture of variational autoencoders,” IEEE Trans. Neural Networks Learn. Syst., vol. 34, no. 1, pp. 461–474, 2023.
  37. D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114, 2013.
  38. S. M. Shah and V. K. N. Lau, “Model compression for communication efficient federated learning,” IEEE Trans. Neural Networks Learn. Syst., vol. 34, no. 9, pp. 5937–5951, 2023.
  39. N. G. Polson and V. Ročková, “Posterior concentration for sparse deep learning,” NeurIPS, vol. 31, 2018.
  40. J. Bai, Q. Song, and G. Cheng, “Efficient variational inference for sparse deep learning with theoretical guarantee,” NeurIPS, vol. 33, pp. 466–476, 2020.
  41. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
  42. H. Xiao, K. Rasul, and R. Vollgraf, “Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms,” arXiv preprint arXiv:1708.07747, 2017.
  43. A. Krizhevsky, G. Hinton, et al., “Learning multiple layers of features from tiny images,” Technical report, University of Toronto, 2009.
  44. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  45. C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in ICML, pp. 1321–1330, PMLR, 2017.
  46. M. Minderer, J. Djolonga, R. Romijnders, F. Hubis, X. Zhai, N. Houlsby, D. Tran, and M. Lucic, “Revisiting the calibration of modern neural networks,” NeurIPS, vol. 34, pp. 15682–15694, 2021.
  47. Oxford university press, 2013.
  48. D. Pati, A. Bhattacharya, and Y. Yang, “On statistical optimality of variational bayes,” in AISTATS, pp. 1579–1588, PMLR, 2018.
Citations (3)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up