Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

ADEPT: Hierarchical Bayes Approach to Personalized Federated Unsupervised Learning (2402.12537v3)

Published 19 Feb 2024 in cs.LG

Abstract: Statistical heterogeneity of clients' local data is an important characteristic in federated learning, motivating personalized algorithms tailored to the local data statistics. Though there has been a plethora of algorithms proposed for personalized supervised learning, discovering the structure of local data through personalized unsupervised learning is less explored. We initiate a systematic study of such personalized unsupervised learning by developing algorithms based on optimization criteria inspired by a hierarchical Bayesian statistical framework. We develop adaptive algorithms that discover the balance between using limited local data and collaborative information. We do this in the context of two unsupervised learning tasks: personalized dimensionality reduction and personalized diffusion models. We develop convergence analyses for our adaptive algorithms which illustrate the dependence on problem parameters (e.g., heterogeneity, local sample size). We also develop a theoretical framework for personalized diffusion models, which shows the benefits of collaboration even under heterogeneity. We finally evaluate our proposed algorithms using synthetic and real data, demonstrating the effective sample amplification for personalized tasks, induced through collaboration, despite data heterogeneity.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Debiasing model updates for improving personalized federated training. In International Conference on Machine Learning, pages 21–31. PMLR, 2021.
  2. Demystifying MMD GANs. In International Conference on Learning Representations, 2018.
  3. Perfed-gan: Personalized federated learning via generative adversarial networks. IEEE Internet of Things Journal, 10(5):3749–3762, 2023.
  4. Self-aware personalized federated learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  5. Decentralized riemannian gradient descent on the stiefel manifold. In International Conference on Machine Learning, pages 1594–1605. PMLR, 2021.
  6. Exploiting shared representations for personalized federated learning. In Marina Meila and Tong Zhang, editors, International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, pages 2089–2099. PMLR, 2021.
  7. Adaptive personalized federated learning. arXiv preprint arXiv:2003.13461, 2020.
  8. Personalized federated learning with moreau envelopes. In Advances in Neural Information Processing Systems, 2020.
  9. Few-shot learning via learning the representation, provably. In International Conference on Learning Representations, 2021.
  10. Bradley Efron. Large-Scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction. Institute of Mathematical Statistics Monographs. Cambridge University Press, 2010.
  11. Personalized federated learning: A meta-learning approach. In Advances in Neural Information Processing Systems, 2020.
  12. An efficient framework for clustered federated learning. In Advances in Neural Information Processing Systems, 2020.
  13. Federated learning of a mixture of global and local models. arXiv preprint arXiv:2002.05516, 2020.
  14. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
  15. John Immerkær. Fast noise variance estimation. Computer Vision and Image Understanding, 64(2):300–302, 1996.
  16. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1–2):1–210, 2021.
  17. Adaptive gradient-based meta-learning methods. In Advances in Neural Information Processing Systems, 2019.
  18. Understanding diffusion objectives as the ELBO with simple data augmentation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  19. On density estimation with diffusion models. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
  20. Fedpop: A bayesian approach for personalised federated learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  21. Personalized federated recommender systems with private and partially federated autoencoders, 2022.
  22. Federated optimization in heterogeneous networks. In Proceedings of Machine Learning and Systems 2020, MLSys, 2020.
  23. Subject-diffusion: Open domain personalized text-to-image generation without test-time fine-tuning. arXiv preprint arXiv:2307.11410, 2023.
  24. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002.10619, 2020.
  25. Federated multi-task learning under a mixture of distributions. Advances in Neural Information Processing Systems, 34, 2021.
  26. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273–1282. PMLR, 2017.
  27. Fine-tuning diffusion models with limited data. In NeurIPS 2022 Workshop on Score-Based Methods, 2022.
  28. A statistical framework for personalized federated learning and estimation: Theory, algorithms, and privacy. In International Conference on Learning Representations, 2023.
  29. Personalized PCA for federated heterogeneous data. In 2023 IEEE International Symposium on Information Theory (ISIT), pages 168–173. IEEE, 2023.
  30. Quped: Quantized personalization via distillation with applications to federated learning. Advances in Neural Information Processing Systems, 34, 2021.
  31. Fedgm: Heterogeneous federated learning via generative learning and mutual distillation. In Euro-Par 2023: Parallel Processing: 29th International Conference on Parallel and Distributed Computing, Limassol, Cyprus, August 28 – September 1, 2023, Proceedings, page 339–351, Berlin, Heidelberg, 2023. Springer-Verlag.
  32. Mirko Polato. Federated variational autoencoder for collaborative filtering. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1–8, 2021.
  33. Variational inference with normalizing flows. In International conference on machine learning, pages 1530–1538. PMLR, 2015.
  34. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500–22510, 2023.
  35. Personalized PCA: Decoupling shared and unique features. arXiv preprint arXiv:2207.08041, 2022.
  36. Federated multi-task learning. In Advances in Neural Information Processing Systems, pages 4424–4434, 2017.
  37. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 2256–2265. JMLR.org, 2015.
  38. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
  39. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
  40. Rethinking few-shot image classification: a good embedding is all you need? In European Conference on Computer Vision, pages 266–282. Springer, 2020.
  41. Probabilistic principal component analysis. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 61(3):611–622, 1999.
  42. Federated learning with diffusion models for privacy-sensitive vision tasks. In 2023 International Conference on Advanced Technologies for Communications (ATC), pages 305–310. IEEE, 2023.
  43. Decentralized collaborative learning of personalized models over networks. In Artificial Intelligence and Statistics, pages 509–517. PMLR, 2017.
  44. Personalized federated learning on non-iid data via group-based meta-learning. ACM Transactions on Knowledge Discovery from Data, 17(4):1–20, 2023.
  45. Fully decentralized joint learning of personalized models and collaboration graphs. In International Conference on Artificial Intelligence and Statistics, pages 864–874. PMLR, 2020.
  46. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023.
  47. Personalized federated learning with first order model optimization. In International Conference on Learning Representations, 2021.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets