Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Federated Variational Inference Methods for Structured Latent Variable Models (2302.03314v2)

Published 7 Feb 2023 in stat.ML, cs.LG, stat.CO, and stat.ME

Abstract: Federated learning methods enable model training across distributed data sources without data leaving their original locations and have gained increasing interest in various fields. However, existing approaches are limited, excluding many structured probabilistic models. We present a general and elegant solution based on structured variational inference, widely used in Bayesian machine learning, adapted for the federated setting. Additionally, we provide a communication-efficient variant analogous to the canonical FedAvg algorithm. The proposed algorithms' effectiveness is demonstrated, and their performance is compared with hierarchical Bayesian neural networks and topic models.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, 2016.
  2. Federated learning via posterior averaging: A new perspective and practical algorithms. In International Conference on Learning Representations (ICLR), 2021.
  3. A fixed-point approach to barycenters in Wasserstein space. Journal of Mathematical Analysis and Applications, 2016.
  4. Automatic structured variational inference. In International Conference on Artificial Intelligence and Statistics, 2021.
  5. Partitioned variational inference: A framework for probabilistic federated learning. arXiv preprint arXiv:2202.12275, 2022.
  6. The DeepMind JAX Ecosystem, 2020. URL http://github.com/deepmind.
  7. Latent Dirichlet allocation. The Journal of Machine Learning Research, 2003.
  8. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax.
  9. Importance weighted autoencoders. In International Conference on Learning Representations (ICLR), 2016.
  10. Optimal transport tools (ott): A JAX toolbox for all things Wasserstein. arXiv preprint arXiv:2201.12324, 2022.
  11. Li Deng. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 2012.
  12. Challenges and opportunities in high dimensional variational inference. Advances in Neural Information Processing Systems, 2021.
  13. Federated stochastic gradient langevin dynamics. In Uncertainty in Artificial Intelligence, pages 1703–1712. PMLR, 2021.
  14. A likelihood-based method for analysing longitudinal binary responses. Biometrika, 1993.
  15. Variational inference with locally enhanced bounds for hierarchical models. In International Conference on Machine Learning (ICML), 2022.
  16. Data analysis using regression and multilevel/hierarchical models. Cambridge University Press, 2006.
  17. Federated learning as variational inference: A scalable expectation propagation approach. In The Eleventh International Conference on Learning Representations. PMLR, 2023.
  18. Structured stochastic variational inference. In International Conference on Artificial Intelligence and Statistics, 2015.
  19. The No-U-Turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. The Journal of Machine Learning Research, 2014.
  20. Differentially private variational inference for non-conjugate models. arXiv preprint arXiv:1610.08749, 2016.
  21. Communication-efficient distributed statistical inference. Journal of the American Statistical Association, 2018.
  22. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 2021.
  23. Federated learning for internet of things: Recent advances, taxonomy, and open challenges. IEEE Communications Surveys & Tutorials, 2021.
  24. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
  25. Auto-encoding variational Bayes. In International Conference on Learning Representations (ICLR), 2014.
  26. FedPop: A Bayesian approach for personalised federated learning. Advances in Neural Information Processing Systems, 2022.
  27. Federated learning in mobile edge networks: A comprehensive survey. IEEE Communications Surveys & Tutorials, 2020.
  28. Learning from uncertain curves: The 2-Wasserstein metric for Gaussian processes. Advances in Neural Information Processing Systems, 2017.
  29. Communication-efficient learning of deep networks from decentralized data. In International Conference on Artificial intelligence and statistics, 2017.
  30. Optimizing semantic coherence in topic models. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, 2011.
  31. Kevin P. Murphy. Probabilistic Machine Learning: Advanced Topics. MIT Press, 2023. URL http://probml.github.io/book2.
  32. Asymptotically exact, embarrassingly parallel MCMC. In Thirtieth Conference on Uncertainty in Artificial Intelligence (UAI), 2014.
  33. Gaussian variational approximation with a factor covariance structure. Journal of Computational and Graphical Statistics, 2018.
  34. Scalable Bayesian inference for time series via divide-and-conquer. arXiv preprint arXiv:2106.11043, 2021.
  35. Statistical aspects of Wasserstein distances. Annual Review of Statistics and its application, 2019.
  36. Normalizing flows for probabilistic modeling and inference. The Journal of Machine Learning Research, 2021.
  37. Composable effects for flexible and accelerated probabilistic programming in NumPyro. arXiv preprint arXiv:1912.11554, 2019.
  38. Dg-lmc: a turn-key and scalable synchronous distributed mcmc algorithm via langevin monte carlo within gibbs. In International Conference on Machine Learning, pages 8577–8587. PMLR, 2021.
  39. Gaussian variational approximations for high-dimensional state space models. Bayesian Analysis, 2022.
  40. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, 2010.
  41. Variational inference with normalizing flows. In International Conference on Machine Learning (ICML), 2015.
  42. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning (ICML), 2014.
  43. The future of digital health with federated learning. NPJ Digital Medicine, 2020.
  44. Sticking the landing: Simple, lower-variance gradient estimators for variational inference. Advances in Neural Information Processing Systems, 2017.
  45. Bayes and big data: The consensus Monte Carlo algorithm. International Journal of Management Science and Engineering Management, 2016.
  46. Implicit copula variational inference. arXiv preprint arXiv:2111.09511, 2021.
  47. Autoencoding variational inference for topic models. In International Conference on Learning Representations (ICLR), 2017.
  48. Scalable Bayes via barycenter in Wasserstein space. The Journal of Machine Learning Research, 2018.
  49. Density ratio estimation in machine learning. Cambridge University Press, 2012.
  50. Gaussian variational approximation with sparse precision matrices. Statistics and Computing, 2018.
  51. Conditionally structured variational Gaussian approximation with importance weights. Statistics and Computing, 2020.
  52. Copula variational inference. Advances in Neural Information Processing Systems, 2015.
  53. QLSD: Quantised Langevin stochastic dynamics for Bayesian federated learning. In International Conference on Artificial Intelligence and Statistics, 2022.
  54. Parallelizing MCMC with random partition trees. Advances in Neural Information Processing Systems, 2015.
  55. Federated learning for healthcare informatics. Journal of Healthcare Informatics Research, 2021.
  56. Personalized federated learning via variational Bayesian inference. In International Conference on Machine Learning, pages 26293–26310. PMLR, 2022.
  57. Federated Learning via Variational Bayesian Inference: Personalization, Sparsity and Clustering, 2023.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now