Papers
Topics
Authors
Recent
Search
2000 character limit reached

Frequentist Guarantees of Distributed (Non)-Bayesian Inference

Published 14 Nov 2023 in math.ST, stat.ML, and stat.TH | (2311.08214v4)

Abstract: Motivated by the need to analyze large, decentralized datasets, distributed Bayesian inference has become a critical research area across multiple fields, including statistics, electrical engineering, and economics. This paper establishes Frequentist properties, such as posterior consistency, asymptotic normality, and posterior contraction rates, for the distributed (non-)Bayes Inference problem among agents connected via a communication network. Our results show that, under appropriate assumptions on the communication graph, distributed Bayesian inference retains parametric efficiency while enhancing robustness in uncertainty quantification. We also explore the trade-off between statistical efficiency and communication efficiency by examining how the design and size of the communication graph impact the posterior contraction rate. Furthermore, We extend our analysis to time-varying graphs and apply our results to exponential family models, distributed logistic regression, and decentralized detection models.

Authors (2)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (53)
  1. Bayesian learning in social networks. The Review of Economic Studies, 78(4):1201–1236, 2011.
  2. Distributed estimation and inference with statistical guarantees. arXiv preprint arXiv:1509.05457, 2015.
  3. Peter J Bickel and Bas J K Kleijn. The semiparametric bernstein–von mises theorem. The Annals of Statistics, 40:206–237, 2012. ISSN 0090-5364.
  4. Some contributions to the asymptotic theory of bayes solutions. Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete, 11:257–276, 1969. ISSN 1432-2064.
  5. The bernstein–von mises theorem and nonregular models. The Annals of Statistics, 42(5):1850–1878, 2014.
  6. Asymptotic agreement in distributed estimation. IEEE transactions on automatic control, 27(3):650–655, 1982.
  7. Asymptotics in statistics: some basic concepts. Springer Science & Business Media, 2000. ISBN 0387950362.
  8. A bernstein–von mises theorem for smooth functionals in semiparametric models. The Annals of Statistics, 43:2353–2383, 2015. ISSN 0090-5364.
  9. Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
  10. Morris H DeGroot. Reaching a consensus. Journal of the American Statistical association, 69(345):118–121, 1974.
  11. Non-bayesian learning. The BE Journal of Theoretical Economics, 10:1–20, 2010.
  12. On the evolution of random graphs. Publ. math. inst. hung. acad. sci, 5(1):17–60, 1960.
  13. Subhashis Ghosal and Aad Van der Vaart. Fundamentals of nonparametric Bayesian inference, volume 44. Cambridge University Press, 2017.
  14. Bayesian Nonparametrics. Springer-Verlag, 2003. doi: 10.1007/b97842. URL https://doi.org/10.1007%2Fb97842.
  15. On reaching a consensus using degroot’s iterative pooling. The Annals of Statistics, 21(1):391–401, 1993.
  16. Learning in social networks. Available at SSRN 2919146, 2017.
  17. John A Gubner. Distributed estimation and quantization. IEEE Transactions on Information Theory, 39(4):1456–1459, 1993.
  18. Stochastic variational inference. Journal of Machine Learning Research, 2013.
  19. Non-bayesian social learning. Games and Economic Behavior, 76:210–225, 2012. ISSN 0899-8256.
  20. Communication-efficient distributed statistical inference. Journal of the American Statistical Association, 2018.
  21. Anya Katsevich. Improved scaling with dimension in the bernstein-von mises theorem for two statistical models. arXiv preprint arXiv:2308.06899, 2023.
  22. Bas J K Kleijn and Aad W Van der Vaart. The bernstein-von-mises theorem under misspecification. Electronic Journal of Statistics, 6:354–381, 2012. ISSN 1935-7524.
  23. An optimization-centric view on bayes’ rule: Reviewing and generalizing variational inference. Journal of Machine Learning Research, 23(132):1–109, 2022.
  24. Social learning and distributed hypothesis testing. In 2014 IEEE International Symposium on Information Theory, pages 551–555. IEEE, 2014.
  25. Pierre-Simon Laplace. Supplément au mémoire sur les approximations des formules qui sont fonction de très grands nombres. Mémoires de l’Académie Royale des sciences de Paris, 1810:559–565, 1809.
  26. Lucien LeCam. On some asymptotic properties of maximum likelihood estimates and related bayes estimates. Univ. California Pub. Statist., 1:277–330, 1953.
  27. Simple, scalable and accurate posterior interval estimation. Biometrika, 104:665–680, 2017. ISSN 0006-3444.
  28. On the robustness to misspecification of α𝛼\alphaitalic_α-posteriors and their variational approximations. Journal of Machine Learning Research, 23(147):1–51, 2022.
  29. Jeffrey W Miller. Asymptotic normality, concentration, and coverage of generalized posteriors. J. Mach. Learn. Res., 22:168–1, 2021.
  30. Scalable and robust bayesian inference via the median posterior. In International conference on machine learning, pages 1656–1664. PMLR, 2014.
  31. Robust and scalable bayes via a median of subset posterior measures. The Journal of Machine Learning Research, 18:4488–4527, 2017. ISSN 1532-4435.
  32. A theory of non-bayesian social learning. Econometrica, 86(2):445–490, 2018a.
  33. A theory of non‐bayesian social learning. Econometrica, 86:445–490, 2018b. ISSN 0012-9682.
  34. Fast convergence rates for distributed non-bayesian learning. IEEE Transactions on Automatic Control, 62:5538–5553, 2017. ISSN 0018-9286.
  35. Asymptotically exact, embarrassingly parallel mcmc. arXiv preprint arXiv:1311.4780, 2013.
  36. Finite sample bernstein–von mises theorem for semiparametric problems. Bayesian Analysis, 10(3):665–710, 2015.
  37. Distributed detection via bayesian updates and consensus. In 2015 34th Chinese Control Conference (CCC), pages 6992–6997. IEEE, 2015.
  38. Variational consensus monte carlo. Advances in Neural Information Processing Systems, 28, 2015.
  39. Learning without recall by random walks on directed graphs. In 2015 54th IEEE Conference on Decision and Control (CDC), pages 5538–5543. IEEE, 2015.
  40. Jeffrey S Rosenthal. Convergence rates for markov chains. Siam Review, 37(3):387–405, 1995.
  41. Lorraine Schwartz. On bayes procedures. Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete, 4:10–26, 1965. ISSN 1432-2064.
  42. Bayes and big data: The consensus monte carlo algorithm. International Journal of Management Science and Engineering Management, 11:78–88, 2016. ISSN 1750-9653.
  43. Distributed detection: Finite-time analysis and impact of network topology. IEEE Transactions on Automatic Control, 61:3256–3268, 2015. ISSN 0018-9286.
  44. Xiaotong Shen. Asymptotic normality of semiparametric and nonparametric posterior distributions. Journal of the American Statistical Association, 97:222–235, 2002. ISSN 0162-1459.
  45. Convergence and asymptotic agreement in distributed decision problems. IEEE Transactions on Automatic Control, 29(1):42–50, 1984.
  46. Non-asymptotic concentration rates in cooperative learning part i: Variational non-bayesian social learning. IEEE Transactions on Control of Network Systems, 2022a. ISSN 2325-5870.
  47. Non-asymptotic concentration rates in cooperative learning part ii: Inference on compact hypothesis sets. IEEE Transactions on Control of Network Systems, 2022b. ISSN 2325-5870.
  48. Empirical processes. In Weak Convergence and Empirical Processes: With Applications to Statistics, pages 127–384. Springer, 2023.
  49. Aad W. Van der Vaart. Asymptotic Statistics, volume 3. Cambridge University Press, 2000. ISBN 0521784506.
  50. Stephen G Walker. Bayesian inference via a minimization rule. Sankhyā: The Indian Journal of Statistics, pages 542–553, 2006. ISSN 0972-7671.
  51. Parallelizing mcmc via weierstrass sampler. arXiv preprint arXiv:1312.4605, 2013.
  52. Parallelizing mcmc with random partition trees. Advances in neural information processing systems, 28, 2015.
  53. Frequentist consistency of variational bayes. Journal of the American Statistical Association, 114(527):1147–1161, 2019.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 4 tweets with 3 likes about this paper.

Sign Up for Free