Moment Propagation

Abstract: We introduce and develop moment propagation for approximate Bayesian inference. This method can be viewed as a variance correction for mean field variational Bayes which tends to underestimate posterior variances. Focusing on the case where the model is described by two sets of parameter vectors, we develop moment propagation algorithms for linear regression, multivariate normal, and probit regression models. We show for the probit regression model that moment propagation empirically performs reasonably well for several benchmark datasets. Finally, we discuss theoretical gaps and future extensions. In the supplementary material we show heuristically why moment propagation leads to appropriate posterior variance estimation, for the linear regression and multivariate normal models we show precisely why mean field variational Bayes underestimates certain moments, and prove that our moment propagation algorithm recovers the exact marginal posterior distributions for all parameters, and for probit regression we show that moment propagation provides asymptotically correct posterior means and covariance estimates.