Bayesian model selection in the $\mathcal{M}$-open setting -- Approximate posterior inference and probability-proportional-to-size subsampling for efficient large-scale leave-one-out cross-validation (2005.13199v1)

Abstract: Comparison of competing statistical models is an essential part of psychological research. From a Bayesian perspective, various approaches to model comparison and selection have been proposed in the literature. However, the applicability of these approaches strongly depends on the assumptions about the model space $\mathcal{M}$, the so-called model view. Furthermore, traditional methods like leave-one-out cross-validation (LOO-CV) estimate the expected log predictive density (ELPD) of a model to investigate how the model generalises out-of-sample, which quickly becomes computationally inefficient when sample size becomes large. Here, we provide a tutorial on approximate Pareto-smoothed importance sampling leave-one-out cross-validation (PSIS-LOO), a computationally efficient method for Bayesian model comparison. First, we discuss several model views and the available Bayesian model comparison methods in each. We then use Bayesian logistic regression as a running example how to apply the method in practice, and show that it outperforms other methods like LOO-CV or information criteria in terms of computational effort while providing similarly accurate ELPD estimates. In a second step, we show how even large-scale models can be compared efficiently by using posterior approximations in combination with probability-proportional-to-size subsampling. We show how to compare competing models based on the ELPD estimates provided, and how to conduct posterior predictive checks to safeguard against overconfidence in one of the models under consideration. We conclude that the method is attractive for mathematical psychologists who aim at comparing several competing statistical models, which are possibly high-dimensional and in the big-data regime.