Rényi's $α$-divergence variational Bayes for spike-and-slab high-dimensional linear regression

Abstract: Sparse high-dimensional linear regression is a central problem in statistics, where the goal is often variable selection and/or coefficient estimation. We propose a mean-field variational Bayes approximation for sparse regression with spike-and-slab Laplace priors that replaces the standard Kullback-Leibler (KL) divergence objective with the Rényi's $α$ divergence: a one-parameter generalization of KL divergence indexed by $α\in (0, \infty) \setminus {1}$ that allows flexibility between zero-forcing and mass-covering behavior. We derive coordinate ascent variational inference (CAVI) updates via a second-order delta method and develop a stochastic variational inference algorithm based on a Monte Carlo surrogate Rényi lower bound. In simulations, our two methods perform comparably to state-of-the-art Bayesian variable selection procedures across a range of sparsity configurations and $α$ values for both variable selection and estimation, and our numerical results illustrate how different choices of $α$ can be advantageous in different sparsity configurations.