Constructing confidence sets after lasso selection by randomized estimator augmentation

Abstract: Although a few methods have been developed recently for building confidence intervals after model selection, how to construct confidence sets for joint post-selection inference is still an open question. In this paper, we develop a new method to construct confidence sets after lasso variable selection, with strong numerical support for its accuracy and effectiveness. A key component of our method is to sample from the conditional distribution of the response $y$ given the lasso active set, which, in general, is very challenging due to the tiny probability of the conditioning event. We overcome this technical difficulty by using estimator augmentation to simulate from this conditional distribution via Markov chain Monte Carlo given any estimate $\tilde{\mu}$ of the mean $\mu_0$ of $y$. We then incorporate a randomization step for the estimate $\tilde{\mu}$ in our sampling procedure, which may be interpreted as simulating from a posterior predictive distribution by averaging over the uncertainty in $\mu_0$. Our Monte Carlo samples offer great flexibility in the construction of confidence sets for multiple parameters. Extensive numerical results show that our method is able to construct confidence sets with the desired coverage rate and, moreover, that the diameter and volume of our confidence sets are substantially smaller in comparison with a state-of-the-art method.