Inference after black box selection

Abstract: We consider the problem of inference for parameters selected to report only after some algorithm, the canonical example being inference for model parameters after a model selection procedure. The conditional correction for selection requires knowledge of how the selection is affected by changes in the underlying data, and current research explicitly describes this selection. In this work, we assume 1) we have in silico access to the selection algorithm and 2) for parameters of interest, the data input into the algorithm satisfies (pre-selection) a central limit theorem jointly with an estimator of our parameter of interest. Under these assumptions, we recast the problem into a statistical learning problem which can be fit with off-the-shelf models for binary regression. The feature points in this problem are set by the user, opening up the possibility of active learning methods for computationally expensive selection algorithms. We consider two examples previously out of reach of this conditional approach: stability selection and multiple cross-validation.