A Leakage Bound for Confidence Sets after Black-Box Selection

Abstract: In many analyses the object reported at the end is not fixed in advance, but is chosen after a preliminary search over variables, subgroups, transformations, models or contrasts. Classical selective-inference methods are most effective when this search can be written as an explicit selection event. This note treats the less structured case in which the selection rule is a black box and inference is required for the target indexed by the selected object. We show that, for any fixed-target confidence procedure, selected-target noncoverage is bounded by the nominal fixed-target noncoverage plus the average total variation distance between the marginal law of the inferential data and its conditional law given the selected object. A mutual-information bound follows immediately. The result recovers sample splitting as the zero-leakage case and gives explicit guarantees for noisy screening through a Gaussian information bound. Thus the inferential cost of black-box selection is quantified by the information that the selected object carries about the inferential sample.

• The paper introduces a finite-sample leakage bound that quantifies how black-box selection increases noncoverage risk, framing the added risk via the expected total variation distance between marginal and conditional distributions.
• It establishes a mutual information upper bound using Pinsker’s inequality to operationalize selection-induced leakage, allowing inferential guarantees without explicit selection event characterization.
• Practical strategies such as sample splitting and noise injection are proposed to mitigate leakage effects, with simulation studies validating the trade-offs in coverage probabilities.

Finite-Sample Leakage Bounds for Confidence Sets after Black-Box Selection

Background and Motivation

Post-selection inference concerns the quantification of statistical uncertainty after a data-dependent selection process—for example, when variables, models, or contrasts are chosen based on the observed data before formal inference is performed. Classical selective inference methodologies, such as those conditioning on explicit selection events or providing universally valid PoSI procedures, presume the selection mechanism is tractable or can be articulated analytically. However, practical scenarios frequently involve selection rules that are algorithmic, stochastic, private, or proprietary, for which characterizing the selection event is infeasible. In these black-box settings, proper inferential guarantees for reported confidence sets are nontrivial due to potential selection-induced information leakage.

Main Results and Theoretical Contributions

The core contribution is a finite-sample coverage inequality that quantifies the additional inferential risk from black-box selection. Formally, for any fixed-target confidence procedure $C_s(D)$ with marginal noncoverage bounded by $\alpha$ for every admissible $s$, the probability of noncoverage for the selected target $\theta_{S^\star}(P)$ satisfies: $$\mathrm{pr}_P\{\theta_{S^\star}(P)\notin C_{S^\star}(D)\} \leq \alpha+\Lambda_{\mathrm{TV}}(P),$$ where $$\Lambda_{\mathrm{TV}}(P) = E_P[ d_{\mathrm{TV}}(\mathcal{L}_P(D \mid S^\star), \mathcal{L}_P(D)) ]$$ denotes the expected total variation distance between the marginal law of the inferential data and its conditional law given the selected object.

A mutual information upper bound is derived via Pinsker's and Jensen's inequalities: $$\mathrm{pr}_P\{\theta_{S^\star}(P)\notin C_{S^\star}(D)\} \leq \alpha + ( I_P(S^\star; D)/2 )^{1/2},$$ which operationalizes the leakage in terms of mutual information between the inferential sample and selection output.

The bound is tight: a sharpness result exhibits instances where the additive risk matches $\alpha + \Lambda_{\mathrm{TV}}(P)$, precluding further improvement in the general unrestricted class.

This formulation decouples the inferential validity from the mathematical characterization of the selection; it depends only on the change in law induced by selection, sidestepping the need for conditioning on complex or unknown selection events.

Practical Implications and Design Principles

This leakage perspective offers concrete guidelines for the design of selection-inference workflows under black-box or adaptive selection mechanisms:

• Sample Splitting: When the selection is performed on data independent of the inferential sample, the leakage term vanishes, recovering exact fixed-target coverage.
• Noisy or Compressed Screening: Introducing noise or restricting the summary observed by the screening rule bounds the information leakage. Specifically, if the selector observes only $W = T(D) + \xi$ with $\xi \sim N(0, \tau^2 I_q)$, the additional coverage loss is quantitatively bounded by the log-determinant of the signal-to-noise ratio of $\alpha$0:

$\alpha$1

If a finite or compressed summary $\alpha$2 is used, the loss can be bounded in terms of the entropy of $\alpha$3.

• Reuse of Inferential Data Without Separation: When the same inferential data are reused with arbitrary screening, the bound is vacuous, and the coverage loss can be substantial.

In all cases, explicit control of information flow between the screening and inference stages governs the validity of post-selection confidence statements.

Empirical Illustration

A simulation experiment sampling $\alpha$4, $\alpha$5 i.i.d. standard normal vectors demonstrates the empirical behavior of the leakage bound. When the same sample is used for both coordinate selection (selecting the maximum empirical mean) and inference, the coverage of a nominal $\alpha$6 interval collapses to approximately $\alpha$7. Pure sample splitting restores coverage to the nominal rate. Introducing Gaussian noise in the screening achieves intermediate trade-offs, with coverage rapidly recovering as the additive noise becomes large relative to the standard error. These observations qualitatively validate the theoretical leakage bound, illustrating the post-selection effect and the efficacy of noise- or compression-induced decoupling.

Limitations and Directions for Future Research

The total variation and mutual information bounds are deliberately agnostic to the structure of the selection or inference procedures and may be conservative for specific settings (e.g., where the inferential target or confidence set has additional structure). More refined dependence measures (max-information, maximal leakage, differential privacy) may sharpen the quantitative bounds. Adaptation to specific confidence procedures (likelihood-based, conformal, etc.) or to settings with high-dimensional nuisance estimation remains unaddressed.

Future work could devise analysis protocols that adaptively select the magnitude of randomization, compression, or splitting to control leakage below a specified tolerance, and to investigate leakage quantification in stratified, high-dimensional, or nonparametric scenarios.

Conclusion

This work establishes a general framework for quantifying selection-induced inferential risk in post-selection inference where the selection mechanism is opaque or non-analytical. The finite-sample leakage bound identifies the precise inferential cost of selection via the dependency it engenders between the selected object and the inferential data, without recourse to selection event specification. The results articulate operational strategies for practitioners to mitigate overfitting and coverage decay via data-splitting, noise addition, or compression in the selection process. This reframing broadens the scope of selective inference theory to encompass fully black-box and adaptive statistical workflows, yielding a principled foundation for robust inferential guarantees in modern data analysis pipelines.