Beyond Fixed False Discovery Rates: Post-Hoc Conformal Selection with E-Variables

Abstract: Conformal selection (CS) uses calibration data to identify test inputs whose unobserved outcomes are likely to satisfy a pre-specified minimal quality requirement, while controlling the false discovery rate (FDR). Existing methods fix the target FDR level before observing data, which prevents the user from adapting the balance between number of selected test inputs and FDR to downstream needs and constraints based on the available data. For example, in genomics or neuroimaging, researchers often inspect the distribution of test statistics, and decide how aggressively to pursue candidates based on observed evidence strength and available follow-up resources. To address this limitation, we introduce {post-hoc CS} (PH-CS), which generates a path of candidate selection sets, each paired with a data-driven false discovery proportion (FDP) estimate. PH-CS lets the user select any operating point on this path by maximizing a user-specified utility, arbitrarily balancing selection size and FDR. Building on conformal e-variables and the e-Benjamini-Hochberg (e-BH) procedure, PH-CS is proved to provide a finite-sample post-hoc reliability guarantee whereby the ratio between estimated FDP level and true FDP is, on average, upper bounded by $1$, so that the average estimated FDP is, to first order, a valid upper bound on the true FDR. PH-CS is extended to control quality defined in terms of a general risk. Experiments on synthetic and real-world datasets demonstrate that, unlike CS, PH-CS can consistently satisfy user-imposed utility constraints while producing reliable FDP estimates and maintaining competitive FDR control.

• The paper presents a novel post-hoc conformal selection (PH-CS) method that uses e-variables to flexibly adjust test selection after observing data.
• It integrates the e–Benjamini-Hochberg procedure with utility-driven optimization to provide finite-sample guarantees on the false discovery rate.
• Empirical evaluations demonstrate that PH-CS reliably meets user-defined constraints and adapts to varied downstream requirements in high-stakes applications.

Post-Hoc Conformal Selection with E-Variables: A Utility-Driven Framework for FDR Control

Introduction and Motivation

Conformal selection (CS) provides finite-sample, distribution-free guarantees for controlling the false discovery rate (FDR) when selecting test points expected to meet user-defined quality constraints. However, CS requires the FDR threshold to be fixed prior to seeing the test or calibration data, restricting users to operate at a single, inflexible reliability level. This inflexibility limits adaptive decision-making in high-stakes domains (e.g., drug discovery, genomics) where the optimal trade-off between selection set size and reliability is often clear only after observing the actual data and considering real-world constraints.

"Beyond Fixed False Discovery Rates: Post-Hoc Conformal Selection with E-Variables" (2604.11305) introduces a utility-driven framework—post-hoc conformal selection (PH-CS)—that enables researchers to adaptively choose the selection set after data observation. PH-CS combines conformal e-variables with the e–Benjamini-Hochberg (e-BH) procedure, generating a path of candidate selection sets with associated false discovery proportion (FDP) estimates. Crucially, it provides finite-sample post-hoc reliability guarantees, permitting selection at any point along this path based on a user-specified utility function expressing the desired trade-off between set size and reliability.

Figure 1: Illustration of the PH-CS framework, including access to calibration/test data, candidate selection sets along a post-hoc path, associated FDP estimates, and optimization based on a user-specified utility.

Problem Formulation and Limitations of Conventional CS

Classical CS maps user constraints into a hard upper bound on FDR, using conformal p-variables and the Benjamini-Hochberg procedure. It implements the FDR constraint

$\text{FDR}(R) \leq \alpha_{\text{max}}$

for a fixed level $\alpha_{\text{max}}$. The targeted FDR must be chosen before observing the data, even though the actual trade-off between selection size and reliability might require post hoc adjustments due to empirical test statistics, enrichment patterns, budget, or other downstream conditions that become apparent only after data observation.

Figure 2: Histograms demonstrating selection set size and realized FDP for conventional CS, which often fails to meet size/reliability constraints on real or synthetic data.

Post-Hoc Conformal Selection (PH-CS) Framework

PH-CS relaxes the rigidity of classical CS by enabling arbitrary, data-driven selection set decisions after observing calibration and test data, while still certifying statistical validity. The framework is built on several key components:

1. Conformal e-variables: For each test point, a conformal e-variable is computed, quantifying evidence against a "random" null hypothesis. E-variables satisfy a mean constraint analogous to supermartingales, enabling level-uniform guarantees across all possible FDR thresholds.
2. e-BH candidate path: The e-BH procedure is used to generate a path of nested candidate selection sets, indexed by cutoff thresholds on the sorted e-variables.
3. Utility-driven selection: A user-specified utility function $U(|\mathcal{R}|, \hat{\alpha}(\mathcal{R}))$ quantifies the relative value of selection set size and FDP estimate. PH-CS outputs the candidate set maximizing this utility.

Formally, the method solves

$$\max_{\mathcal{R}} U\left(|\mathcal{R}|, \hat{\alpha}(\mathcal{R})\right)$$

where $\hat{\alpha}(\mathcal{R})$ is a data-driven conservative estimate of the FDP for candidate set $\mathcal{R}$.

Figure 3: Selection set size and realized FDP histograms under constrained-size utility with PH-CS, showing consistent satisfaction of user-imposed constraints.

The crucial theoretical contribution is a reliability guarantee: for any utility function and any data-dependent choice of operating point along the e-BH path, the FDP estimate $\alpha^\text{PH-CS}$ satisfies

$$\mathbb{E}\left[\frac{\text{FDP}}{\alpha^\text{PH-CS}}\right] \leq 1,$$

implying that $\mathbb{E}[\alpha^\text{PH-CS}]$ is an (approximate) upper bound for the true FDR. The method retains validity under minimal assumptions (exchangeability, monotone scores), supports continuous or binary requirements, and allows for user- or application-specific utilities.

Extensions: Risk-Controlled and Weighted Selection

PH-CS generalizes to settings with continuous loss metrics (risk-controlled selection, PH-RCS) and non-uniform candidate prioritization (weighted selection):

• For continuous losses $\mathcal{L}(X, Y)\in[0,1]$, selection quality is measured by expected per-candidate risk, and risk-adjusted conformal e-variables are used.
• Arbitrary non-negative priority weights $\alpha_{\text{max}}$0 can be assigned to test points (with the requirement $\alpha_{\text{max}}$1), and the framework's guarantee persists, providing a principled means for power-boosting or focusing discovery efforts.

Empirical Evaluation

Extensive experiments on synthetic and real datasets (e.g., Recruitment, Musk, Shuttle) show that PH-CS robustly satisfies user-defined utility or size constraints on every realization—a capacity unattainable with conventional CS, which frequently violates size constraints or misestimates realized FDP. PH-CS maintains competitive FDR, and its declared FDP estimates closely track the actual rate of false discoveries.

Figure 4: Realized utility distribution on synthetic data under the additive trade-off utility, evidencing higher utility on average for PH-CS versus classical CS.

Figure 5: Utility, selection size, and FDP histograms on real data with aggressive and conservative utility parameterizations, showing PH-CS trade-off flexibility and reliability.

Figure 6: Scatter plots reveal close agreement between estimated and realized FDP for PH-CS across random seeds and datasets.

These empirical results confirm that PH-CS guarantees post-hoc control, adapts to downstream needs, and delivers accurate FDP estimates.

Practical and Theoretical Implications

PH-CS addresses a central limitation of FDR-controlling selection: the inability to adapt selection stringency after observing data. By decoupling FDR estimation from a priori threshold commitments, it enables practitioners to adjust to operational realities, retrospective priorities, or utility-optimal operating points while retaining finite-sample validity. This flexibility is possible only via the use of e-variables and the e-BH procedure; conventional p-value-based techniques do not support post-hoc inference without additional assumptions or conservatism.

In practice, PH-CS streamlines utility-driven selection in high-throughput scientific discovery, high-dimensional feature screening, and any scenario involving complex post-selection or operational constraints.

On a theoretical level, PH-CS leverages the level-uniformity of e-variables for simultaneous valid error control over a continuum of potential operating points, representing a significant conceptual advance over p-variable-based regimes.

Future Directions

Future work may extend PH-CS to address covariate shift and adaptive data settings, employ more powerful e-variable constructions, and integrate advanced e-BH boosting strategies for enhanced statistical power without sacrificing post-hoc guarantees.

Conclusion

PH-CS inherits the distribution-free rigor of conformal inference and augments it with robust, flexible, and utility-driven post-hoc FDR control. It achieves reliable error guarantees in finite samples, supports continuous/weighted risk, and satisfies arbitrary data-driven utility constraints. This framework opens new avenues for adaptive, application-sensitive inference and selection in scientific and industrial workflows.