False Coverage Rate Control

• False Coverage Rate (FCR) is a metric that measures the proportion of confidence or prediction intervals that fail to include target parameters among those selected via data-driven procedures.
• It extends the false discovery rate concept to post-selection interval estimation, addressing multiplicity and bias in areas like regression and high-dimensional statistics.
• Various procedures, including BY adjustments, LORD–CI, and CAP, are used to control FCR in both offline and online settings to ensure valid and reliable inference.

The false coverage rate (FCR) is a fundamental metric in selective inference, characterizing the expected proportion of reported confidence or prediction intervals that fail to cover their target parameters, taken over only those intervals that are actually constructed and reported after a data-driven selection procedure. FCR extends the false discovery rate (FDR) concept from multiple hypothesis testing to the post-selection interval estimation setting, directly addressing multiplicity and selection bias that arise when analysts report results based on data-dependent filtering of features, samples, or prediction sets. Rigorous FCR control is essential for reliable post-selection reporting in various domains, including regression, classification, high-dimensional statistics, and online inference.

1. Mathematical Formulation and Definitions

Let $S$ denote a selection set of indices determined by an analyst’s rule applied to available data; the size of $S$ is $|S|$. For each $i\in S$, a confidence or prediction interval (or set) $C_i$ is constructed for target $Y_i$. The false coverage proportion (FCP) is defined as

$$\mathrm{FCP} = \frac{\#\{i \in S: Y_i \notin C_i\}}{\max\{|S|,1\}}.$$

The false coverage rate is the expectation of FCP under the data-generating model,

$\mathrm{FCR} = \mathbb{E}[\mathrm{FCP}].$

This generalizes to both fixed-sample (offline) and online (sequential) settings, as well as to the construction of confidence intervals, prediction intervals, and more general selective inference objects (Gazin et al., 2024, Weinstein et al., 2014, Bao et al., 2023, Bao et al., 2024, Weinstein et al., 2019).

2. FCR vs. FDR and Conditional Coverage

FCR occupies a distinct role relative to FDR and ordinary marginal or conditional coverage:

• FDR is the expected proportion of false rejections among reported discoveries in hypothesis testing,

$$\mathrm{FDR} = \mathbb{E}\left[\frac{\#\{\text{false rejections}\}}{\max\{\#\{\text{rejections}\},1\}}\right].$$

• FCR similarly quantifies error rate for intervals; it is the expected proportion of non-covering intervals among those constructed. Unlike classical marginal coverage, which controls $P(Y_i \in C_i) \ge 1-\alpha$ for each $S$0, FCR explicitly targets the post-selection consequence that non-covering intervals may be over-represented among the reported set due to data-driven selection (Weinstein et al., 2014, Bao et al., 2024).
• Conditional coverage conditions on the selection event, seeking $S$1, but is not always achievable in finite samples or under selection dependence (Bao et al., 2023, Bao et al., 2024).

3. Frameworks and Procedures for FCR Control

Offline Selective Setting

Benjamini–Yekutieli (BY) FCR Adjustment:

For independent estimates $S$2 of parameters, the BY procedure adjusts the nominal level of confidence intervals based on the selection rule. If $S$3 is the set of selected indices, construct intervals at level $S$4, where $S$5 and $S$6 is the desired FCR threshold. This ensures global FCR$S$7 for any monotone and symmetric marginal confidence interval procedure (Weinstein et al., 2014).

Informative Selective Prediction (InfoSP) and InfoSCOP:

In the conformal prediction context, InfoSP and InfoSCOP provide the first finite-sample FCR-controlling procedures for post-selection informative prediction sets. Both enforce constraints $S$8 (e.g., interval length restrictions, exclusion of null sets/classes) and output only those conformal sets satisfying these informativeness conditions, adjusting the conformal nominal level via a BH-like step-up and threshold inflation $S$9 (Gazin et al., 2024).

Online and Adaptive Settings

LORD–CI Procedure:

In online inference with potentially infinite sequences, LORD–CI adapts the allocation of type-I error across intervals via a predictable, monotone updating rule ensuring for all $|S|$0:

$|S|$1

where $|S|$2 is the nominal level of the $|S|$3th interval. Under monotone selection, this ensures FCR$|S|$4 for all $|S|$5 (Weinstein et al., 2019).

CAP and Dynamic CAP (DtACI):

In the online conformal prediction setting, CAP (Calibration after Adaptive Pick) controls the real-time aggregate FCR by selecting adaptive calibration sets—via decision-driven or symmetric-threshold rules—and using residual quantiles among those calibration points to construct intervals for selected test points. CAP is extended to shifting distributions in DtACI, ensuring long-run FCR control under non-exchangeable regimes (Bao et al., 2024).

Conditional Conformal Prediction

SCOP:

SCOP (Selective Conditional conformal Prediction) mirror the selection decision onto the calibration set and constructs the conformal interval only for calibration and test indices selected under the same rule, achieving exact or nearly exact FCR control (under exchangeable or calibration-assisted selection) and typically yielding intervals less inflated than BY-adjusted marginal intervals (Bao et al., 2023).

4. Theoretical Guarantees

The described FCR procedures attain their guarantees under minimal assumptions:

• InfoSP / InfoSCOP: Under exchangeability (regression and classification) and monotonicity conditions, reported sets are always informative and FCR$|S|$6 (Gazin et al., 2024).
• BY-Adjusted CIs: Under independence and monotonicity, global $|S|$7 is ensured (Weinstein et al., 2014).
• LORD–CI: mFCR control for arbitrary predictable selection; FCR control under monotonic selection (Weinstein et al., 2019).
• CAP/SCOP: Exact FCR in finite samples under strong exchangeability; non-asymptotic bounds and anti-conservative lower bounds computed under more general designs (Bao et al., 2023, Bao et al., 2024).

Central proof techniques involve super-uniformity of conformal p-values, monotone selection cardinality, marginal versus conditional error rate invariance, and technical lemmas handling dependence between selection and interval construction.

5. Applications and Practical Considerations

FCR-controlling post-selection procedures are vital in diverse high-dimensional and selective reporting contexts, such as:

• Selective Sign Inference: Construction of sign-determining intervals in genomics and neuroscience, where balanced power and interval length is achieved by MQC marginal intervals under BY scheme (Weinstein et al., 2014).
• Selective Conformal Prediction: Informativeness constraints (e.g., excluding null regions or labels, limiting set widths) implemented in classification and regression, tested on synthetic, imaging, and experimental biology datasets (Gazin et al., 2024).
• Online Prediction: Drug selection, anomaly detection, financial volatility; CAP/SCOP methods validated on real and synthetic data, outperforming naïve conformal or overly conservative marginal adjustments in empirical FCR and interval tightness (Bao et al., 2024, Weinstein et al., 2019).

Tables, examples, and algorithmic pseudocode are available in the respective papers for practical reference; all procedures are executable with modest computational cost and allow flexible integration of user-defined informative constraints.

6. Contemporary Directions and Extensions

Recent efforts address key limitations and generalizations:

• Distribution Shifts: DtACI extends FCR control under non-exchangeable, dynamically shifting environments by adaptively tuning nominal levels based on monitored error (Bao et al., 2024).
• Generalized Selection Rules: Future research aims to weaken monotonicity or symmetry assumptions while retaining FCR control, targeting adaptive selection in broader application domains (Bao et al., 2024).
• Multi-dimensional and Composite Targets: Extensions to multi-dimensional parameters, selective localization, and composite hypothesis settings have been formalized, with FCR extended as the false-localization rate (FLR) and related metrics (Weinstein et al., 2014, Weinstein et al., 2019).

Researchers are encouraged to consult cited articles for the mathematical underpinnings, implementation intricacies, and simulation validations that support FCR as the rigorous standard for post-selection interval error control.