Online Selective Conformal Prediction

• Online selective conformal prediction is a rigorous framework that constructs prediction sets for sequential data using adaptive selection rules.
• It employs advanced methods like PEMI and CAP to ensure finite-sample, selection-conditional coverage and robust false coverage rate control under asymmetric data dependencies.
• Recent frameworks leverage permutation-based, adaptive, and gradient-based algorithms to maintain coverage accuracy and efficiently adjust to nonstationary or adversarial data environments.

Online selective conformal prediction (OSCP) is a mathematically rigorous framework for constructing prediction sets or intervals for sequentially arriving data, where predictions are only issued adaptively at user-specified or algorithmically determined time points. The primary challenge is to guarantee exact or targeted coverage conditional on having selected a point for inference, despite the resulting asymmetry and dependencies in the selection process that invalidate classical conformal exchangeability. Recent advances provide practical, finite-sample, and often distribution-free methodologies for constructing and calibrating such selective conformal prediction sets in the online regime.

1. Formal Problem Definition

In the canonical OSCP setup, we observe a data stream $(Z_1, ..., Z_{t-1})$ with $Z_i = (X_i, Y_i)$ and, at each time $t$, a new feature $X_t$ arrives. A selection mechanism $$\mathcal{S}_t : ((\mathcal{X} \times \mathcal{Y})^{t-1} \times \mathcal{X}) \to \{0,1\}$$ outputs $S_t = \mathcal{S}_t(Z_1, ..., Z_{t-1}, X_t)$, indicating whether to issue a prediction set for $Y_t$.

The key objective is to construct a (possibly random) prediction set $\widehat{C}_{\alpha, t}\subseteq\mathcal{Y}$ such that, under exchangeability,

$$\mathbb{P}[Y_t\in\widehat{C}_{\alpha, t}\mid S_t=1]\geq 1-\alpha$$

for target miscoverage rate $\alpha\in(0,1)$. This coverage must hold conditional on selection—that is, only at those time points $Z_i = (X_i, Y_i)$0 where $Z_i = (X_i, Y_i)$1.

Additional desiderata include finite-sample false coverage rate (FCR) control, defined as the (conditional) expectation of the proportion of non-covering prediction sets among all selected points up to time $Z_i = (X_i, Y_i)$2:

$Z_i = (X_i, Y_i)$3

2. Exchangeability, Calibration, and Selection Asymmetry

Canonical conformal prediction depends on exchangeability or (approximate) symmetry among the calibration and test samples. In the OSCP setting, the presence of a selection rule that adapts to history introduces asymmetric selection—the event $Z_i = (X_i, Y_i)$4 can break the symmetry between the selected test point and the calibration data, invalidating naive calibration procedures and coverage claims.

Fundamental errors arise when attempting to use calibration pools constructed via heuristics such as filtering past data by the selection rule (S-FULL, ADA, FULL), which fail to restore the requisite exchangeability (Sale et al., 21 Mar 2025). Without careful calibration set design, such approaches do not achieve selection-conditional coverage or FCR control. Precise symmetry between the selected test datum and the calibration pool must be actively restored.

3. Frameworks and Algorithms for Online Selective Conformal Prediction

Recent work has produced a rich class of OSCP frameworks addressing asymmetric and adaptive selection:

3.1. PErmutation-based Mondrian Conformal Inference (PEMI)

The PEMI method (Zheng et al., 10 Feb 2026) introduces a general permutation-based reference set for calibration. For each candidate $Z_i = (X_i, Y_i)$5, it identifies all (or a sampled subset of) permutations $Z_i = (X_i, Y_i)$6 of $Z_i = (X_i, Y_i)$7 that would result in selection under the observed rule, forming the reference set $Z_i = (X_i, Y_i)$8. It then computes a p-value

$Z_i = (X_i, Y_i)$9

and inverts this for the conformal set $t$0. Under exchangeability, PEMI yields finite-sample exact selection-conditional coverage for any selection rule (including fully history-dependent, asymmetric rules), handles offline data extensions, multiple test points, and achieves tight FCR control by conditioning on the trajectory of selections.

3.2. Adaptive Calibration Protocols: CAP and EXPRESS

The CAP ("Calibration after Adaptive Pick") algorithm (Bao et al., 2024) restores symmetry by adaptively selecting a calibration set from historical data according to a data-driven "adaptive pick" rule that ensures exchangeability between the selected test and its calibration points. For symmetric or decision-driven selection rules, explicit adaptive pick constructions yield tractable calibration sets and enable application of split-conformal quantiles to form prediction intervals. CAP provides real-time FCR control and can be extended for covariate shift or nonstationary settings via integration with dynamically-tuned conformal inference.

EXPRESS and K-EXPRESS (Sale et al., 21 Mar 2025) select only past points that have undergone the exact same sequence of selections as the test point (over all or last $t$1 time steps), guaranteeing full symmetry between selected test and calibration points, correcting errors in earlier literature.

3.3. Adaptive Online Algorithms and Regret

OnlineSCI (Humbert et al., 14 Aug 2025) and related gradient-based or bandit-style online algorithms tackle the convergence and adaptation properties of OSCP under arbitrary or adversarial sequences, including coverage tradeoffs, adjustment to drifting data, and the rate at which instantaneous or FCR error rates converge to their targets under adaptive point predictor updates.

3.4. Model and Rule Selection

Stability-based selectors (e.g., via differential privacy concepts and MinSE linear programming) (Hegazy et al., 25 Jun 2025) allow online selection among multiple conformal predictors or models with guaranteed (slightly inflated) post-selection coverage in adversarial or distribution-shifted settings.

4. Theoretical Guarantees

The current state-of-the-art methodologies provide a spectrum of exact and approximate guarantees:

• Selection-Conditional Coverage: For PEMI, CAP, and EXPRESS variants, under data exchangeability and correct calibration set construction, finite-sample conditional coverage $t$2 holds, for arbitrarily complex selection rules (Zheng et al., 10 Feb 2026, Bao et al., 2024, Sale et al., 21 Mar 2025).
• FCR Control: Strong FCR control is achieved using PEMI and adaptive pick strategies (e.g., CAP, K-EXPRESS), with exact or sharp finite-sample bounds, both in i.i.d. and certain covariance-shift settings (Zheng et al., 10 Feb 2026, Bao et al., 2024).
• Robustness to Distribution Shift: Dynamic tuning and incorporation of adaptive conformal inference (e.g., CAP-DtACI) maintain long-run FCR control in nonstationary or adversarial environments (Bao et al., 2024, Humbert et al., 14 Aug 2025).
• Online Regret Analysis: For gradient-based or expert-advice style algorithms, sublinear regret and convergence rates to the oracle threshold are proven under mild conditions (Humbert et al., 14 Aug 2025, Ge et al., 2024).
• Validity under Model Selection: Stability-based selection among multiple online predictors guarantees coverage with at most a multiplicative ($t$3) and additive ($t$4) inflation, even under arbitrarily chosen selectors (Hegazy et al., 25 Jun 2025).

5. Efficient Instantiations and Practical Implementation

Implementation efficiency depends on the complexity of the selection rule and the calibration set construction:

• Covariate-based selection (selection depends only on features): The reference set $t$5 becomes independent of $t$6, and the PEMI p-value reduces to a permutation quantile test over residuals.
• Selection rules driven by conformal $t$7-values or $t$8-values: PEMI provides explicit two-regime calibration pools partitioned by the threshold, allowing efficient set construction.
• Selection by earlier outcomes: Partitioning $t$9 into intervals using order statistics of the model predictions yields reference sets constant within intervals, leading to fast inversion and computation (Zheng et al., 10 Feb 2026).
• EXPRESS/K-EXPRESS/EXPRESS-M: By restricting the calibration pool to entries matching on recent or all past selections, these methods trade off coverage accuracy and the occurrence of infinite/vacuous prediction sets (Sale et al., 21 Mar 2025).
• CAP/OnlineSCI: Adaptive or gradient-based calibration with user-specified step size sequences control the rate of threshold updating for efficient convergence (Bao et al., 2024, Humbert et al., 14 Aug 2025).
• Stochastic feedback settings: Semi-bandit variants adapt to partial information by exploiting monotone thresholding and Dvoretzky–Kiefer–Wolfowitz uncertainty bands (Ge et al., 2024).

Computational requirements for modern OSCP methods are dominated by permutation sampling (in PEMI, $X_t$0 with $X_t$1 typically sufficient), or by adaptive calibration set construction and threshold updates for CAP/OnlineSCI ($X_t$2 to $X_t$3 per step).

6. Empirical Performance and Evaluation

Comprehensive empirical studies establish the utility of rigorous OSCP frameworks:

• Drug discovery (DAVIS dataset): PEMI, CAP, and EXPRESS variants reach target selection-conditional coverage, maintain moderate prediction set sizes, and minimize the rate of infinite or vacuous sets under various decision- and data-driven selection rules (Zheng et al., 10 Feb 2026, Bao et al., 2024, Sale et al., 21 Mar 2025).
• Stock volatility and regression benchmarks: CAP-DtACI and related adaptive methods recover long-run FCR close to the nominal $X_t$4 under distributional drift, yielding shorter prediction intervals than conservative baselines such as LORD-CI (Bao et al., 2024).
• Categorical/Active Learning and Document Retrieval: Stochastic online OSCP variants outperform baseline methods in partial feedback settings, maintaining regret lower bounds and global coverage rates on large multiclass and information retrieval tasks (Ge et al., 2024).
• Practical trade-offs: Methods not restoring symmetry (vanilla conformal, S-FULL, ADA) under-cover or produce unwieldy intervals, while symmetry-preserving methods achieve the desired guarantees but may yield longer or vacuous intervals in complex selection regimes (Sale et al., 21 Mar 2025).

A summary of empirical outcomes across the literature:

Method	Coverage Control	FCR Control	Prediction Set Size	Computational Demand
PEMI	Finite-sample exact	Yes	Moderate	$X_t$5
CAP/K-EXPRESS/EXPRESS	Finite-sample exact	Yes	Shorter/Fewer infinite	Variable
OnlineSCI	Asymptotic/gradual	Asymptotic	Adaptive	$X_t$6
Non-symmetric (FULL)	Fails	Fails	Shorter (illusory)	$X_t$7

7. Extensions and Ongoing Developments

OSCP remains an area of rapid development, with several prominent directions:

• Multiple model and rule selection: Adaptive and stable algorithms allow pointwise or data-dependent selection of conformal predictors with robust coverage-inflation bounds (Hegazy et al., 25 Jun 2025).
• Dynamic/adaptive calibration: OnlineSCI and dynamic tuning enable OSCP to operate in non-i.i.d., covariate-shifted, and adversarial scenarios with rigorous convergence rates to optimal thresholds (Humbert et al., 14 Aug 2025, Bao et al., 2024).
• Semi-bandit and partial feedback: Effective procedures for cases where labels are only revealed for confident predictions, facilitating reliable uncertainty quantification in computation- or label-budgeted deployments (Ge et al., 2024).
• Computational scaling: As OSCP is applied to large-scale and real-time environments, scalable permutation sampling, calibration set management, and stochastic optimization are increasingly prioritized.

Theoretical advances continue to clarify the limits of exact selection-conditional coverage, FCR control, and robust model selection under complex online, adaptive, or dependent data-generating scenarios. Leading methodologies, including PEMI, CAP, and their adaptive variants, form the backbone of current practice in OSCP, offering finite-sample, distribution-free guarantees across an array of selection and application regimes (Zheng et al., 10 Feb 2026, Bao et al., 2024, Sale et al., 21 Mar 2025, Humbert et al., 14 Aug 2025, Hegazy et al., 25 Jun 2025).