Multi-Model Online Conformal Prediction

• Multi-Model Online Conformal Prediction is an adaptive ensemble framework that constructs prediction sets for sequential data while ensuring a user-specified marginal coverage.
• It utilizes graph-structured subset selection and online weight updates to reduce computational cost and prediction set size compared to naive multi-model approaches.
• The framework demonstrates robust empirical performance with sublinear regret and improved efficiency under distribution shifts on benchmark datasets.

A multi-model online conformal prediction algorithm is an adaptive framework for uncertainty quantification that leverages an ensemble of pre-trained prediction models to construct prediction sets for sequentially arriving data. The procedure aims to guarantee marginal coverage (i.e., the frequency with which the true label appears in the prediction set is at least $1-\alpha$ for a user-specified $\alpha$), while also minimizing the size of the prediction sets and operational overhead. Recent developments in this area address critical challenges that arise with large candidate model pools, including computational complexity and the inefficiency induced by poorly performing models. Notably, graph-structured mechanisms have been introduced to enable scalable selection of efficient model subsets at each round, achieving valid coverage guarantees, sublinear regret, and significantly improved efficiency compared to classical multi-model conformal prediction approaches (Hajihashemi et al., 4 Jan 2026, Hajihashemi et al., 26 Jun 2025).

1. Problem Formulation and Core Notation

The online multi-model conformal prediction setting considers data arriving sequentially as pairs $(x_t, y_t)$ for $t=1, \ldots, T$, where $x_t\in\mathcal{X}$ is an input and $y_t\in\mathcal{Y}$ is the label. At each round $t$:

• The algorithm observes $x_t$ and must form a prediction set $C_t(x_t)\subseteq\mathcal{Y}$ before $y_t$ is revealed.
• The true label $y_t$ is observed, and the prediction set is evaluated for coverage and efficiency.

A pool of $M$ pre-trained models $\{M_1,\dots, M_M\}$ is available. For each model $m$, a nonconformity function $S^m(x, y)$ assigns a score representing the degree to which $y$ is atypical for $x$ under model $m$. Each model also maintains a time-varying miscoverage parameter $\alpha_t^m$.

The coverage guarantee sought is: $$\frac{1}{T}\sum_{t=1}^T 1\{y_t\in C_t(x_t)\}\geq 1-\alpha\,.$$

The set size $|C_t(x_t)|$ serves as a direct measure of prediction efficiency.

2. Challenges and Naive Multi-Model Approaches

The naïve Multi-Model Online Conformal Prediction (MOCP) approach computes conformal sets for all $M$ candidate models at each round. Model selection can then be performed via weighted sampling or exponential-weights based on each model's historical prediction efficiency or coverage. However, the complexity per round is $O(M\cdot \text{cost}_\text{quantile})$. As $M$ grows, the cost of computing and maintaining all quantiles, as well as the combinatorial inefficiency introduced by suboptimal models (which can result in much larger prediction sets), becomes prohibitive (Hajihashemi et al., 4 Jan 2026, Hajihashemi et al., 26 Jun 2025). Empirical studies demonstrate that this inefficiency is not merely a computational artifact but is associated with tangible increases in set sizes and wall-clock time (Hajihashemi et al., 26 Jun 2025).

3. Graph-Structured Model Subset Selection

Recent developments have introduced graph-based mechanisms to select effective subsets of models, reducing computational and statistical inefficiency:

Bipartite Feedback Graph

A bipartite graph $G_t=(V_\ell\cup V_s, E_t)$ is maintained where:

• Left nodes ($V_\ell$): each corresponds to a model $M_m$; each is assigned a weight $w_t^m>0$ updated based on past loss.
• Right nodes ($V_s$): "selective nodes" (cardinality $J$); each represents a possible candidate subset formed by stochastic sampling.

Edge construction:

• For each $m=1,\ldots, M$, a sampling probability $p_t^m$ is defined as a convex combination of the normalized model weight and a fixed exploratory term: $$p_t^m = (1-\eta_e)\frac{w_t^m}{\sum_{i=1}^M w_t^i}+\frac{\eta_e}{M}$$.
• For each selective node $j=1,\ldots, J$, $N$ independent samples from $\{1,\ldots,M\}$ are drawn according to $p_t^m$. A model $m$ is included in $j$’s subset if selected at least once: $A_t(j,m)=1$.

Subset selection algorithm entails:

• Compute the sum of weights for all models covered by each selective node.
• Select a selective node proportionally to this sum.
• Use the selected node's model subset $S_t$ for downstream prediction and weight updating.

This approach ensures the computational complexity per round is $O(JN)$ (in contrast to $O(M)$ with full-candidate scans), with $J,N\ll M$ yielding substantial efficiency gains (Hajihashemi et al., 4 Jan 2026, Hajihashemi et al., 26 Jun 2025).

4. Prediction Set Construction and Online Updates

Once the subset $S_t$ is determined, a single model $\hat{m}\in S_t$ is sampled according to normalized weights. Its conformal set is computed as: $$C_t(x_t) = \left\{ y \in \mathcal{Y} : S^{\hat{m}}(x_t, y) \leq \hat{q}_{\alpha_t^{\hat{m}}}^{\hat{m}}\right\},$$ where $\hat{q}_{\alpha_t^m}^m$ is the empirical quantile at level $1-\alpha_t^m$ of past nonconformity scores for model $m$: $$\hat{q}_{\alpha_t^m}^m = \mathrm{Quantile}\left(\frac{\lceil t(1-\alpha_t^m)\rceil}{t-1}, \left\{S^m(x_\tau, y_\tau)\right\}_{\tau=1}^{t-1}\right).$$

Model weights and $\alpha_t^m$ are updated via scale-free online gradient descent (OGD) on the pinball loss, and exponential-weights updates based on loss feedback: $$\alpha_{t+1}^m = \alpha_t^m - \eta \frac{\nabla_{\alpha_t^m}L(\bar{\alpha}_t^m, \alpha_t^m)}{\sqrt{\sum_{\tau=1}^t \|\nabla_{\alpha_\tau^m}L\|^2}}; \quad w_{t+1}^m = w_t^m \exp(-\epsilon L(\bar{\alpha}_t^m, \alpha_t^m)).$$ This yields robust empirical coverage control and optimal long-run regret properties (Hajihashemi et al., 4 Jan 2026).

5. Theoretical Guarantees

Graph-structured multi-model online conformal prediction algorithms exhibit the following guarantees:

• Coverage: For target miscoverage $\alpha$, over the time horizon $T$, the expected coverage converges to $1-\alpha$ with small error:

$$\Bigl|\frac1T\sum_{t=1}^T P\{y_t\notin C_t\}-\alpha \Bigr| = O\bigl(T^{-1/4} \log T\bigr) \rightarrow 0.$$

• Set Size Efficiency: The average width is bounded (under mild distributional assumptions on the scores) above the minimal achievable by any single model, plus an vanishingly small term,

$$\mathbb{E}\left[|C_t(x_t)|\right] \leq \min_m \mathbb{E}\left[|C^m_t(x_t)|\right] + O\left(\sqrt{\frac{\log M}{T}}\right).$$

• Sublinear Regret: Cumulative pinball regret relative to the best fixed model is sublinear: $O(\sqrt{T})$. These results apply under both adversarial and stationary (exchangeable/i.i.d.) regimes, and are validated empirically (Hajihashemi et al., 4 Jan 2026, Hajihashemi et al., 26 Jun 2025).

6. Empirical Performance and Comparative Analysis

Quantitative experiments validate that graph-structured algorithms (such as GMOCP and its size-aware variant EGMOCP) deliver valid coverage and consistently reduced set sizes and runtimes. For instance, on CIFAR-100C under abrupt distribution shifts,

• Standard MOCP: coverage ≈89.7%, average width ≈12.6, runtime ≈14 s;
• GMOCP ($J=2, N=3$): coverage ≈89.5%, average width ≈10.9 (–14%), runtime ≈11.5 s (–18%);
• EGMOCP (size feedback): coverage ≈89.4%, average width ≈6.3 (–43%), runtime ≈15.6 s.

Similar reductions are observed across TinyImageNet-C and other synthetic distribution-shift benchmarks. Across all datasets, strong empirical coverage and favorable singleton coverage fractions are reported (Hajihashemi et al., 26 Jun 2025, Hajihashemi et al., 4 Jan 2026).

Algorithm	Coverage (%)	Avg Width	Runtime (s)
MOCP	89.7	12.6	14
GMOCP $J=2,N=3$	89.5	10.9	11.5
EGMOCP	89.4	6.3	15.6

This tabulation highlights substantial improvements in efficiency.

7. Extensions, Limitations, and Open Problems

Key limitations include the dependence on graph parameters $(J,N)$, which trade off between exploration and computational cost; suboptimal selection can either degrade efficiency or negate computational gains. Algorithmic performance also relies on the careful tuning of weight-update hyperparameters $(\eta_e, \epsilon)$. Theoretical bounds on expected width are not always explicit.

Prospective directions include:

• Adaptive control of graph parameters $(J,N)$ based on empirical regret or set width.
• Incorporation of calibration-point nodes rather than selective nodes in the graph for tighter filtering.
• Extension to regression and structured-output prediction tasks.
• Development of tighter bounds on the trade-off between width-regret and high-probability coverage, and integration with data-dependent conformal score learning.

These advances set a template for scalable and principled uncertainty quantification under distributional shift, informing the design of state-of-the-art ensemble conformal predictors (Hajihashemi et al., 4 Jan 2026, Hajihashemi et al., 26 Jun 2025).