Conformal Ensemble Score

• Conformal Ensemble Score is a scalar nonconformity measure constructed from multiple predictive signals to maintain valid marginal coverage in conformal prediction.
• It aggregates scores through methods such as weighted combinations, uncertainty-aware ranking, and posterior-predictive transformations, thereby improving efficiency and stability.
• The approach relies on exchangeability principles and conditional calibration theory, with applications ranging from regression to vision and medical imaging.

A “conformal ensemble score” (Editor’s term) denotes a scalar conformity or nonconformity statistic used in conformal prediction that is constructed from multiple predictive signals rather than from a single raw model score. In current literature, this umbrella covers at least five recurring constructions: weighted combinations of several score functions, symmetric aggregation of model-wise scores after e-value normalization, uncertainty-aware aggregation of multiple stochastic realizations into a single ranking statistic, posterior-predictive transformations of a base score, and feature-dependent rectification of a baseline score (Luo et al., 2024, Alami et al., 7 Dec 2025, Zeng et al., 22 May 2026, Cabezas et al., 10 Feb 2025, Plassier et al., 22 Feb 2025). Across these variants, the common objective is to preserve the core conformal guarantee of marginal coverage while improving efficiency, ranking stability, or approximate conditional validity relative to single-score conformal prediction.

1. Conceptual scope

In standard split conformal prediction, one calibrates a scalar score on a held-out calibration set and thresholds that score to form a prediction set. The conformal ensemble perspective retains this calibration step but replaces the single score by a score produced through aggregation, normalization, or adaptation. The aggregation may occur across multiple models, multiple outputs, multiple stochastic evaluations, or multiple score functions, and it may also be implemented indirectly through score-distribution modeling or covariate-dependent score transformations (Alami et al., 7 Dec 2025, Zeng et al., 22 May 2026, Plassier et al., 22 Feb 2025).

A central misconception is to equate conformal ensemble scoring with simple probability averaging. The literature is broader. Some methods average or weight probabilities only after first converting model-specific outputs into standardized objects such as e-values; some use posterior tail probabilities that depend jointly on score magnitude and uncertainty; some transform a single base score through a learned local map so that its key conditional quantile is approximately constant across covariates; and some switch among models online rather than aggregating them statically (Alami et al., 7 Dec 2025, Zeng et al., 22 May 2026, Plassier et al., 22 Feb 2025, Hajihashemi et al., 2024).

Another important distinction is between marginal and conditional validity. Exact finite-sample marginal coverage is often inherited from standard conformal prediction once the aggregated score is fixed, but conditional coverage is generally unattainable in a fully distribution-free sense. Several conformal ensemble score constructions therefore target either improved approximate conditional coverage or asymptotic conditional coverage rather than exact conditional validity (Plassier et al., 22 Feb 2025, Cabezas et al., 10 Feb 2025).

2. Mathematical constructions

A common pattern is to build a scalar score $S_{\mathrm{ens}}(x,y)$ and then run an ordinary conformal calibration step on $\{S_{\mathrm{ens}}(X_i,Y_i)\}$. This suggests the generic schematic

$$S_{\mathrm{ens}}(x,y)=\phi\big(s_1(x,y),\dots,s_K(x,y);x\big),$$

although the actual literature instantiates $\phi$ in several distinct ways.

Paradigm	Representative construction	Representative paper
Weighted score aggregation	$$\langle w, s(x,y)\rangle = \sum_{j=1}^d w_j s_j(x,y)$$	(Luo et al., 2024)
Symmetric aggregation after e-value normalization	$F_{\text{test}}(y)=f(\mathbf{E}_{\text{test}}(y))$, often $\Phi(\mathbf{x})=\sum_{k=1}^K \phi(x_k)$	(Alami et al., 7 Dec 2025)
Uncertainty-aware aggregation of stochastic scores	$$r(f(x)_i,\sigma_i^2)=\inf\{\beta:f(x)_i\ge t_\beta^*(\sigma_i^2)\}$$	(Zeng et al., 22 May 2026)
Posterior-predictive transformation of a base score	$s'(x,y)=F(s(x,y)\mid x,D)$	(Cabezas et al., 10 Feb 2025)
Feature-dependent rectification of a baseline score	$\tilde V(x,y)=f_{\widehat{\tau}(x)}^{-1}(V(x,y))$	(Plassier et al., 22 Feb 2025)
Out-of-bag quantile ensemble score	$\{S_{\mathrm{ens}}(X_i,Y_i)\}$0	(Gupta et al., 2019)

The weighted aggregation framework in multiclass classification is the most explicit score-level ensemble formulation. There, base scores such as THR, APS, and RANK are assembled into $\{S_{\mathrm{ens}}(X_i,Y_i)\}$1, and the conformal score is the weighted linear combination $\{S_{\mathrm{ens}}(X_i,Y_i)\}$2. Prediction sets are then upper level sets of this aggregated score, with weights selected to minimize average prediction set size (Luo et al., 2024).

SACP takes a different route. For each model $\{S_{\mathrm{ens}}(X_i,Y_i)\}$3, it first converts the per-model nonconformity score into an e-value, then aggregates the resulting e-value vector $\{S_{\mathrm{ens}}(X_i,Y_i)\}$4 through a symmetric function $\{S_{\mathrm{ens}}(X_i,Y_i)\}$5. The default family is $\{S_{\mathrm{ens}}(X_i,Y_i)\}$6, which yields a single ensemble score $\{S_{\mathrm{ens}}(X_i,Y_i)\}$7 subsequently used exactly as a conformal score (Alami et al., 7 Dec 2025).

The $\{S_{\mathrm{ens}}(X_i,Y_i)\}$8-value framework is ensemble-based in a stochastic rather than model-indexed sense. Multiple posterior draws, prompt paraphrases, or evaluator runs generate repeated scores per candidate; the $\{S_{\mathrm{ens}}(X_i,Y_i)\}$9-value then measures how likely a candidate’s latent score belongs to the top-ranked group after accounting for both mean score and uncertainty. That $$S_{\mathrm{ens}}(x,y)=\phi\big(s_1(x,y),\dots,s_K(x,y);x\big),$$0-value becomes the conformal nonconformity score (Zeng et al., 22 May 2026).

EPICSCORE and RCP enlarge the notion further. EPICSCORE maps a base score through the posterior predictive CDF of the score itself, while RCP learns a feature-dependent transformation of a baseline conformity score so that its $$S_{\mathrm{ens}}(x,y)=\phi\big(s_1(x,y),\dots,s_K(x,y);x\big),$$1-quantile is approximately flat across covariates (Cabezas et al., 10 Feb 2025, Plassier et al., 22 Feb 2025).

3. Coverage, validity, and theoretical structure

The fundamental reason conformal ensemble scores are viable is that conformal prediction does not require the score to come from a single model. Once the aggregated score is computed in the same way for calibration and test points, standard exchangeability arguments apply. This is made explicit in several papers. RCP proves finite-sample marginal coverage

$$S_{\mathrm{ens}}(x,y)=\phi\big(s_1(x,y),\dots,s_K(x,y);x\big),$$2

under monotone score transformation and distinct rectified scores (Plassier et al., 22 Feb 2025). The $$S_{\mathrm{ens}}(x,y)=\phi\big(s_1(x,y),\dots,s_K(x,y);x\big),$$3-value method states the same principle in candidate-set form: $$S_{\mathrm{ens}}(x,y)=\phi\big(s_1(x,y),\dots,s_K(x,y);x\big),$$4 so long as the $$S_{\mathrm{ens}}(x,y)=\phi\big(s_1(x,y),\dots,s_K(x,y);x\big),$$5-value is used as the calibration and test score in the same way (Zeng et al., 22 May 2026).

A second theoretical theme is that aggregation may improve conditional behavior even when only marginal coverage is exact. RCP introduces a quantile error function

$$S_{\mathrm{ens}}(x,y)=\phi\big(s_1(x,y),\dots,s_K(x,y);x\big),$$6

and derives conditional coverage bounds whose deviation from $$S_{\mathrm{ens}}(x,y)=\phi\big(s_1(x,y),\dots,s_K(x,y);x\big),$$7 depends explicitly on $$S_{\mathrm{ens}}(x,y)=\phi\big(s_1(x,y),\dots,s_K(x,y);x\big),$$8. The method is asymptotically conditionally valid if $$S_{\mathrm{ens}}(x,y)=\phi\big(s_1(x,y),\dots,s_K(x,y);x\big),$$9 (Plassier et al., 22 Feb 2025). EPICSCORE proves finite-sample marginal coverage and, under uniform convergence of the predictive CDF of the score, asymptotic conditional coverage for all $\phi$0 (Cabezas et al., 10 Feb 2025).

A third theme is that the aggregation class itself can be analyzed statistically. Weighted aggregation of classification scores connects the induced function class

$\phi$1

to subgraph classes studied in Vapnik-Chervonenkis theory, which yields uniform concentration results and oracle inequalities for expected set size (Luo et al., 2024). SACP proves validity for any symmetric aggregation function used on the per-model e-values and derives a worst-case efficiency bound in regression involving model disagreement and the largest individual conformal interval width at a more conservative level (Alami et al., 7 Dec 2025).

The transductive literature adds a distinct theoretical contribution. For arbitrary exchangeable scores, including adaptive ones that may use the covariates of the test and calibration samples at training stage, the joint distribution of the $\phi$2 conformal p-values follows a Pólya urn model, and the empirical distribution function $\phi$3 satisfies a DKW-type concentration inequality (Gazin et al., 2023). This result is directly relevant to conformal ensemble scoring because it shows that once an adaptive or ensemble score remains exchangeable, its batchwise conformal p-values admit a universal joint law.

Dynamic environments motivate yet another theory layer. The multi-model ensemble online conformal framework of MOCP and SAMOCP does not aggregate scores into a fixed scalar once and for all; instead it selects the model used for creating prediction sets on the fly from multiple candidate models. The resulting algorithms are proved to achieve strongly adaptive regret over all intervals while maintaining valid coverage (Hajihashemi et al., 2024).

4. Principal design paradigms

One major paradigm is score-function aggregation. In multiclass classification, several conformal scores may preserve the same class ordering while emphasizing different geometry: THR uses $\phi$4, APS uses cumulative mass, and RANK uses normalized rank. Weighted aggregation treats these as complementary statistics and learns a convex combination that minimizes prediction set size subject to conformal calibration (Luo et al., 2024). SACP operates at a similar level of abstraction but standardizes model-specific score scales through e-values before aggregation, thereby making the ensemble score permutation-invariant in the model index and directly comparable across heterogeneous predictors (Alami et al., 7 Dec 2025).

A second paradigm is uncertainty-aware aggregation of repeated stochastic scores. The $\phi$5-value method addresses settings where a candidate receives many scores rather than one: Bayesian posterior samples, MC-dropout passes, prompt paraphrases, or evaluator reruns. It is explicitly designed to retain variability information that average-then-calibrate discards. Under the Normal–Normal empirical Bayes model, the $\phi$6-value is the smallest $\phi$7 such that the observed score exceeds a variance-dependent threshold $\phi$8, so high-variance candidates are automatically penalized through shrinkage (Zeng et al., 22 May 2026).

A third paradigm is score-distribution modeling. EPICSCORE does not combine several raw scores directly. Instead, it models the conditional distribution of a single base score $\phi$9 through a Bayesian or ensemble model and then uses the posterior predictive CDF $$\langle w, s(x,y)\rangle = \sum_{j=1}^d w_j s_j(x,y)$$0 as a score transformation. This creates a score that expands predictive regions in data-sparse regions and contracts them where the score distribution is sharp, without abandoning distribution-free marginal guarantees (Cabezas et al., 10 Feb 2025).

A fourth paradigm is feature-dependent rectification. RCP starts from any baseline conformity score $$\langle w, s(x,y)\rangle = \sum_{j=1}^d w_j s_j(x,y)$$1 and estimates the conditional $$\langle w, s(x,y)\rangle = \sum_{j=1}^d w_j s_j(x,y)$$2-quantile of a transformed scalar score. The rectified score

$$\langle w, s(x,y)\rangle = \sum_{j=1}^d w_j s_j(x,y)$$3

acts like a learned, covariate-dependent rescaling of the base score. The paper emphasizes that this is particularly beneficial for adaptive confidence sets in multi-output problems where standard conformal quantile regression approaches have limited applicability (Plassier et al., 22 Feb 2025).

A fifth paradigm is ensemble construction through nested or out-of-bag predictors. The nested conformal framework shows that one may start from a family of nested sets $$\langle w, s(x,y)\rangle = \sum_{j=1}^d w_j s_j(x,y)$$4, define the score $$\langle w, s(x,y)\rangle = \sum_{j=1}^d w_j s_j(x,y)$$5, and then aggregate these leave-one-out or out-of-bag scores in a cross-conformal fashion. QOOB is the clearest instance: its score is derived from out-of-bag quantile regression forests and then aggregated across training points, yielding an ensemble conformal score that combines quantile regression, cross-conformalization, ensemble methods, and out-of-bag predictions (Gupta et al., 2019).

A sixth paradigm is score refinement or online model selection. In information retrieval, a monotone transformation combines cosine similarity, max-score normalization, and rank-dependent discounting into a refined score that produces significantly smaller conformal retrieval sets while maintaining statistical guarantees (Intrator et al., 2024). In dynamic online classification, MOCP and SAMOCP assign exponential weights to models or experts based on conformal pinball losses, so the effective conformal score is selected or averaged according to its recent coverage-efficiency performance (Hajihashemi et al., 2024).

5. Applications and empirical behavior

The empirical record of conformal ensemble scores is heterogeneous because the underlying tasks differ sharply, but a common pattern is improvement in efficiency or local calibration at fixed or near-fixed marginal coverage.

In multi-output regression, RCP is evaluated on multi-target UCI-style datasets and synthetic bivariate settings. The reported findings are that all methods achieve nominal marginal coverage, but base methods often under-cover in certain regions or slabs, whereas RCP applied on top of each base method substantially improves worst-slab coverage and reduces conditional coverage error. In synthetic two-moon mixtures, RCP applied to multivariate density-based scores yields prediction sets close to oracle highest-density regions (Plassier et al., 22 Feb 2025).

In vision and language, the $$\langle w, s(x,y)\rangle = \sum_{j=1}^d w_j s_j(x,y)$$6-value framework reports that target coverage is preserved while ranking stability improves and set size decreases when variability is informative. On CLIP-based VLM benchmarks, the paper reports, for example, that CIFAR-10 CLIP set size drops from about 1.4 to 1.3, and ImageNet CLIP drops from 6.9 or 10.1 to 5.7. The same work emphasizes that when variability vanishes, the method reverts to CP-like behavior (Zeng et al., 22 May 2026).

EPICSCORE is evaluated on 13 standard regression datasets and CIFAR-100. The paper reports that EPICSCORE, especially the MDN+MC dropout variant, often achieves the best AISL across datasets in both quantile- and regression-based settings, tends to widen intervals in data-sparse or outlier regions while keeping intervals compact in dense regions, and on CIFAR-100 achieves SSC closer to target, with $$\langle w, s(x,y)\rangle = \sum_{j=1}^d w_j s_j(x,y)$$7 versus $$\langle w, s(x,y)\rangle = \sum_{j=1}^d w_j s_j(x,y)$$8 for APS at $$\langle w, s(x,y)\rangle = \sum_{j=1}^d w_j s_j(x,y)$$9 (Cabezas et al., 10 Feb 2025).

Model-level aggregation methods also show strong efficiency gains. Weighted aggregation of THR, APS, and RANK on CIFAR-100 consistently outperforms each single-score conformal predictor while maintaining valid coverage (Luo et al., 2024). SACP and SACP++ report empirical coverage close to nominal on both regression and classification tasks, while SACP++ almost always yields the smallest sets among methods with correct coverage (Alami et al., 7 Dec 2025).

In medical image classification under domain shift, CE-ViTs averages the Softmax vectors of three ViT experts trained on HAM10000, Dermofit, and ISIC, then conformalizes the ensemble distribution. The reported coverage rate is $F_{\text{test}}(y)=f(\mathbf{E}_{\text{test}}(y))$0, representing an improvement of $F_{\text{test}}(y)=f(\mathbf{E}_{\text{test}}(y))$1 compared to the HAM10000 model, and the average prediction set size for challenging misclassified samples increases from $F_{\text{test}}(y)=f(\mathbf{E}_{\text{test}}(y))$2 to $F_{\text{test}}(y)=f(\mathbf{E}_{\text{test}}(y))$3, which the paper interprets as more appropriate uncertainty under domain shift (Zoravar et al., 21 May 2025).

Other domains show the breadth of the idea. In conformal information retrieval, a monotone score refinement produces compact retrieval sets on BEIR benchmarks while maintaining statistical guarantees (Intrator et al., 2024). In data assimilation with the modified shallow water model, ensemble-based and mean-based conformal intervals are compared with standard deviation intervals and ensemble spread, and conformal interval widths are even used as perturbation amplitudes inside the data assimilation cycle (George et al., 25 Jun 2026). In explanation approximation, conformal regression intervals around surrogate SHAP predictors make it possible to compare different approximation models by how informative their intervals are (Alkhatib et al., 2023).

6. Limitations and open directions

The literature does not present a single canonical definition of the term. Several papers explicitly note that they do not use the phrase “conformal ensemble score,” even though their constructions map naturally onto it (Plassier et al., 22 Feb 2025, Cabezas et al., 10 Feb 2025, Alami et al., 7 Dec 2025). The term therefore designates a family resemblance rather than a unique formal object.

Exchangeability remains the central assumption behind finite-sample coverage. This is explicit in split conformal aggregation methods, in EPICSCORE, in SACP, in $F_{\text{test}}(y)=f(\mathbf{E}_{\text{test}}(y))$4-value CP, and in transductive adaptive-score theory (Cabezas et al., 10 Feb 2025, Alami et al., 7 Dec 2025, Zeng et al., 22 May 2026, Gazin et al., 2023). Dynamic-environment methods relax the static data assumption operationally, but their guarantees are cast in terms of coverage error and regret rather than classical i.i.d. exchangeability (Hajihashemi et al., 2024).

Effectiveness also depends on how informative the ensemble ingredients are. The $F_{\text{test}}(y)=f(\mathbf{E}_{\text{test}}(y))$5-value method is strongest when score variability reflects epistemic uncertainty rather than noise; when variability is small, the method is designed to behave like standard CP (Zeng et al., 22 May 2026). EPICSCORE’s asymptotic conditional guarantee depends on uniform convergence of the predictive CDF of the score, and poor Bayesian or ensemble approximations can weaken local adaptivity even though marginal coverage remains guaranteed (Cabezas et al., 10 Feb 2025). RCP depends on learning a conditional quantile transform accurately enough that the quantile error function $F_{\text{test}}(y)=f(\mathbf{E}_{\text{test}}(y))$6 is small (Plassier et al., 22 Feb 2025).

A further limitation is sample efficiency. Several methods split calibration data into multiple parts: RCP uses a quantile-learning set and a proper conformal calibration set, while EPICSCORE introduces an inner split of the calibration data to learn the predictive distribution of the score and then calibrate the transformed score (Plassier et al., 22 Feb 2025, Cabezas et al., 10 Feb 2025). This can reduce effective sample size, especially in small-data regimes.

Open directions already identified in the cited works include deeper symmetric aggregators for SACP beyond the simple $F_{\text{test}}(y)=f(\mathbf{E}_{\text{test}}(y))$7 family, adaptation of EPICSCORE to Jackknife+, CV+, and multi-split conformal frameworks, richer ensemble and Bayesian architectures for score-distribution modeling, and broader treatment of distribution shift or OOD robustness (Alami et al., 7 Dec 2025, Cabezas et al., 10 Feb 2025). The transductive theory for adaptive scores suggests another direction: once an ensemble score can be made exchangeable through permutation-invariant training on calibration and test covariates, its batchwise p-values inherit the universal Pólya urn law and corresponding concentration bounds (Gazin et al., 2023). This suggests that future work on conformal ensemble scores is likely to focus less on the existence of valid aggregation schemes than on how to optimize them for conditional calibration, dynamic robustness, and computational efficiency.