Aggregation in conformal e-classification

Abstract: Aggregating conformal predictors is a standard way of balancing their predictive and computational efficiency while retaining their validity, at least approximately. An important advantage of conformal e-predictors is that they are easier to aggregate without sacrificing their validity. This paper studies experimentally cross-conformal e-prediction, which is an existing method of aggregating conformal e-predictors, and its modifications that are conceptually simpler and more flexible.

• The paper shows that averaging conformal e-values preserves exact validity, unlike p-value aggregation, while simulations identify deleted full conformal e-prediction as the strongest baseline.
• It finds that cross-conformal e-prediction has no universally optimal fold count, with the best choice varying by class count, and that repeated aggregation improves log-efficiency through Jensen’s inequality.
• It introduces RICEP and BICEP, which offer tunable or automatically balanced training–calibration splits and provide practical alternatives when the appropriate calibration fraction is unknown.

Overview and motivation

This paper by Vladimir Vovk studies the aggregation of conformal predictors in a classification setting, with an emphasis on conformal e-prediction. The starting point is a well-known tension in conformal prediction: full conformal prediction is computationally expensive, while inductive (split) conformal prediction sacrifices predictive efficiency. A standard remedy is to aggregate several inductive conformal predictors, as in cross-conformal prediction (2605.07963). For p-value-based methods such as cross-conformal p-prediction (CCP) and jackknife+, aggregation weakens provable validity: the guaranteed error probability may increase by roughly a factor of 2, even though empirical validity typically remains intact. The key structural advantage of e-values is that their arithmetic mean is again an e-value, so cross-conformal e-predictors (CCEP) inherit the validity of conformal e-predictors exactly—there is no mismatch between provable and typical empirical validity.

The paper's stated goal is modest: it contains no non-trivial theoretical results beyond several propositions, and its contribution is a critical discussion of CCEP together with simulation studies of more flexible modifications, namely repeated inductive conformal e-prediction (RICEP) and balanced inductive conformal e-prediction (BICEP). All experiments are conducted under a known Bayesian model with Jeffreys's prior, in the spirit of the Burnaev–Wasserman programme; the model serves only to define nonconformity measures, while validity of the resulting predictors holds under exchangeability alone.

Efficiency criteria

Predictive efficiency is measured by two log-scale criteria evaluated in the limit of an infinite test set. For conformal p-prediction, the paper introduces the AFS criterion: the average of $-\ln p_i^y$ over all test observations and all false labels $y$. A short proof establishes that AFS is conditionally proper, extending the argument of Remark 3.15 in Algorithmic Learning in a Random World to handle the singularity at $p=0$. The analogue for e-values is the AFES criterion from Vovk's earlier work on conformal e-prediction: the average of $\ln e_i^y$ over false labels. Logarithmic scales are justified by additivity—for e-values via Kelly-type arguments, for p-values via precedents ranging from Fisher's combined probability test to S-values (p-surprisals) and algorithmic randomness theory.

A companion "modified AFES" criterion averages $\ln e_i^y$ over all labels rather than only false ones. Under this criterion the optimal nonconformity measure is proportional to the inverse conditional probability of the label, whereas under AFES it is proportional to the odds against the label. This distinction generates paired "optimal" and "suboptimal" versions of every predictor studied later.

Bayesian benchmarks

The data-generating mechanism is a Dirichlet-multinomial model: $\theta \sim Dir_\alpha$ on the simplex over $\mathbf{Y}=\{1,\dots,Y\}$, with $\alpha=0.5$ (Jeffreys's prior) throughout. The Bayes predictor generalizes Laplace's rule of succession, giving $P(y)=(n_y+\alpha)/(n+Y\alpha)$.

Three propositions characterize optimal Bayesian p- and e-variables under these criteria. The optimal smoothed p-variable has the usual randomized form $p_y = A_y + \tau B_y$, where $y$0 and $y$1 aggregate probability mass below and at $y$2 respectively. The optimal Bayesian e-value under AFES is

$y$3

shown to be the unique solution of the corresponding optimization problem (uniqueness follows from strict concavity of the logarithm). These finite-training-set results extend Proposition 15 of the conformal e-prediction paper, which assumed infinite training sets. The suboptimal variant replaces odds against the label by inverse probability, matching the modified criterion. These Bayes predictors serve as upper benchmarks ("e-Bayes", "p-Bayes") against which all conformal predictors are compared.

Full and inductive conformal prediction

For full conformal e-prediction (CEP), the paper introduces a parameter $y$4 interpolating between deleted ($y$5), quasi-studentized ($y$6), and ordinary ($y$7) definitions—an issue specific to e-values, since the full conformal p-value does not depend on this choice. Simulation results with training size $y$8 show deleted CEP strictly dominates ordinary CEP across $y$9 classes, with intermediate performance at $p=0$0. Accordingly, $p=0$1 is fixed for all subsequent experiments. The choice $p=0$2 is deliberate: it yields e-values comparable to conventional thresholds near $p=0$3 and $p=0$4, and admits many small divisors usable as fold counts.

For inductive conformal prediction, a notable finding concerns smoothing. Smoothed p-values substantially outperform deterministic ones on the AFS criterion, especially for binary classification where ties are pervasive; for $p=0$5 the deterministic curves fall so far below the smoothed ones that they are omitted from the plots. E-values require no smoothing at all, which the paper lists as a genuine advantage. Another finding is that there is no universally good calibration fraction: for $p=0$6 the optimal calibration set is small, while for $p=0$7 nearly the entire training set should be used for calibration. This heterogeneity motivates the flexible aggregation methods introduced later.

Cross-conformal prediction: divergent behaviour of p- and e-variants

The experimental contrast between CCP and CCEP is one of the paper's sharpest results. For CCP, AFS quality improves monotonically in the number of folds $p=0$8, and leave-one-out CCP coincides with the full conformal p-predictor—a fact established formally by showing that the aggregated $p=0$9 and $\ln e_i^y$0 quantities match those of the full predictor when $\ln e_i^y$1. The cost is that CCP lacks provable validity.

CCEP behaves differently: quality is not monotone in $\ln e_i^y$2. With $\ln e_i^y$3 the optimum occurs at $\ln e_i^y$4; with $\ln e_i^y$5 at $\ln e_i^y$6; with $\ln e_i^y$7 at $\ln e_i^y$8, with smaller (infeasible) values apparently preferable. Thus, unlike cross-validation folklore where $\ln e_i^y$9 or $\ln e_i^y$0 are broadly reasonable, no universal fold count exists for CCEP. Two structural limitations follow: the calibration fraction per component is constrained to $\ln e_i^y$1 with integer $\ln e_i^y$2, so at most half the training data can be used for calibration, and the fraction cannot be tuned finely.

To address the first limitation when small calibration fractions are desirable, the paper proposes inverse CCEP, in which each fold serves as the proper training set and the remaining folds as calibration. For binary classification, inverse CCEP attains its optimum at $\ln e_i^y$3 and visibly outperforms plain CCEP at $\ln e_i^y$4. Inverse CCP, by contrast, is essentially useless—the asymmetry again traces back to averaging being valid only for e-values.

RICEP and BICEP

The central methodological proposal is repeated inductive conformal e-prediction (RICEP): choose an arbitrary proper-training-set size $\ln e_i^y$5, draw $\ln e_i^y$6 random splits into proper training and calibration sets, compute the ICEP e-value for each split, and average. Two consequences follow. First, by Jensen's inequality,

$\ln e_i^y$7

so quality increases with $\ln e_i^y$8: RICEP 100 dominates RICEP 10, which dominates RICEP 1, with diminishing increments as the averaged quantities stabilize. Second, large-$\ln e_i^y$9 RICEP is pseudo-deterministic, mitigating the volatility of single-split ICEP. The same Jensen-gap mechanism explains why CCEP outperforms ICEP at matched split proportions, since CCEP averages $\theta \sim Dir_\alpha$0 component e-values.

Balanced ICEP (BICEP) removes the need to choose $\theta \sim Dir_\alpha$1 altogether: the calibration-set size is drawn uniformly from $\theta \sim Dir_\alpha$2 (or from $\theta \sim Dir_\alpha$3 in semi-BICEP), followed by a uniform random split of that size. Experiments show BICEP performs best precisely when no prior knowledge about the appropriate split proportion exists—as for $\theta \sim Dir_\alpha$4, where very large calibration fractions are needed—while paying a price relative to tuned RICEP when such knowledge is available, as for $\theta \sim Dir_\alpha$5 and $\theta \sim Dir_\alpha$6.

The paper summarizes the RICEP-versus-CCEP trade-off explicitly: RICEP offers pseudo-determinism, arbitrary split proportions, and controllable Jensen gap independent of the proportion, but CCEP may yield slightly better quality at equal numbers of repetitions. The recommendation is to use RICEP when a reasonable calibration percentage is known a priori, BICEP otherwise, and partial BICEP—with a non-uniform prior over calibration sizes—when partial knowledge exists.

Limitations and open questions

Several caveats bear directly on the results. All evidence is simulation-based, restricted to object-free multinomial classification under a known Bayesian soft model; nothing is claimed about regression or problems with covariates. The paper itself concedes that the Jensen gap is partly an artefact of the logarithmic efficiency criterion: whether the improvement from RICEP 1 to RICEP 100 reflects a "real" gain is explicitly left as a direction for further research, though the variance reduction from averaging is described as undoubtedly real—a point illustrated by a histogram experiment in which individual e-values for the least likely class separate bimodally into clusters above and below 800, while their average remains concentrated. Finally, the optimality propositions require the Bayesian assumption, and the practical guidance on choosing between RICEP, BICEP, and partial BICEP rests on empirical rather than theoretical grounds.

Conclusion

The paper demonstrates that e-value-based aggregation occupies a distinct position within conformal prediction: unlike p-value aggregation, it preserves exact validity, requires no smoothing, and supports flexible modifications. Its concrete contributions are the identification of the deleted version as the preferred full conformal e-predictor, the demonstration that no universal fold count exists for CCEP, and the introduction of RICEP and BICEP as flexible, pseudo-deterministic alternatives whose Jensen gap and split proportions can be controlled independently. The main open question left by the work is whether the Jensen-gap-driven efficiency gains of repeated aggregation correspond to genuine predictive improvement or are artefacts of the logarithmic evaluation criterion.