Accelerating Conformal Prediction via Approximate Leave-One-Out

Abstract: While conformal prediction provides a general framework for uncertainty quantification in predictive inference, its application is often limited by computational cost. Recent methods, including Jackknife+ and Jackknife-minmax, achieve faster computation by trading a slight loss of efficiency relative to full conformal prediction, but still requires computing leave-one-out refits for all observations. In this paper, we further accelerate conformal prediction by incorporating approximate leave-one-out (ALO) estimators, and establish asymptotic coverage and efficiency. While our proof draws on methods developed for analyzing the consistency of ALO cross-validation risk estimators in high-dimensional statistics, it requires adaptations to handle conformal prediction, where leave-$i$-out residuals are needed for predictions at $x_{n+1}$ rather than just at the training covariate $x_i$. Simulation results validate our theoretical findings, showing that the ALO-based methods achieve coverage and efficiency comparable to the exact methods, while significantly reducing the runtime.

• The paper proposes an ALO surrogate that bypasses full leave-one-out re-fitting, reducing computational complexity from O(np^2) to O(p^2).
• It establishes uniform nonasymptotic and asymptotic error bounds, ensuring that fast conformal intervals maintain valid coverage even at new covariates.
• Empirical results on synthetic and UCI datasets demonstrate near-identical coverage and interval lengths, with computation times reduced by one to two orders of magnitude.

Accelerating Conformal Prediction via Approximate Leave-One-Out

Overview and Motivation

This paper addresses the prohibitive computational complexity inherent in state-of-the-art conformal prediction procedures for distribution-free predictive inference. Canonical conformal methods, including Full Conformal Prediction (FCP), Jackknife+, and Jackknife-minmax, entail either grid-searching over candidate targets with model re-fitting (FCP) or $n$ leave-one-out (LOO) ERM re-fits (Jackknife+/-minmax). These requirements severely impede scalability for large $n$ or $p$.

Recent work has explored ALO (Approximate Leave-One-Out) techniques (e.g., AMP-based (Clarté et al., 2024), influence function-based [AppFCP_via_IF, Bhatt_FastCP], high-dimensional theory [RadMaleki2020]) to reduce this overhead. However, prior approaches either require restrictive assumptions (e.g., i.i.d. isotropic design, diagonal $\Sigma$), do not guarantee uniform approximation at new covariates, or lack theoretical validation vis-à-vis conformal coverage and efficiency.

This study provides a comprehensive framework for accelerating Jackknife+ and Jackknife-minmax algorithms using theoretically justified ALO surrogates. In particular, it proves nonasymptotic and asymptotic error bounds for ALO-based conformal score computation at arbitrary test covariates and establishes that the fast algorithms asymptotically retain coverage and efficiency.

Methodological Advances

Approximate Leave-One-Out Estimation

The authors leverage recent advances in high-dimensional risk estimation for convex M-estimators to derive ALO surrogates for leave-one-out coefficients. Given standard regularity and stability assumptions on the ERM, the Newton step and Woodbury lemma provide the ALO formula: $$\tilde{\theta}_{/i} = \hat{\theta} + \frac{\mathbf{J}^{-1} \boldsymbol{x}_i\dot{\ell}_i(\hat{\theta})}{1 - \boldsymbol{x}_i^\top \mathbf{J}^{-1} \boldsymbol{x}_i\ddot{\ell}_i(\hat{\theta})}$$ where $\mathbf{J}$ is the empirical Hessian and the quantities $\dot{\ell}_i(\cdot), \ddot{\ell}_i(\cdot)$ denote the first and second derivatives of the loss evaluated at the sample.

This approximate update obviates explicit LOO re-fitting for each observation, reducing computational complexity from $O(np^2)$ down to $O(p^2)$ when $\mathbf{J}^{-1}$ is cached.

Theoretical Guarantees

The paper establishes uniform approximation bounds for ALO at both in-sample and new covariates under high-dimensional asymptotics ($n$0 with $n$1). Crucially, the proof of uniform control at test covariate $n$2 adapts and extends decomposition techniques in [RadMaleki2020], reducing the statistical analysis of LOO intervals at test covariates to the in-sample setting via a carefully controlled error propagation analysis.

Letting $n$3 and $n$4, the fast Jackknife+ prediction interval is

$n$5

where $n$6 are the empirical quantiles.

The main results, under suitable design and regularity conditions, are:

• Uniformly in $n$7: $n$8.
• Asymptotic equivalence of ALO and exact conformal intervals: The upper and lower endpoints of accelerated Jackknife+/minmax intervals converge (in probability) to those of the exact versions.
• Coverage and efficiency preservation:

$n$9

$p$0

• Asymptotic equivalence of FCP, split conformal, Jackknife+, and Jackknife-minmax intervals in prototypical high-dimensional linear models.

These results decisively close the gap between computational efficiency and statistical guarantees for fast conformal prediction in high dimensions—under substantially weaker assumptions than prior work.

Empirical Results

The experimental evaluation covers both synthetic and real-world datasets. In high-dimensional simulations with Gaussian features and both Ridge and pseudo-Huber regularization, fast Jackknife+ and Jackknife-minmax achieve:

• Empirical coverage nearly identical to their exact counterparts;
• Interval lengths nearly unchanged;
• Computational time reductions of one to two orders of magnitude;
• Jaccard index for interval overlap close to 1 (0.999+ across configurations).

Results on UCI Concrete and Energy Efficiency datasets further confirm practical parity in interval coverage and length with dramatic acceleration.

These outcomes are stable across a range of feature dimensions ($p$1), $p$2 regimes, and regularization strengths.

By accommodating general covariance structure $p$3 (beyond isotropic/diagonal designs, which break AMP/Taylor-based fast CP), the method provides robustness to correlated designs.

Theoretical and Practical Implications

Theoretical Impact: The results establish that computational shortcuts using ALO remain sound for conformal inference in high dimensions, bridging statistical learning theory, numerical linear algebra, and predictive inference. The uniform error control at new covariate points is essential for finite-sample guarantees in predictive uncertainty quantification.

The asymptotic equivalence of different split/conformal methods with LOO approximation, in the regime $p$4, implies fundamental limits to further improvements without extra assumptions or additional distributional information.

Practical Consequences: The methodology makes conformal prediction feasible and principled for large-scale applications where naive LOO or full conformal are computationally prohibitive, including genomics, neuroscience, and real-time model deployment at scale.

By removing the isotropic design restriction, the technique will be robust under more realistic feature dependence, e.g., in finance and biomedical data.

Speculation on Future Directions

Several promising avenues emerge:

• Relaxation to non-convex settings and deep networks: The machinery is currently limited to convex loss and twice-differentiable regularization. Extensions to nonconvex or piecewise-smooth landscapes, perhaps using variants of influence function or generalized Hessian techniques, form an important research direction.
• Calibration under data heterogeneity: The method assumes exchangeability; extensions to covariate shift, strong dependence, or adversarial contamination should be explored, potentially using robust risk estimation tools.
• Adaptive regularization and online settings: Incorporating model selection or adaptive regularization strategies within the ALO framework can further enhance robustness and practicality.
• Unified analysis with CLT-based plug-in intervals: The paper notes that, under strong independence, conformal and CLT-based plug-in intervals coincide asymptotically. Exploring where this equivalence breaks, e.g., beyond sub-Gaussian or heavy-tailed designs, would clarify the precise limits of inference.

Conclusion

This work rigorously integrates fast leave-one-out approximation into the conformal prediction framework, providing both strong theoretical justification and substantial empirical gains. The results simultaneously advance the theory of computational-statistical trade-offs in predictive inference and make distribution-free uncertainty quantification substantially more accessible for large data and high-dimensional tasks.

Reference: "Accelerating Conformal Prediction via Approximate Leave-One-Out" (2606.31915)