Conformalized Percentile Interval: Finite Sample Validity and Improved Conditional Performance

Abstract: Conformal prediction provides distribution-free predictive intervals with finite-sample marginal coverage. However, achieving conditional validity and interval efficiency (in terms of short interval length) remains challenging, particularly in complex settings with heteroskedasticity, skewed responses, or estimation errors. We propose a conformal-style calibration method for responses obtained by the probability integral transform (PIT) of the conditional cumulative distribution function (CDF) estimated via neural networks to construct a finite-sample-adjusted percentile interval with the shortest length determined by the estimated conditional CDF. Calibrating in PIT space is effective because PIT values are asymptotically feature-independent when the CDF estimator is accurate, which mitigates feature-dependent miscoverage and improves conditional calibration. On the other hand, our percentile calibration adapts to the empirical PIT distribution, which is robust against a possibly imperfect estimation of the conditional CDF. We prove the finite-sample marginal coverage property of the proposed method and show its asymptotic conditional coverage under mild consistency conditions. Experiments on diverse synthetic and real-world benchmarks demonstrate better conditional calibration and substantially shorter intervals than existing methods.

• The paper introduces CPI, which leverages conditional CDF estimation and PIT-based calibration to ensure finite-sample validity and improved conditional coverage.
• The methodology decouples endpoint calibration to effectively address issues with asymmetry and heteroskedasticity in prediction intervals.
• Empirical results show that CPI achieves a 34–36% reduction in interval width compared to symmetric methods while maintaining robust coverage across various datasets.

Conformalized Percentile Intervals: Finite-Sample Validity and Conditional Performance

Introduction and Motivation

Uncertainty quantification (UQ) is critical in modern predictive systems, particularly for applications where reliable risk assessment is necessary. While conformal prediction methodologies provide valid marginal coverage guarantees under minimal assumptions, strong per-instance reliability—namely, conditional coverage—remains an open challenge, especially in the presence of heteroskedasticity, asymmetry, or distributional misspecification.

The paper "Conformalized Percentile Interval: Finite Sample Validity and Improved Conditional Performance" (2605.03233) introduces a method, the Conformalized Percentile Interval (CPI), that leverages estimated conditional CDFs and a probability integral transform (PIT)-based calibration to produce finite-sample-valid intervals that are robust in the conditional sense and offer enhanced interval efficiency. This approach solves key limitations in prior distribution-aware conformal prediction frameworks.

Methodology: Conformalized Percentile Interval

CPI diverges from conventional conformal frameworks through two primary methodological innovations: endpoint calibration in PIT space and independent adjustment of coverage cutoffs.

The construction proceeds as follows:

1. Conditional CDF Estimation: Given a training set, a conditional distribution estimator (here, a neural network CDE following [hu2023nncde]) is fit to produce an estimator $\widehat F(y|x)$.
2. Probability Integral Transform: On a hold-out calibration set, responses are mapped via $U_j = \widehat F(Y_j|X_j)$. Under ideal estimation, the PIT values are uniformly distributed on $[0,1]$ and independent of the features.
3. Endpoint Selection: CPI chooses lower and upper PIT cutoffs $(u_{\mathrm{lo}}, u_{\mathrm{hi}})$ as independent order statistics from the calibration PIT sample. This allows for intervals that flexibly adapt to distributional asymmetry and heteroskedasticity.
4. Interval Construction: For a new input $x$, CPI forms the prediction interval $$\big[\widehat F^{-1}(u_{\mathrm{lo}}|x),\, \widehat F^{-1}(u_{\mathrm{hi}}|x)\big]$$.

This endpoint-based calibration generalizes beyond symmetric, score-based approaches (e.g., DCP). Notably, CPI's finite-sample marginal coverage guarantee holds irrespective of the underlying distribution or accuracy of the CDF estimator, provided data are exchangeable.

Comparative Analysis: CPI versus DCP

CPI and DCP both utilize PIT values for calibration, but DCP enforces symmetric cutoff intervals around a fixed center, leading to inefficiencies under asymmetric estimation errors or skewed distributions. CPI, by calibrating endpoints separately, better adapts to the empirical structure of the PIT distribution, particularly when the CDF estimator is biased or when model misspecification induces asymmetric deviation from uniformity.

This effect is visually apparent in right-skewed settings, such as the Beta(3,1) PIT distribution, where CPI intervals concentrate around high-density regions while DCP intervals are unnecessarily wide due to their symmetry constraint.

Figure 2: DCP center range $[0.45, 0.55]$ (red shading) versus the mean of $\mathrm{Beta}(3,1)$; CPI endpoints closely track the oracle optimal interval, while DCP's constraint leads to suboptimal coverage regions.

Empirical results confirm that CPI achieves a 34–36% reduction in PIT interval width relative to DCP for fixed coverage levels, and these gains transfer to narrower prediction intervals in the response space.

Theoretical Guarantees

The finite-sample marginal validity of CPI is established through standard exchangeable arguments: for any strictly calibration-set-independent starting point $z$, the construction yields coverage at least $1-\alpha$ for any test covariate $U_j = \widehat F(Y_j|X_j)$0. Furthermore, under consistency of the conditional CDF estimator and continuity of the target distribution, CPI achieves asymptotic conditional coverage at level $U_j = \widehat F(Y_j|X_j)$1 for almost every $U_j = \widehat F(Y_j|X_j)$2. These guarantees are robust to the choice of the starting point $U_j = \widehat F(Y_j|X_j)$3, which can be data-driven for interval length optimization.

Empirical Results: Simulations and Real Data

Conditional Calibration and Interval Efficiency

In heteroskedastic, multi-modal synthetic settings, CPI attains coverage near the nominal target while producing markedly shorter intervals than DCP, CQR, and residual-based approaches. Notably, CPI maintains conditional coverage above under-coverage thresholds even in extremes of heteroskedasticity, where standard approaches fail.

Figure 1: Smoothed conditional coverage curves versus a covariate, illustrating CPI and DCP maintain stable coverage across the feature space, with dashed and dotted lines marking nominal and under-coverage thresholds.

Real-World Datasets

Experiments on six real datasets further support CPI's strengths. For each principal component-derived subgroup, CPI not only preserves conditional coverage close to the nominal level but systematically yields the narrowest intervals among all methods.

Figure 3: Coverage versus interval width on the Abalone dataset, stratified by principal component quartile groups, demonstrating CPI achieves shortest intervals while maintaining nominal coverage.

Analogous efficacy is observed across other datasets, including Airfoil, Computer, Concrete, AutoMPG, and Crime, with similar visualizations available (Figures 5–9).

Practical Implications and Future Directions

CPI offers substantial improvements in the practical deployment of conformal prediction intervals, particularly in contexts characterized by:

• Non-Gaussian, skewed, or heteroskedastic errors,
• Model misspecification or covariate shift,
• Demands for robust conditional reliability or adaptivity to local data geometry.

The plug-and-play nature of the CPI framework means it can be combined with any conditional CDF estimator, e.g., normalizing flows, quantile regression forests, or high-capacity neural models. Additionally, the length-optimal starting point optimization—whether pointwise via grid search or amortized through an auxiliary model—introduces no additional risk to coverage, only improving efficiency.

Future research directions include:

• Integration of localized, weighted calibration or group-based coverage constraints;
• Extensions to high-dimensional or structured response spaces (e.g., multivariate quantile regions);
• Theoretical investigation of efficiency boundaries for asymmetric noise and distributional shift scenarios;
• Combination with endogenous or adaptive conformal strategies, especially under distribution drift.

Conclusion

The Conformalized Percentile Interval framework delivers a robust, finite-sample-valid method for predictive interval construction that demonstrably outperforms existing approaches in terms of conditional calibration and interval efficiency, especially in non-ideal and distributionally complex regimes. By decoupling endpoint calibration and sidestepping symmetry constraints, CPI sets a new standard for adaptive, practical uncertainty quantification in predictive modeling (2605.03233).