Confidence Curves in Uncertainty Quantification

• Confidence curves are statistical tools that generalize conventional confidence intervals into function-valued summaries, accurately representing uncertainty over entire domains.
• Methodologies for constructing confidence curves include parametric likelihood, nonparametric techniques, bootstrap ensembles, and deep learning approaches to ensure uniform coverage and optimal rates.
• Applications span model tuning, functional data analysis, and scientific computing, enabling rigorous calibration diagnostics and comparative evaluation across various fields.

Confidence curves are graphical and analytic devices for representing statistical uncertainty as a function-valued summary; for a statistical parameter, regression function, prediction, or performance metric, a confidence curve or simultaneous confidence band characterizes the range where the true underlying quantity lies with a prescribed probability uniformly over its domain. Confidence curves generalize the classical notion of a confidence interval from a fixed-point to a function, aiming to visualize and rigorously quantify uncertainty across a continuum, such as parameter space, covariate values, or tuning/trade-off indices.

1. Conceptual Foundations and Terminology

A confidence curve is any data-dependent function $c(\cdot)$ such that, for a target function or parameter $\theta(\cdot)$, the set $\{\theta(x): c(x) \leq 1-\alpha\}$ forms a $100(1-\alpha)\%$ simultaneous confidence region over $x$ (or $p$, $t$, etc.), often with uniform coverage. For scalar inference, confidence curves are closely tied to confidence distributions (CDs): a function $C(\theta;x)$ mapping the parameter space to $[0,1]$ so that, under the true value $\theta_0$, $C(\theta_0;X)$ is uniformly distributed (see (Wimbush et al., 2021, Blasi et al., 2016)).

For regression functions $f(\cdot)$, quantile curves $Q_\tau(\cdot)$, ROC curves, enrichment curves, densities, and more, confidence curves typically take the form of simultaneous upper and lower functions $L(\cdot), U(\cdot)$ such that

$$\Pr\left\{L(x) \leq \theta(x) \leq U(x) \text{ for all } x \in D\right\} \geq 1-\alpha$$

where $D$ is the domain of interest.

In uncertainty quantification, confidence curves also refer to rank-based summaries of errors and provided uncertainties (see (Pernot, 2022)), where the curve $c_S(k)$ measures the error as a function of the top $k\%$ largest uncertainties discarded.

2. Methodologies for Constructing Confidence Curves

Parametric and Likelihood-Based Confidence Curves

• Confidence distributions and tail-symmetry: Median bias correction of the likelihood ratio yields third-order tail-symmetric confidence curves $cc^*(\theta)$, whose level-sets correspond to equal-tailed intervals and achieve coverage errors $O(n^{-3/2})$ in regular models (Blasi et al., 2016).
• Singh plots (confidence curves for CDs): For a CD $C(\theta; x)$, the empirical CDF of $C(\theta_0; x_i)$ across simulated datasets forms the confidence curve ("Singh plot"), which when agreeing with the diagonal confirms exact frequentist coverage (Wimbush et al., 2021).
• Feldman–Cousins approach: In particle physics, confidence curves are constructed via toy Monte Carlo simulations of the likelihood-ratio ordering, generating a function $1-\mathrm{CL}(\mu)$ whose sub-level sets define confidence intervals with correct coverage, including under nonstandard boundaries (Karbach, 2011).

Functional and Nonparametric Simultaneous Bands

• Shape-restricted regression: Multiscale Gaussian maxima control ensures uniform bands for monotone/convex functions, using local means over intervals and critical boundaries $\Gamma(u) = \sqrt{2\log(e/u)}$; post-processing (isotonic or convex projection) enforces global shape constraints (Duembgen, 2013).
• Isotonic quantile regression: Honest confidence bands for monotone quantile curves use union-intersect tests over all subintervals, with coverage guaranteed by stochastic dominance against binomial distributions; computation scales as $O(n^2)$ and achieves $O\left((\log n/n)^{\beta/(2\beta+1)}\right)$ width in smooth areas (Duembgen et al., 2022).
• Density curves via Wasserstein geometry: For Fréchet means in Wasserstein space, simultaneous bands are built for quantile functions using Gaussian process approximations, Karhunen–Loève simulations, and projection to monotone cones; density-level bands are obtained via the delta method and Hadamard-differentiable transforms (Petersen et al., 2019).

Deep Learning Predictive Curves

• Bootstrap–ensemble methodology: Nonparametric bands for deep nets are computed by (i) forming an ensemble to suppress optimization noise and (ii) bootstrapping over data resamples. Simultaneous (sup-norm) bands over prediction domains or time indices ($t$ for survival) are constructed via empirical quantiles of $\{\sup_x |\Delta^{(b)}(x)|\}$ (Arie et al., 20 Jun 2024).

ROC, Precision–Recall, and Enrichment Curves

• ROC bands (weighted SVM, hit enrichment): Empirical ROC or enrichment curves indexed by cost/trade-off parameter are accompanied by uniform bands using multiplier or exponential bootstrap of the underlying risk or recall processes, with critical values estimated by the supremum-norm over the index domain (Luckett et al., 2018, Ash et al., 2019).
• Tuning curves in model selection: Exact, distribution-free bands for median/mean tuning curves are constructed from simultaneous CDF bands using the Learned-Miller–DeStefano approach, propagated to tuning curves via $F_k(y) = F(y)^k$ (Lourie et al., 2023).

Geometric and Manifold-valued Curves

• Functional data on SO(3), regression contrasts: Confidence tubes or bands for curves in $SO(3)$ (gait analysis) or for nonlinear contrasts use Gaussian kinematic formulae, Hotelling-type processes, and expected Euler characteristic heuristics to control excursion probabilities over possibly curved domains (Telschow et al., 2019, Lu et al., 2015).

3. Theoretical Guarantees and Asymptotic Properties

• Uniform Coverage: Most constructions, including multiscale tests (Duembgen, 2013), ensemble bootstrap (Arie et al., 20 Jun 2024), Wasserstein bands (Petersen et al., 2019), and ROC bootstraps (Luckett et al., 2018), guarantee asymptotic, or even nonasymptotic, coverage of $1-\alpha$ simultaneously over the entire domain.
• Rate-Optimality: Confidence bands for shape-restricted functions achieve the minimax optimal rate for band width, e.g., $O((\log n/n)^{\beta/(2\beta+1)})$ (Hölder smoothness), and for isotonic curves $O((\log n/n)^{1/3})$ (Duembgen, 2013, Duembgen et al., 2022).
• Calibration diagnostics: Singh plots graphically diagnose over-/under-coverage, while probabilistic confidence curves for uncertainty quantification provide direct tests of calibration and tightness against a probabilistic reference model (Pernot, 2022).
• Anti-concentration and critical values: Many bands rely on precise control of the maximum deviation of a Gaussian/Brownian process (Kolmogorov, supremum), whose quantiles are computed via Monte Carlo or explicit formulas (Dunker et al., 2017, Bengs et al., 2019).

4. Practical Implementation

Methodology	Domain	Key Steps/Computational Notes
Multiscale bands	Regression curves	$O(n^2)$ interval evaluation; post-process with pool-adjacent-violators algorithm
Wasserstein bands	Density curves	Eigen-decomposition for GP simulation; monotonic projection
Deep learning bands	Predictions, survival	Ensemble runs $M=50\text{–}200$, bootstrap $B=100\text{–}500$; parallelized refits
Tuning curves	Hyperparameter search	Distribution-free LD bands; opda library in Python
ROC/enrichment curves	Classification, ranking	Bootstrap/sup-norm; kernel estimation for covariance

Confidence curves are typically computed on fine grids over their domain ($x$, $p$, $k$, $t$, etc.), with critical values estimated either analytically (e.g., Kolmogorov quantiles for Brownian bridges) or via Monte Carlo/bootstrap approximation. Resampling, multiplier, or weighted bootstraps are preferred for empirical processes with dependence structure; ensemble averaging is vital for deep learning models to isolate data vs. algorithmic uncertainty (Arie et al., 20 Jun 2024).

5. Applications and Interpretation

Confidence curves and bands have become indispensable for:

• Comparing predictive models over hyperparameter or tuning budgets, with simultaneous bands preventing misleading conclusions from point estimates alone (Lourie et al., 2023).
• Uncertainty validation in scientific computing and metrology, notably in computational chemistry, where confidence curves with probabilistic references rigorously test both association and calibration (Pernot, 2022).
• Functional data analysis, including growth curves, gait patterns on manifolds (Lu et al., 2015, Telschow et al., 2019), and ratio-of-quantile inference in economics or medicine (Dunker et al., 2017).
• Image and edge detection, with confidence sets for jump curves produced via kernel-based contrast processes and Gaussian approximation (Bengs et al., 2019).
• Ranking/early enrichment in drug discovery, with simultaneous bands quantifying the significance of observed hit-enrichment curves (Ash et al., 2019).
• Classification and model selection, with ROC and related curves equipped with honest uncertainty quantification via bootstrapped or conformal bands (Luckett et al., 2018).

Simultaneous coverage is essential: pointwise intervals often drastically under-cover when interpreted uniformly (see (Dunker et al., 2017)). Confidence curves enable valid comparison between methods, calibrated uncertainty estimates, and identification of significant features or deviations across entire functional domains.

6. Limitations and Future Directions

Most confidence curve methodologies require independence or weak dependence (i.i.d. data, smoothness, shape constraints), accurate simulation or approximation for critical values (especially extreme quantiles), and may suffer edge effects (bandwidth for quantiles, density estimation for Wasserstein bands). Extensions to dependent data, non-Gaussian error distributions, multivariate and manifold-valued domains, and integration with proper scoring or Bayesian analogs are active areas of research (Lu et al., 2015, Telschow et al., 2019, Petersen et al., 2019).

Open questions include analytic calculation of probabilistic reference bands without Monte Carlo, robust adjustment for non-Normal error distributions, adaptive selection of domain-restricted intervals for computational efficiency, and unified frameworks for confidence curves across statistical and machine learning paradigms.

7. References and Tools

Below is a table of representative methods and their main computational tools or packages:

Method/Domain	Paper Reference	Tool/Algorithm
Tuning curves, model selection	(Lourie et al., 2023)	opda Python library
Wasserstein density bands	(Petersen et al., 2019)	Algorithm 1–6 (Appendix)
Isotonic quantile curves	(Duembgen et al., 2022)	$O(n^2)$ back-scan algorithm
ROC curve bands (SVM)	(Luckett et al., 2018)	Weighted exponential bootstrap
Rank-based error CCs (UQ)	(Pernot, 2022)	ErrViewLib R package
Hit enrichment bands	(Ash et al., 2019)	R package “chemmodlab”

Confidence curves thus unify rigorous uncertainty quantification, calibration diagnostics, and principled comparison for a broad class of statistical and predictive methodologies across regression, classification, geometry, and uncertainty validation.