Conformal-Calibrated Interval (CCI)

• Conformal-Calibrated Interval (CCI) is a framework that constructs set-valued predictors ensuring valid uncertainty quantification using conformal prediction techniques.
• It employs a calibration procedure with an uncertainty-scaling function and conformal quantiles to achieve finite-sample marginal coverage even in non-i.i.d. or misspecified settings.
• CCI integrates an abstention mechanism based on interval endpoints to support uncertainty-aware decision-making in high-stakes applications like clinical diagnostics.

A Conformal-Calibrated Interval (CCI) is a statistically principled interval or set-valued predictor constructed to provide valid uncertainty quantification for predictions from machine learning models. CCIs generalize classical conformal prediction to accommodate flexible underlying predictors, explicit uncertainty-scaling, and group-conditional or composite targets, with finite-sample marginal (or group/multitarget) coverage guarantees even for complex, misspecified, or non-i.i.d. data regimes. The CCI framework is central in modern uncertainty-aware decision support, calibration of probabilistic classifiers, and risk-controlling sets in high-stakes applications, as exemplified by its deployment in clinical smart home systems for early detection of urinary tract infection flare-ups (Ugwu et al., 23 Nov 2025).

1. Formal Definition and Construction

Let $\mathcal X$ be a feature space and $Y\in\{0,1\}$ a binary outcome. Suppose a pre-trained probabilistic classifier $f:\mathcal X\to[0,1]$ outputs $p=f(x)=\widehat P(Y=1\mid x)$. To improve interpretability and enable coverage calibration, define a transformed label: $Y' = 0.25 + 0.5 Y \in \{0.25, 0.75\}$ On a calibration set $\{(x_i, y_i)\}_{i=1}^n$, compute: $p_i = f(x_i), \quad y_i' = 0.25 + 0.5 y_i$ with per-instance nonconformity score

$S(p_i, y_i') = \frac{(y_i' - p_i)^2}{\sigma(p_i)}$

where $\sigma(p)$ is a positive-valued, user-chosen or theoretically justified uncertainty-scaling function. The conformal quantile is

$$\hat q = \min\left\{q : \frac{1}{n}\sum_{i=1}^n \mathbf{1}\{S(p_i, y_i') \leq q\} \geq 1-\alpha \right\}$$

For a new input $x$, compute $p=f(x)$ and use the class-anchor

$$y'_{\rm test} = \begin{cases} 0.25 & p < 0.5 \ 0.75 & p \geq 0.5 \end{cases}$$

The Conformal-Calibrated Interval is

$$C(x) = \left\{\, p'\in[0,1] \,\middle|\, S(p', y'_{\rm test}) \leq \hat q \right\}$$

which, due to the construction, forms a closed interval

$$|p' - y'_{\rm test}| \leq \sqrt{\hat q \cdot \sigma(p')}$$

and numerically, the endpoints are

$$L(x) = \inf\{p' \mid |p' - y'_{\rm test}| \leq \sqrt{\hat q\,\sigma(p')}\}, \quad U(x) = \sup\{p' \mid |p' - y'_{\rm test}| \leq \sqrt{\hat q\,\sigma(p')}\}$$

The reported CCI for a test $x$ is $[L(x), U(x)]$ (Ugwu et al., 23 Nov 2025).

2. Theoretical Guarantees and Statistical Rationale

Under exchangeability (the joint law of calibration plus test is invariant to order), the quantile $\hat q$ ensures that for the calibration and new test example

$$\Pr\left\{ S\big(f(X_{\mathrm{test}}), Y'_{\mathrm{test}}\big) \leq \hat q \right\} \geq 1-\alpha$$

This yields a marginal coverage guarantee: $$\Pr\left\{ Y'_{\mathrm{test}} \in C(X_{\mathrm{test}}) \right\} \geq 1-\alpha$$ As $Y'\in\{0.25,0.75\}$, this means the true label-anchor for the test case lies in the CCI with the prescribed probability. The scaling function $\sigma(p)$ modulates the uncertainty inflation (the default linear form, $1 + (1 - |p - 0.5|)$, penalizes near-decision-boundary calibration), but the coverage guarantee holds for any positive $\sigma$ (Ugwu et al., 23 Nov 2025).

3. Calibration Procedure and Algorithm

The CCI construction is a special instance of full conformal prediction and can be implemented, as in (Ugwu et al., 23 Nov 2025), by the following pipeline:

Input: D_train, D_cal, classifier f, miscoverage α 1. Train f on D_train. 2. For each (x_i, y_i) in D_cal: p_i ← f(x_i) y_i' ← 0.25 + 0.5 · y_i σ_i ← σ(p_i) s_i ← (y_i' - p_i)^2 / σ_i 3. q̂ ← (⌈(n+1)(1-α)⌉)-th smallest of {s_i} 4. For a new test x: p̂ ← f(x) y' ← 0.25 + 0.5 · 1[p̂ ≥ 0.5] Solve |p - y'| ≤ sqrt(q̂ · σ(p)) over p ∈ [0,1] Output interval [L, U]

There is no requirement for regularization beyond the choice of $\sigma(p)$. Additional smoothing or randomized quantile tie-breaking can be used if desired (Ugwu et al., 23 Nov 2025).

4. Abstention and Decision Rule

Beyond outputting CIs, the CCI enables an abstention mechanism. Compute:

• If $L(x) \geq 0.5$: the interval lies entirely above $0.5$, so predict "UTI".
• If $U(x) < 0.5$: the interval lies entirely below $0.5$, so predict "No UTI".
• Otherwise: abstain (i.e., "I don't know").

This abstention capability provides an uncertainty-aware decision threshold, which is critical in clinical and safety-critical scenarios (Ugwu et al., 23 Nov 2025).

5. Trade-offs, Hyperparameters, and Extensions

• Significance level $\alpha$: Lower $\alpha$ yields wider intervals and higher coverage.
• Uncertainty-scaling $\sigma(p)$: Default is linear, but any positive function is permitted.
• Label-anchor mapping: Fixed at $\{0.25, 0.75\}$ but can be adapted if needed.
• Base classifier $f$: CCI is agnostic to $f$: logistic regression, neural nets, and other probabilistic classifiers can all be wrapped, yielding varying interval sharpness but identical coverage.
• Width control: Interval width is explicitly reportable and should be averaged empirically across test points for performance assessment (Ugwu et al., 23 Nov 2025).

6. Empirical Evaluation and Practical Impact

In real-world clinical deployment, the CCI-calibrated system achieved:

• Outperformance of baselines in recall and other classification metrics.
• The lowest observed abstention proportion and interval width among tested methods.
• Validation from a survey of 42 nurses, who found CCI outputs valuable for guiding clinical decision-making in early UTI detection.

This result demonstrates the critical practical utility of CCIs in operational decision support systems where both accurate prediction and explicit, actionable uncertainty quantification are essential (Ugwu et al., 23 Nov 2025).

7. Broader Context and Relationship to Conformal Approaches

The CCI as instantiated above is a conformal extension that applies to general probabilistic classifiers for classification with abstention. It is rooted in conformal prediction theory, which provides finite-sample, model-agnostic marginal and conditional coverage guarantees via exchangeability-based score quantile calibration. The use of a tunable uncertainty-scaling function, anchor mapping, and abstention logic distinguish the CCI framework from classical conformal prediction, enabling reliable, sharp, and uncertainty-aware predictions in demanding applications (Ugwu et al., 23 Nov 2025).