Making Conformal Predictors Robust in Healthcare Settings: a Case Study on EEG Classification

Abstract: Quantifying uncertainty in clinical predictions is critical for high-stakes diagnosis tasks. Conformal prediction offers a principled approach by providing prediction sets with theoretical coverage guarantees. However, in practice, patient distribution shifts violate the i.i.d. assumptions underlying standard conformal methods, leading to poor coverage in healthcare settings. In this work, we evaluate several conformal prediction approaches on EEG seizure classification, a task with known distribution shift challenges and label uncertainty. We demonstrate that personalized calibration strategies can improve coverage by over 20 percentage points while maintaining comparable prediction set sizes. Our implementation is available via PyHealth, an open-source healthcare AI framework: https://github.com/sunlabuiuc/PyHealth.

• The paper demonstrates that personalized conformal prediction improves EEG classification coverage by up to 25% compared to traditional methods.
• It shows that neighborhood calibration efficiently addresses distribution shifts and label noise in patient-specific EEG data.
• Empirical results confirm that instance-level calibration maintains compact prediction sets while closely matching nominal coverage in clinical settings.

Robust Conformal Prediction for EEG Signal Classification Under Distribution Shift

Introduction

Uncertainty quantification remains a crucial requirement for clinical machine learning tasks, particularly in high-stakes diagnostics such as EEG classification. Conformal Prediction (CP) frameworks provide finite-sample prediction set guarantees under minimal distributional assumptions. However, the i.i.d. assumption underlying standard CP is routinely violated in the healthcare context due to patient heterogeneity, distributional shifts, and pervasive annotation noise. This study systematically evaluates conformal approaches for EEG seizure classification, focusing on methods robust to patient-driven shifts and label uncertainty. The central contribution is evidence—both theoretical and empirical—that personalized conformal prediction substantially improves coverage compared to traditional conformal methods, without inflating the size of prediction sets.

Distributional challenges in clinical EEG analysis are highlighted, where noisy multi-annotator labels and cohort-induced dataset shift invalidate simple i.i.d.-calibrated uncertainty quantification.

Figure 1: EEG classification presents i.i.d. violations and annotation uncertainty, resulting in unreliable label ground truth and significant feature distribution shift.

Conformal Prediction Framework and Neighborhood Calibration

Standard CP produces marginally valid prediction sets using a calibration distribution presumed to be drawn from the same process as future test samples. In EEG and broader healthcare, this assumption fails due to strong inter-patient and acquisition heterogeneity. This work analyzes four variants of CP:

• Naive CP: Classic split CP, calibrating prediction sets globally.
• Covariate CP: CP augmented with density-ratio–based weights to mitigate covariate shift, following [tibshirani2019conformal].
• K-means CP: Calibrates within patient-assigned clusters, partially personalizing the calibration.
• Neighborhood CP (NCP): Employs k-nearest neighbor calibration in embedding space, providing adaptive, patient-specific set sizes and thresholds as per [ghosh2023ncp].

NCP constructs test-sample-adaptive calibration distributions by leveraging learned representations $\Phi(x)$, addressing local heterogeneity in conditional label distributions unaccounted for by global calibration.

Theoretical Analysis: Covariate Shift and Coverage Error

Formally, under covariate shift (where $P_{\mathrm{test}}(Y|X) = P_{\mathrm{cal}}(Y|X)$ but $P_{\mathrm{test}}(X) \neq P_{\mathrm{cal}}(X)$), the marginal coverage guarantee of standard CP is generally violated. The authors present a novel decomposition of the coverage error:

$$P_{\mathrm{test}}(Y \in \widehat{C}(X)) - (1-\alpha) = \mathrm{Cov}_{P_{\mathrm{cal}}}(w(X), s(X))$$

where $w(X)$ is the test-over-calibration density ratio and $s(X)$ is the slack in conditional coverage. This indicates that the sensitivity of marginal coverage to distribution shift is governed by the covariance between the shift and local calibration slack. Reducing the variance of $s(X)$, as achieved by NCP, thus directly reduces worst-case error under shift. Unlike density ratio reweighting, which is unreliable under high-dimensional feature spaces, local CP adaptation targets slack variability without explicit density estimation.

Experimental Evaluation

Experiments are conducted on the TUAB (binary abnormal/normal) and TUEV (6-class event detection) benchmarks, including over 500,000 labeled EEG segments annotated by expert consensus. All models employ ContraWR [yang2023selfsupervisedeegrepresentationlearning_contrawr] as the base encoder.

Empirical results compare the different CP strategies in terms of actual coverage attained at varying miscoverage levels $\alpha$ and corresponding prediction set sizes.

Figure 2: Personalized conformal predictors (NCP, K-means CP) yield markedly higher empirical coverage than global approaches, particularly for larger $\alpha$; only NCP approaches target coverage at high $\alpha$.

Contrary to prior results in simpler domains, Covariate CP does not outperform Naive CP. The failure of kernel density estimation to provide meaningful weights in high-dimensional EEG representation spaces highlights the practical limitations of explicit covariate shift correction. Instead, personalization—whether via cluster-level or, preferably, sample-level calibration—offers robust improvement. NCP achieves up to 25% higher empirical coverage than Naive CP at $\alpha=0.2$ without significant increase in prediction set size.

Figure 3: NCP and K-means CP maintain compact prediction sets for lower $\alpha$, increasing only modestly when higher coverage is demanded.

Critically, NCP narrows the gap between empirical and nominal coverage across nearly all $\alpha$, though it still exhibits shortfall at extreme low $\alpha$ (e.g., $\alpha=0.01$ relevant for highest-risk scenarios), reflecting ongoing challenges for conformal reliability in safety-critical applications.

Implications and Future Work

This study demonstrates empirically and theoretically that personalized calibration via neighborhood CP solves core practical failure modes of standard CP under real-world healthcare distribution shifts. This finding carries important implications:

• Practical Deployment: Achieving coverage closer to nominal rates with modest prediction set sizes enables more trustworthy model integration in clinical workflows involving diagnostic EEG.
• Limits of Density Weighting: Density-ratio reweighting via kernel density estimation is not a viable fix in high-dimensional, structured signal modalities such as EEG.
• Generalization to Other Modalities: Domain generalization research in healthcare should focus on adaptive, instance-level calibration in uncertainty quantification, not merely population-level adjustments.

Future directions include integrating more advanced representation learning architectures (e.g., motif-based EEG tokenizers [pradeepkumar2026tokenizingsinglechanneleegtimefrequency]) to further increase neighborhood relevance and guarantee coverage in the low-$\alpha$ regime, as well as comprehensive extension of personalized CP to broader healthcare domains with similar heterogeneity and label reliability constraints.

Conclusion

The paper provides a rigorous assessment of conformal prediction methods under distributional shift for clinical EEG signal classification. By introducing theoretical coverage-error decompositions and empirically validating the superiority of neighborhood-based calibration, the work establishes personalized conformal prediction as an indispensable tool for robust, uncertainty-aware clinical ML. Persistent challenges regarding ultra-high-confidence coverage and extension to other data types will inform new research at the intersection of uncertainty quantification, domain generalization, and healthcare AI.