Confidence intervals uncovered: Are we ready for real-world medical imaging AI? (2409.17763v2)

Abstract: Medical imaging is spearheading the AI transformation of healthcare. Performance reporting is key to determine which methods should be translated into clinical practice. Frequently, broad conclusions are simply derived from mean performance values. In this paper, we argue that this common practice is often a misleading simplification as it ignores performance variability. Our contribution is threefold. (1) Analyzing all MICCAI segmentation papers (n = 221) published in 2023, we first observe that more than 50% of papers do not assess performance variability at all. Moreover, only one (0.5%) paper reported confidence intervals (CIs) for model performance. (2) To address the reporting bottleneck, we show that the unreported standard deviation (SD) in segmentation papers can be approximated by a second-order polynomial function of the mean Dice similarity coefficient (DSC). Based on external validation data from 56 previous MICCAI challenges, we demonstrate that this approximation can accurately reconstruct the CI of a method using information provided in publications. (3) Finally, we reconstructed 95% CIs around the mean DSC of MICCAI 2023 segmentation papers. The median CI width was 0.03 which is three times larger than the median performance gap between the first and second ranked method. For more than 60% of papers, the mean performance of the second-ranked method was within the CI of the first-ranked method. We conclude that current publications typically do not provide sufficient evidence to support which models could potentially be translated into clinical practice.

• The paper presents a systematic review of MICCAI segmentation studies, showing over 50% omit reporting performance variability.
• The authors develop a method to approximate unreported standard deviation from mean Dice coefficients and validate it with challenge data.
• The study reconstructs 95% confidence intervals and finds that many performance gains are not statistically significant.

Confidence Intervals Uncovered: Are We Ready for Real-World Medical Imaging AI?

"Confidence intervals uncovered: Are we ready for real-world medical imaging AI?" authored by Evangelia Christodoulou et al. scrutinizes the prevalent practices in reporting AI performance metrics within the domain of medical image analysis. The paper argues that the common reliance on mean performance values as a sole metric in scientific publications is a misleading simplification that undermines the efficacy and reliability of these models for real-world clinical applications.

Key Contributions

The paper offers three primary contributions:

1. Systematic Review of MICCAI 2023 Segmentation Papers: The authors conducted a comprehensive analysis of 221 segmentation papers published in MICCAI 2023. They found that more than 50% of these papers do not assess performance variability, and only one paper reported confidence intervals (CIs) for model performance.
2. Approximation of Unreported Standard Deviation (SD): Using data from the Medical Segmentation Decathlon challenge, the authors proposed a method to approximate the unreported SD in segmentation papers as a function of the mean Dice similarity coefficient (DSC). This approximation was then validated using data from 56 previous MICCAI challenges, demonstrating that the method can accurately reconstruct CIs using information available in publications.
3. Reconstruction of CIs for MICCAI 2023 Segmentation Papers: The authors reconstructed 95% CIs around the mean DSC for MICCAI 2023 segmentation papers. They found that the median CI width is three times larger than the median performance gap between the first and second-ranked methods. Furthermore, for over 60% of the papers, the mean performance of the second-ranked method was within the CI of the first-ranked method.

Methodology

The methodology is split into two main sections:

1. Systematic Review: The authors systematically reviewed all segmentation papers from MICCAI 2023, focusing on how performance variability was reported. They found that most papers either did not report variability at all or provided insufficient details about the method of computing reported SDs.
2. Data Approximation and Reconstruction: To address the lack of reported variability, the authors approximated the SD using a second-order polynomial function of the mean DSC. They then reconstructed CIs using this approximation and compared them to actual SDs computed from previous challenge data, which validated their approach.

Key Results

The results from the systematic review revealed that the current practice in the medical image analysis community often overlooks performance variability. Specifically:

• Quantitative Findings: Of the papers reviewed, 54.8% did not report any form of variability, and only 0.5% reported CIs. Furthermore, 83.3% of the papers claimed to outperform the state-of-the-art methods without providing sufficient evidence through variability reporting.
• Implications of CI Widths: The median CI width was 0.03, substantially larger than the median performance gap of 0.01 between the top two ranked methods. This suggests that the claimed performance improvements are not statistically significant in most cases.

Implications and Future Directions

The findings have significant implications for the clinical translation of AI models in medical imaging. The lack of variability reporting and the reliance on mean performance values can lead to misleading conclusions about a model's reliability. This is particularly critical in healthcare applications where model performance variability can have substantial impacts on patient outcomes.

Future Research: The paper advocates for a shift towards more robust and transparent reporting practices, emphasizing the importance of CIs and other variability measures. It also suggests adopting statistical frameworks such as superiority margins from clinical trials to assess the clinical relevance of reported performance gains.

Community Standards: The authors call for the medical image analysis community to align its reporting practices with guidelines that emphasize variability reporting. This includes adapting reproducibility checklists and fostering norms that require the inclusion of CIs in performance reports.

Conclusion

The analysis presented in "Confidence intervals uncovered" highlights a crucial oversight in the current reporting practices within the medical imaging community. By demonstrating the limited use of variability metrics and proposing a method to approximate these missing values, the authors provide a pathway towards more reliable and clinically translatable AI models. This paper serves as a pivotal call to action for the community to adopt more rigorous and transparent performance assessment methodologies.