QU-BraTS Sub-Challenge Overview

• QU-BraTS Sub-Challenge is a framework that rigorously evaluates uncertainty quantification in deep learning-based brain tumor segmentation, addressing the clinical need for reliable predictions.
• Its evaluation metric integrates voxel-level Dice scores with FTP and FTN measures to balance segmentation accuracy and uncertainty filtering.
• The challenge employs multi-modal MRI data and a composite ranking system to drive improvements in uncertainty-aware neuro-oncological imaging workflows.

The QU-BraTS Sub-Challenge is a specialized task within the broader MICCAI BraTS (Brain Tumor Segmentation) challenge framework, established to rigorously evaluate, benchmark, and advance methods for quantifying uncertainty in deep learning-based brain tumor segmentation. This sub-challenge was motivated by the clinical imperative to complement increasingly accurate automated segmentation methods with robust estimates of prediction reliability, thereby facilitating clinically actionable deployment that allows expert clinicians to triage review efforts focusing on the most uncertain or error-prone regions.

1. Purpose and Motivation

The primary objective of the QU-BraTS sub-challenge is to provide a standardized, clinically relevant framework for the assessment and comparison of voxel-wise uncertainty estimates in state-of-the-art brain tumor segmentation algorithms. While the main BraTS challenge focuses on multi-compartment glioma segmentation (whole tumor—WT, tumor core—TC, enhancing tumor—ET) using multi-parametric MRI, the QU-BraTS extension specifically addresses the critical lack of visibility into model reliability. In clinical neuro-oncology workflows, a segmentation map that is “highly confident” when correct and “uncertain” when likely incorrect enables targeted review, accurate surgical/radiation planning, and mitigates the risk of silent, undetected errors (Mehta et al., 2021). The challenge’s rationale is rooted in the limitations of Dice-based ranking alone, as high overlap metrics do not reliably indicate the model’s ability to localize prediction errors or convey when human intervention is warranted.

2. QU-BraTS Evaluation Metric and Scoring System

The sub-challenge defines an explicit, multi-component uncertainty evaluation metric. For each tumor compartment $r$, participants are required to submit both a binary segmentation map $P^r$ and a normalized voxel-wise uncertainty map $U^r(x)\in[0,100]$. For a given uncertainty threshold $\tau$, voxels with uncertainties above $\tau$ are filtered (“marked uncertain”); scoring is then computed only on the remaining (“certain”) voxels.

Three primary curves are constructed as a function of the threshold $\tau$:

• Dice Score Curve $\mathrm{DSC}_r(\tau)$: Dice similarity coefficient on retained voxels.
• Filtered True Positive Ratio (FTP) $\mathrm{FTP}_r(\tau)$: Fraction of correct positive voxels erroneously filtered out.
• Filtered True Negative Ratio (FTN) $\mathrm{FTN}_r(\tau)$: Fraction of correct negative voxels erroneously filtered out.

Integrated AUCs over the threshold range quantify these curves:

$$\mathrm{Score}_r = \frac{1}{3} \Big( \mathrm{AUC}_1 + (1-\mathrm{AUC}_2) + (1-\mathrm{AUC}_3) \Big)$$

where $P^r$0 integrates Dice, $P^r$1 FTP, and $P^r$2 FTN (Mehta et al., 2021).

This composite metric rewards models that (i) maintain high segmentation accuracy on “certain” voxels, (ii) minimize under-confidence (i.e., avoid discarding too many correct predictions), and (iii) avoid trivial solutions (e.g., assigning blanket uncertainty to all voxels to boost Dice on the remainder).

3. Dataset and Submission Protocol

QU-BraTS 2020 used the BraTS dataset, comprising 369 training, 125 validation (withheld labels), and 166 test cases, each with four MRI modalities (T1, T1Gd, T2, FLAIR) and multilabel expert segmentations for WT, TC, and ET. Participants are provided only the images for validation and test submissions and must generate both the multi-class segmentation and per-entity [0,100] uncertainty maps. Evaluation is executed centrally on the CBICA IPP server (Mehta et al., 2021). Team rankings are determined for each test case/entity via the composite uncertainty score, and cumulative ranking is then computed over all entities and cases.

4. Results and Benchmark Analysis

Fourteen teams participated in QU-BraTS 2020, with the top-ranking methods notably distinct from the most accurate pure segmenters. The leading team (SCAN) achieved a normalized cumulative rank score (NRS) of approximately 0.14, with the next tier comprising UmU and DSI_Med (NRS ≈ 0.28) and QTIM (NRS ≈ 0.31). Lower-tier methods scored in the NRS range of 0.50-0.60. Notably, segmentation quality and uncertainty quality were found to be complementary but distinct; several teams with mid-range Dice scores for segmentation achieved high QU-BraTS ranks, whereas some high Dice performers ranked poorly for uncertainty. This orthogonality justifies the need for explicit, uncertainty-aware benchmarks in addition to standard segmentation evaluation (Mehta et al., 2021).

Breakdown by tumor entity showed that SCAN led across WT, TC, and ET. Inferior methods were frequently characterized by “over-filtering” (excessive fraction of correct voxels marked uncertain) or trivial uncertainty heuristics (e.g., 1–softmax).

5. Clinical Implications and Complementarity

The clinical motivation underlying the QU-BraTS task is to enable operational segmentation pipelines where the majority of confidently predicted voxels can be auto-accepted, focusing specialist attention only on voxels/sites where uncertainty is high. In high class-imbalance settings (e.g., small tumors in large healthy volumes), it is essential that uncertainty methods avoid flooding the background with spurious uncertainty, which defeats the triaging purpose. QU-BraTS thus bridges toward a practical workflow for neuro-oncological imaging, in which error-prone boundaries and rare mis-segmentations are prioritized for expert review (Mehta et al., 2021). The framework also penalizes models that falsely discard large numbers of correct segmentations, maintaining the overall efficiency of automated review.

6. Best Practices, Open Issues, and Recommendations

The sub-challenge findings emphasize several best-practice conclusions:

• Multi-component scoring is necessary: Reliance on Dice alone allows trivial solutions (all positives filtered out). Penalties for FTP and FTN are required to reflect clinically meaningful uncertainty.
• Threshold-averaged evaluation: Reporting uncertainty scores over a range of thresholds ($P^r$3) provides a more comprehensive assessment than single-point metrics.
• Statistical significance in ranking: With permutation testing (100,000 draws) to assess rank differences, ties are resolved only when differences in empirical cumulative scores are not statistically significant.
• Open methodological questions: Current metrics are best suited to sufficiently large volumes; more stable alternatives (e.g., AUCs of precision/recall) may be advantageous in low-volume tasks.
• Ground-truth uncertainty: The acquisition of explicit rater-annotated uncertainty, or ensemble labels highlighting areas of low confidence, remains an open area for future challenges (Mehta et al., 2021).

A plausible implication is that future extensions may involve multi-class uncertainty quantification and spatially correlated uncertainty approaches to further align with clinical review strategies.

7. Integration with Broader BraTS Program and Impact

QU-BraTS sits alongside other specialized sub-challenges (e.g., missing modality, domain shift, inpainting) in the BraTS suite, each targeting a specific obstacle to robust clinical deployment of medical imaging AI. Its composite scoring methodology and task-specific benchmarking provide a reference for uncertainty quantification across the neuroimaging domain, fostering reproducibility and enabling rigorous comparison between algorithms. The public release of code and test protocols further enhances the reproducibility and extensibility of the evaluation framework (Mehta et al., 2021). The overarching impact is to facilitate clinician trust and more efficient, risk-aware deployment of AI tools in neuro-oncology care.