Reliability analysis for BraTS-GoAT segmentation: a controlled robustness study of deep-ensemble uncertainty

Abstract: Deep networks segment brain tumours accurately in-distribution, but can fail silently when the input differs from their training data. That risk is central to clinical deployment and is the premise of the BraTS-GoAT generalizability task. We ask not only how well a model segments, but whether its uncertainty knows when it is wrong. On BraTS-GoAT (Task 3) we train a 5-fold cross-validated nnU-Net baseline (one held-out prediction per case) and a 3-seed deep ensemble. Both are evaluated for calibration and error detection on a per-region relevant mask, aggregated per case. In-distribution the 3-seed ensemble improves modestly over the already strong single model on the same held-out split, with the clearest gain in calibration. The separation appears under shift. In a controlled robustness study using graded synthetic corruptions as a proxy for acquisition shift, the single model's confidence stays flat while its accuracy and calibration degrade. Inter-member disagreement instead rises steeply, about a quarter to a third above the clean condition, several times the single model's response. On the official validation leaderboard the 5-fold ensemble of those folds attains whole-tumour Dice 0.87. The generalization gap is concentrated on the harder regions, with a characteristic failure of missing small, satellite lesions on unseen cohorts. In the synthetic study, disagreement among the 3-seed members is a more sensitive case-level indicator of acquisition shift than single-model confidence. Its per-voxel error localisation weakens as severity grows. The contribution is a rigorous, honest reliability comparison rather than a claim that any one uncertainty method dominates.

• The paper shows that a 3-seed deep ensemble modestly improves calibration, reducing relevant-mask ECE by roughly 0.010–0.011 while producing only small Dice and AUROC gains over a single nnU-Net.
• The paper finds that inter-member disagreement is substantially more responsive to synthetic acquisition shifts than single-model confidence, rising about 23% under severe bias-field corruption and 31% under severe blur.
• The paper demonstrates that external performance declines are driven mainly by missed small satellite lesions and catastrophic boundary-error tails, while cautioning that synthetic corruptions cannot replace validation on genuinely unseen cohorts.

Overview

This paper presents a reliability analysis of deep-ensemble uncertainty for brain tumour segmentation on the BraTS-GoAT generalizability challenge (Task 3). Rather than pursuing architectural novelty, the authors treat nnU-Net as a fixed, competitive substrate and direct their effort at a question central to clinical deployment: does a model's uncertainty know when it is wrong, particularly when acquisition conditions depart from training? The study pairs a 5-fold cross-validated single nnU-Net (ResEnc-L) with a 3-seed deep ensemble on identical held-out data, evaluates calibration and error detection on per-region relevant masks, runs a controlled robustness study with graded synthetic corruptions as a proxy for acquisition shift, and anchors external performance on the official validation leaderboard. The stated contribution is deliberately framed as an honest comparison rather than a claim that any one uncertainty method dominates.

Data, baseline, and experimental design

The task uses 1351 labelled BraTS-GoAT training cases with four co-registered MRI sequences at $1\,\mathrm{mm}^3$ resolution, evaluated on the nested whole tumour (WT), tumour core (TC), and enhancing tumour (ET) regions. Notably, about 2.4% of cases contain no enhancing tumour, where over-segmentation is heavily penalised; these cases turn out to be salient for error analysis. Training is from scratch under GoAT rules, with no external data or pretrained weights.

The segmentation baseline is nnU-Net v2.8.0 in its ResEnc-L 3d_fullres configuration, trained 1000 epochs per fold (~29 h per run on an A100) with stock settings. Pooling the five held-out folds yields one prediction per case from a model that did not train on it. The same five networks are averaged at inference to form the 5-fold challenge ensemble used only for leaderboard submission — the paper is careful to distinguish these two configurations, since the fold models' overlapping training sets make the 5-fold ensemble unsuitable for label-referenced uncertainty.

The uncertainty method is a 3-seed deep ensemble trained on a single fixed split, whose members' disagreement is therefore a clean, label-referenced signal on that split's 271 held-out cases. The paired baseline is the fold-0 network on the same cases. Two signals are compared: maximum-softmax confidence (available to both models) and inter-member predictive variance (ensemble only). A methodological strength is the evaluation protocol: metrics are computed on a lightly dilated union of predicted and reference region (the "relevant mask"), which removes class-imbalance dilution that renders all-brain ECE uninformative, and aggregated per case to avoid a Simpson's-paradox inversion of AUROC.

In-distribution results

The cross-validated single model is strong: mean Dice 0.925/0.909/0.869 (WT/TC/ET), with medians well above means (e.g., median HD95 near 1 mm against means of 6–11 mm). The gap between medians and means reflects a hard-case tail concentrated in the 33 no-ET cases, where the model hallucinates small enhancing regions; excluding them drops mean ET HD95 from 10.73 mm to 5.62 mm.

On the relevant mask, the single model is overconfident in every region (per-case ECE 0.070–0.080), yet its confidence remains a usable error detector, with AUROC 0.857–0.875 exceeding chance in 98–99% of cases. This overconfidence-with-usable-ranking profile defines the gap an ensemble should close.

The paired comparison shows the 3-seed ensemble significantly better on almost every metric ($p<0.001$, Wilcoxon signed-rank), but effect sizes are small. The clearest gain is calibration: pooled ECE falls by roughly 0.010–0.011 in every region. Dice improves by only 0.003–0.007 and detection AUROC by 0.004–0.009. The boundary effect is region-specific and non-monotone: ET HD95 improves significantly (13.59 → 12.14 mm) while TC HD95 worsens (3.14 → 4.27 mm), with WT unchanged. The authors offer a plausible but untested mechanism — averaging pulls in far outliers while smoothing boundaries that were already tight — and correctly flag it as such. The honest summary is that in-distribution, the ensemble is a modest, calibration-focused improvement, echoing QU-BraTS's conclusion that no single uncertainty method dominates.

Robustness under synthetic shift

The controlled robustness study degrades validation images with four MRI-realistic corruptions (Gaussian noise, bias field, blur, gamma) at graded severity inside the brain mask, before nnU-Net's normalisation, so the model sees genuinely degraded input. Segmentation quality degrades gracefully — bias field and blur bite most, dropping region-averaged Dice from 0.900 to 0.882 under severe bias — while noise and gamma barely register.

The informative result concerns the uncertainty signals. Under severe bias, the single model's mean uncertainty rises only ~5% while its ECE climbs ~25% (0.077 → 0.096): it becomes worse and more miscalibrated while remaining equally confident — precisely the silent-failure signature the paper targets. Inter-member disagreement instead rises steeply, approximately +23% under severe bias and +31% under severe blur, several times the single model's response. The sensitivity ordering on identical data is disagreement ≫ 3-seed confidence > single confidence. However, disagreement's per-voxel error localisation weakens as severity grows, so its value is as a case-level shift indicator rather than a voxel-level localiser. An important caveat, which the authors state plainly: this is a proxy for acquisition shift, not a substitute for validation on genuinely unseen cohorts.

External leaderboard evaluation

On the official validation set (451 cases submitted, 450 scored), the 5-fold ensemble attains mean per-case Dice 0.875 (WT), 0.814 (TC), and 0.782 (ET), with bootstrap intervals roughly ±0.02. Mean HD95 is high (17.1/25.0/40.0 mm) against medians of 2.2/2.0/1.4 mm, and whole-tumour HD95 spans 11.5–23.6 mm under bootstrap resampling while its median stays near 2.3 mm — evidence that the catastrophic tail, not typical performance, drives the boundary statistics.

The Dice gap against internal numbers (0.050/0.095/0.087) exceeds the interval half-widths by two to four times, so it is not a sampling artefact. The authors are appropriately cautious here: the internal estimate uses the cross-validated single model with test-time mirroring enabled, while the leaderboard uses the 5-fold ensemble with mirroring disabled, so the gap is an external anchor consistent with the synthetic robustness trend but not protocol-matched. Lesion-wise analysis attributes the decline to under-detection of small satellite lesions rather than uniform degradation — errors dominated by omission (mean false negatives 1.6–2.5 per case versus false positives 0.3–1.1), i.e., the model misses tumour quietly.

Limitations and open questions

The paper is explicit about its bounds. The robustness study relies on synthetic corruptions with severity capped by the authors' own judgement and a single fixed random realisation per condition. The ensemble-versus-single comparison rests on one held-out split, though per-fold stability checks show calibration (ECE 0.0697±0.0028 WT) and detection AUROC stable across all five folds; notably, only "ET has the lowest Dice and detection AUROC" survives all five folds, so cross-region orderings should be treated cautiously. Disagreement is inherently small in magnitude, limiting its use to relative change rather than absolute thresholds. The mechanism behind the asymmetric HD95 effects was not tested. Open questions left by the paper include how MC-dropout and TTA compare as cheaper estimators, whether the observed trends hold on real external cohorts, and how pseudo-label noise from semi-supervised extensions would affect calibration.

Conclusion

This work delivers a disciplined reliability assessment showing that a single nnU-Net is already a strong in-distribution reliability baseline, that a 3-seed deep ensemble adds mainly calibration gains, and that inter-member disagreement is markedly more sensitive than single-model confidence to corruption-induced shift — the regime where silent failure matters most. The combination of paired comparisons, relevant-mask metric computation, and candid reporting of regressions makes the analysis a useful template for uncertainty evaluation in medical segmentation, even as its conclusions remain bounded by synthetic proxies and a single-split design.