Spatially-Aware Evaluation of Segmentation Uncertainty (2506.16589v1)

Abstract: Uncertainty maps highlight unreliable regions in segmentation predictions. However, most uncertainty evaluation metrics treat voxels independently, ignoring spatial context and anatomical structure. As a result, they may assign identical scores to qualitatively distinct patterns (e.g., scattered vs. boundary-aligned uncertainty). We propose three spatially aware metrics that incorporate structural and boundary information and conduct a thorough validation on medical imaging data from the prostate zonal segmentation challenge within the Medical Segmentation Decathlon. Our results demonstrate improved alignment with clinically important factors and better discrimination between meaningful and spurious uncertainty patterns.

• The paper introduces three novel metrics (BUC, BA-ECE, SPACE) that quantify uncertainty by integrating spatial context with anatomical boundaries.
• It employs distance-based calibration and Gaussian smoothing to directly assess the alignment between predicted uncertainty and actual segmentation errors.
• Experiments on prostate MRI segmentation demonstrate significant accuracy and effect size improvements over traditional voxel-wise evaluation methods.

Spatially-Aware Evaluation of Segmentation Uncertainty: A Technical Synthesis

The paper "Spatially-Aware Evaluation of Segmentation Uncertainty" (2506.16589) addresses a critical limitation in the evaluation of segmentation uncertainty—namely, the inadequacy of metrics that ignore spatial context and anatomical structure. While prevalent uncertainty assessment metrics treat each voxel independently, this approach can obscure distinctions between clinically or operationally meaningful uncertainty (e.g., concentrated at structure boundaries) and spurious, spatially diffuse uncertainty. The authors offer three spatially contextualized metrics designed specifically for segmentation tasks, with comprehensive validation in the domain of prostate zonal segmentation.

Context and Shortcomings of Voxel-wise Metrics

The standard paradigm for segmentation uncertainty evaluation involves adapting prediction calibration or discrimination metrics from classification or detection to the voxel level. Examples include Expected Calibration Error (ECE), Area Under the ROC Curve (AUC-ROC), and Patch Accuracy versus Patch Uncertainty (PAvPU), among others. These approaches are spatially agnostic:

• Reliability metrics such as ECE aggregate error over all voxels, masking where the uncertainty appears.
• Discrimination metrics can rank which voxels are most associated with error, but do not penalize spatial incoherence (e.g., random speckles vs. structured boundary uncertainty).
• Selective prediction metrics such as AURC summarize the error-vs-coverage tradeoff, but similarly disregard spatial distribution.

This lack of spatial awareness is problematic for applications in medical imaging and safety-critical autonomous systems, where boundary-localized uncertainty is expected and clinically relevant.

Spatially-Aware Metrics: Definitions and Rationale

The paper introduces three novel metrics for segmentation uncertainty evaluation that explicitly incorporate spatial and boundary features:

1. Boundary Uncertainty Concentration (BUC):
   • Quantifies the degree to which high uncertainty localizes at predicted boundaries, compared to other regions. By computing the mean uncertainty within a narrow band around predicted boundaries versus the remaining region, BUC yields values near 1 when uncertainty is concentrated where errors commonly occur (i.e., at the boundaries).
   • Particularly relevant for clinical contexts, where boundary ambiguity is inherent.
2. Boundary-Aware Expected Calibration Error (BA-ECE):
   • Modifies calibration error by binning voxels not by predicted confidence, but by their signed distance to the ground-truth boundary. Within each band, the discrepancy between mean uncertainty and observed error forms a local calibration error, with closer-to-boundary bands weighted more.
   • Captures miscalibration specifically in regions of anatomical or operational importance.
3. Spatially-Aware Calibration Error (SPACE):
   • Evaluates the local spatial correspondence between predicted uncertainty and actual errors by smoothing both maps with a Gaussian kernel and then computing the average local absolute difference.
   • Low SPACE indicates that, in a neighborhood sense, uncertainty maps successfully align with actual segmentation failures.

Experimental Protocol and Quantitative Results

The metrics were assessed on prostate zonal segmentation using nnU-Net segmentations on a multi-parametric MRI dataset. Two sets of uncertainty maps per case were generated via Monte Carlo dropout: (a) high-quality, boundary-focused maps using adaptive dropout rates, and (b) low-quality, diffuse maps with fixed rates. Both map types were compared against actual errors (from full inference with no dropout) using spatially-aware and traditional metrics.

Key evaluation procedures included:

• Accuracy: Fraction of cases where a metric prefers the high-quality map.
• Effect Size (Cohen's d): Standardized difference between metrics on high- vs low-quality maps.
• Mean improvement: Relative performance gain of high-quality maps.

Notable results (see Table 1 of the paper):

• SPACE achieved 95.83% accuracy, Cohen's d of 1.34, and mean difference of 49.8%.
• BUC and BA-ECE achieved similarly high accuracies (91.67% and 89.58%), with robust effect sizes.
• All spatially-aware metrics significantly outperformed traditional metrics (e.g., ECE, AUC-ROC, PAvPU), both in accuracy and effect size; BA-ECE provided +6.25% higher accuracy, 68% higher effect size, and 48% higher mean improvement relative to ECE.

Pairwise McNemar tests with Holm correction established statistical significance of these improvements.

Practical and Theoretical Implications

The spatially-integrated metrics proposed directly address the operational need for meaningful uncertainty quantification in segmentation—particularly medically oriented tasks, where localized mispredictions drive downstream decisions. The demonstrated superiority of these metrics has several immediate implications:

• Segmentation Model Benchmarking: Model development efforts, especially in medical imaging, should adopt spatially-aware metrics alongside traditional ones for a more holistic understanding of uncertainty maps.
• Automated Quality Control: QA systems can better distinguish structured, interpretable uncertainty from random noise, providing more actionable flags for clinician review or model retraining.
• Uncertainty-Driven Model Selection: When multiple candidate models are available, these metrics can guide selection based not just on overall accuracy/calibration, but on reliability within clinically significant spatial regions.
• Design of Training Objectives: Integrating these metrics (e.g., as regularization terms) during model training could incentivize generation of uncertainty maps that are both discriminative and semantically meaningful.

Limitations and Future Directions

Although the presented evidence is robust for prostate segmentation, generalization to other anatomical domains, segmentation tasks, and imaging modalities remains incompletely assessed. For broader adoption:

• Further experiments should establish metric sensitivity and specificity across varying structure sizes, noise levels, and modality artifacts.
• Adaptive parameterization (e.g., Gaussian kernel bandwidth in SPACE, boundary width in BUC) may require context-specific tuning.
• Integration with interactive systems (e.g., semi-automated segmentation with uncertainty overlays) can provide direct user feedback for clinical validation.

Conclusion

Spatially-aware evaluation metrics represent a significant methodological advance in segmentation uncertainty assessment, capturing aspects overlooked by standard approaches. By quantifying not only the amount, but also the spatial organization of uncertainty with respect to anatomical features, these metrics provide essential tools for both researchers and practitioners committed to the trustworthy deployment of automated segmentation systems. Their adoption has the potential to refine model development cycles, improve clinical interpretability, and enable new avenues of research in uncertainty-informed decision support.