Delving Aleatoric Uncertainty in Medical Image Segmentation via Vision Foundation Models

Abstract: Medical image segmentation supports clinical workflows by precisely delineating anatomical structures and lesions. However, medical image datasets medical image datasets suffer from acquisition noise and annotation ambiguity, causing pervasive data uncertainty that substantially undermines model robustness. Existing research focuses primarily on model architectural improvements and predictive reliability estimation, while systematic exploration of the intrinsic data uncertainty remains insufficient. To address this gap, this work proposes leveraging the universal representation capabilities of visual foundation models to estimate inherent data uncertainty. Specifically, we analyze the feature diversity of the model's decoded representations and quantify their singular value energy to define the semantic perception scale for each class, thereby measuring sample difficulty and aleatoric uncertainty. Based on this foundation, we design two uncertainty-driven application strategies: (1) the aleatoric uncertainty-aware data filtering mechanism to eliminate potentially noisy samples and enhance model learning quality; (2) the dynamic uncertainty-aware optimization strategy that adaptively adjusts class-specific loss weights during training based on the semantic perception scale, combined with a label denoising mechanism to improve training stability. Experimental results on five public datasets encompassing CT and MRI modalities and involving multi-organ and tumor segmentation tasks demonstrate that our method achieves significant and robust performance improvements across various mainstream network architectures, revealing the broad application potential of aleatoric uncertainty in medical image understanding and segmentation tasks.

• The paper presents a novel framework leveraging singular value decomposition of foundation model features to quantify intrinsic aleatoric uncertainty.
• It demonstrates improved segmentation results by filtering high-uncertainty samples and applying dynamic uncertainty-aware optimization across multiple benchmarks.
• The approach enables annotation-efficient training with robust performance on diverse architectures and imaging modalities.

Aleatoric Uncertainty Quantification in Medical Image Segmentation with Vision Foundation Models

Introduction

The paper "Delving Aleatoric Uncertainty in Medical Image Segmentation via Vision Foundation Models" (2604.10963) proposes a novel framework for quantifying and leveraging aleatoric uncertainty in medical image segmentation, capitalizing on the universal feature representations of large-scale vision foundation models (VFMs). In contrast to the prevailing focus on model-based uncertainty and architectural design, this work systematically addresses the intrinsic data-level uncertainty, specifically aleatoric uncertainty, which arises from the stochasticity and annotation noise inherent to medical datasets. The proposed paradigm encompasses both analytic quantification via singular value energy spectra and practical mechanisms for robust training, aiming to enhance segmentation performance and generalizability across diverse modalities and annotation regimes.

Quantification of Aleatoric Uncertainty via Semantic Perception Scale

The central innovation is the introduction of an annotation-free, representation-driven metric for per-sample difficulty and inherent uncertainty. Specifically, multi-channel feature vectors are extracted from medical VFMs (e.g., MedSAM2) following the input of image volume $X$. The structural complexity of these feature vectors is characterized via singular value decomposition (SVD), where the decay pattern and energy distribution across singular values reflect the variability and informativeness in the latent code. The normalized Shannon entropy of the squared singular value distribution yields the semantic perception scale (SPS), acting as a surrogate for feature diversity. Linear and logarithmic normalization ultimately map this SPS to a bounded Aleatoric Uncertainty Value (AUV), interpretable as the sample’s intrinsic ambiguity.

The empirical behavior of AUV is visualized in training images of varying annotation quality.

Figure 1: Training samples ranked by MedSAM2-derived AUV, with lower values corresponding to easier, less ambiguous samples.

Singular value decay curves substantiate the relationship between SPS and sample classification: rapid decay indicates compactness and lower uncertainty, while slower decay corresponds to richer, more robust feature representations in ambiguous cases.

Figure 2: Singular value decay and cumulative energy plots illustrating feature space disparities across samples of different difficulty.

Following this quantification, the pipeline incorporates both inference and utility phases—quantifying AUV, then using these estimates for training data filtering and loss reweighting.

Figure 3: Overview of the framework with aleatoric uncertainty quantification and its applications in data filtering and uncertainty-aware optimization.

Aleatoric Uncertainty-Driven Applications

Data Filtering Strategies

The first application utilizes AUV to systematically eliminate potentially detrimental, noisy training examples. Various strategies are considered, such as filtering based on global AUV, class-wise AUV, and differing image normalization inputs, with filtering thresholds chosen via empirical quantiles (e.g., removing the top 5–10% most uncertain samples). Removal criteria can be class-agnostic or class-specific, particularly relevant in tumor-localization tasks.

The effectiveness of these mechanisms is verified by histogram and distribution analysis of AUV across different filtering regimes.

Figure 4: Class-wise and whole-sample AUV histograms under multiple input normalizations.

Experimental results demonstrate that data filtering predicated on VFM-derived AUV outperforms baselines utilizing simple data variance or task-specific feature extractors, with MedSAM2-based quantification yielding the most consistent Dice/mIoU improvement across CT and MRI benchmarks.

Dynamic Uncertainty-Aware Optimization

The second pillar is the introduction of dynamic uncertainty-aware optimization (DUO), which adaptively modulates training loss contributions using the SPS. Loss weights for each semantic class are adjusted in proportion to uncertainty, and a dedicated label noise denoising module is integrated to address annotation artifacts. This flexible, plug-and-play optimization strategy is validated across model architectures based on CNN, Transformer, and Mamba, and robustly increases boundary accuracy and local Dice scores.

Boundary precision gains are evident when visualizing segmentation overlays before and after DUO adoption.

Figure 5: Comparison of segmentation results using baseline and DUO strategies, with clear improvements at ambiguous boundaries.

Training curve analysis covers the stabilization and convergence properties imparted by DUO.

Figure 6: Validation loss and Dice evolution during dynamic uncertainty-aware optimization training; DUO yields faster convergence and higher accuracy.

Empirical Analysis and Theoretical Insights

The methodology is systematically evaluated on five public 3D medical image segmentation datasets, including LiTS, TotalSegmentator, WORD, FeTA, and KiTS23, covering both organ and lesion segmentation tasks. Key findings include:

• Filtering the top 5–10% most-uncertain samples (as measured by MedSAM2-AUV) leads to mean Dice increases of up to 1.5–1.8 points relative to raw baselines and much larger improvements in tumor-specific classes.
• DUO yields stable ~0.5–1.0 Dice improvements across all baseline architectures, with only minimal computational/memory overhead.
• A comprehensive ablation ablates reliance on annotation, class-specific versus all-class filtering, and compares to alternative uncertainty quantification methods such as Mahalanobis distance and Fisher metric. MedSAM2-SPS/AUV achieves the highest Pearson/Spearman correlation with out-of-sample Dice, indicating high predictive fidelity for error localization.

Feature-space visualization using t-SNE confirms that MedSAM2’s predicted uncertainty values align with empirical sample difficulty, especially in mis-annotated or outlier tumor cases.

Figure 7: t-SNE visualization connecting predicted feature vectors and AUV, showing model focus divergence on ambiguous samples.

Furthermore, the most performant data filtering leverages normalized images combined with class-specific evaluation, but reasonable improvements are also obtainable using raw (unaligned) images, enabling annotation-light deployment.

Practical and Theoretical Implications

Integrating aleatoric uncertainty quantification at the data level enables the construction of robust, annotation-efficient medical image models. From a practical standpoint, filtering uncertain samples and adjusting optimization objectives according to intrinsic sample ambiguity directly improves out-of-distribution (OOD) robustness and reduces overfitting to annotation noise. The plug-and-play design ensures facile extension to existing pipelines, agnostic to model backbone or imaging modality.

Theoretically, the use of singular value structures in semantic feature spaces anchors uncertainty quantification in a robust, model-agnostic latent representation, leveraging the high-dimensional geometry and stability of foundation models. This aligns the statistical notion of sample uncertainty with model cognitive bias, a key advance over task-bound or probabilistic methods requiring repeated sampling or architectural changes.

Future directions include extending these quantification frameworks to weak supervision, federated settings, and online sample weighting in dynamic annotation regimes. The generality and efficacy of the proposed quantification suggest promise for broader deployment in reliable medical-AI workflow design.

Conclusion

This work provides a rigorous, annotation-independent methodology for quantifying and utilising aleatoric uncertainty in medical image segmentation by exploiting the representation space of vision foundation models. The framework demonstrates significant increases in segmentation accuracy and robustness through principled data curation and adaptive optimization, with compelling generality across tasks, organs, and architectures. These findings have important implications for the development of trustworthy, uncertainty-aware clinical decision support systems and reveal new avenues for principled model-dataset co-design.