DeferredSeg: A Multi-Expert Deferral Framework for Trustworthy Medical Image Segmentation

Abstract: Segmentation models based on deep neural networks demonstrate strong generalization for medical image segmentation. However, they often exhibit overconfidence or underconfidence, leading to unreliable confidence scores for segmentation masks, especially in ambiguous regions. This undermines the trustworthiness required for clinical deployment. Motivated by the learning-to-defer (L2D) paradigm, we introduce DeferredSeg, a deferral-aware segmentation framework, i.e., a Human--AI collaboration system that determines whether to defer predictions to human experts in specific regions. DeferredSeg extends the base segmentor with an aggregated deferral predictor and additional routing channels that dynamically route each pixel to either the base segmentor or a human expert. To train this routing efficiently, we introduce a pixel-wise surrogate collaboration loss that supervises deferral decisions. In addition, to preserve spatial coherence within deferral regions, we propose a spatial-coherence loss that enforces smooth deferral masks, thereby enhancing reliability. Beyond single-expert deferral, we further extend the framework to a multi-expert setting by introducing multiple discrepancy experts for collaborative decision-making. To prevent overloading or underutilizing individual experts, we further design a load-balancing penalty that evenly distributes workload across expert branches. We evaluate DeferredSeg on three challenging medical datasets using MedSAM and CENet as the base segmentor for fair comparison. Experimental results show that DeferredSeg consistently outperforms the baseline, demonstrating its effectiveness for trustworthy dense medical segmentation. Moreover, the proposed framework is model-agnostic and can be readily applied to other segmentation architectures.

• The paper introduces a pixel-wise deferral mechanism that dynamically routes uncertain regions to human experts, significantly improving segmentation accuracy.
• It integrates spatial coherence, surrogate collaboration, and load-balancing to optimize expert utilization within critical medical applications.
• Quantitative results on synthetic and real-expert benchmarks show marked improvements in Dice and Jaccard scores, underscoring its clinical viability.

Pixel-Wise Deferral in Medical Image Segmentation: The DeferredSeg Framework

Introduction and Motivation

Medical image segmentation underpins critical applications such as diagnosis, treatment planning, and intraoperative navigation. While deep learning-based segmentors like U-Net and MedSAM offer impressive generalization, they are prone to unreliable confidence calibration, particularly in ambiguous or low-contrast regions. Such over- or under-confidence undermines deployment for high-stakes clinical workflows, especially where local uncertainty at anatomical boundaries may induce critical segmentation errors.

Traditional learning-to-defer (L2D) frameworks allow for instance-level triage to human experts when model confidence is low, primarily in classification. This paradigm is insufficient for dense, pixel-level prediction tasks, where spatial heterogeneity in uncertainty is common. “DeferredSeg: A Multi-Expert Deferral Framework for Trustworthy Medical Image Segmentation” (2604.12411) formalizes and addresses this gap by introducing DeferredSeg, a pixel-level, multi-expert deferral mechanism tightly integrated with state-of-the-art segmentation architectures.

Figure 1: DeferredSeg enables spatially adaptive Human-AI collaboration by routing ambiguous image regions (as determined by a pixel-wise deferral map) between AI and expert annotators.

DeferredSeg Framework: Architecture and Methodology

DeferredSeg augments a base segmentation model with key tailorable mechanisms:

1. Pixel-wise Deferral Predictor: Outputs a spatial deferral map, routing each pixel either to the model or to expert(s).
2. Surrogate Collaboration Loss: A softmax-based, differentiable loss supervising both model and expert routes, incorporating a consistent pixel-wise extension of L2D objectives.
3. Spatial Coherence Loss: Enforces spatial smoothness in the deferral map to provide contiguous, interpretable routing regions, mitigating high-frequency, non-local routing that plagues ViT-based decoders.
4. Load-Balancing Penalty: Explicit regularization ensures equitable workload allocation in multi-expert scenarios, preventing collapse to a single over-utilized or under-utilized expert.

Integration with MedSAM extends the frozen encoder with routing tokens and a CNN-based deferral predictor. Decoder tokens are resized from three (MedSAM default) to $3+J+1$ for $J$ experts, with fine-tuning restricted to the segmentation and routing heads for computational and data efficiency.

Figure 2: DeferredSeg pipeline overlays MedSAM, extending its decoder with routing heads and an aggregated deferral predictor for pixel-wise human-AI triage.

Quantitative Performance and Comparative Analysis

Evaluation spans three synthetic-expert datasets (PROMISE12 prostate MRI, LiTS liver/tumor CT, AMOS22 multi-organ CT/MRI) and a real-expert retinal fundus benchmark (Chaksu):

• DeferredSeg achieves superior Dice and Jaccard scores across all organs, especially in boundary and low-contrast regions.
• In PROMISE12, DeferredSeg (1-expert) yields $97.06\%\pm0.05$ DSC compared to fine-tuned MedSAM’s $86.17\%\pm1.53$ (Prostate). For LiTS tumor, it reaches $91.01\%\pm4.30$ DSC against MedSAM’s $59.20\%\pm0.26$.
• Performance is robust to the number of experts, with the single-expert routing typically yielding the highest overlap (DSC, Jaccard), while additional experts bolster system sensitivity, especially in highly ambiguous regimes.
• Plug-and-play ability is validated by integrating DeferredSeg with CENet, realizing improvements of $20$ DSC points for PROMISE12 (CENet: $68.96\%\pm0.16$ vs DeferredSeg-CENet: $88.85\%\pm0.07$).
• On Chaksu, a real-expert setting, DeferredSeg outperforms baselines in system overlap, confirming its transfer to real-world, non-synthetic expert variability.

  Figure 3: System Dice and Jaccard indices as a function of expert number; highest overlap at $J=1$, with partial recovery and increased sensitivity for $J$0.

Analysis of Framework Components

Ablations highlight the indispensability and complementarity of each core loss:

• Spatial Coherence Loss consistently improves routing focus, especially near ambiguous regions.
• Load-Balancing Penalty is essential in multi-expert setups; removing it produces routing collapse, degrading both overlap and expert utilization.
• The aggregation of all components yields maximal improvements, with gains distributed across system and branch-specific metrics.

Scaling studies show DeferredSeg robustly handles three or more complementary expert branches, with system Dice/Jaccard stabilizing and sensitivity maximized as routing dispersion increases judiciously.

Figure 4: Complete heatmap grid of system performance as a function of spatial coherence and load-balancing hyperparameters, exhibiting a clear optimum.

Practical Implications and Future Directions

DeferredSeg operationalizes pixel-level collaborative AI in segmentation, supporting region-selective human review in scalable annotation workflows—aligning with real-world clinical desiderata for focused, risk-aware expert involvement. Its model-agnostic design enables seamless extension to arbitrary dense segmentation architectures, including foundation models adapting to novel organs or modalities.

The demonstrated ability to accommodate synthetic and real experts, jointly optimize routing, and maintain spatial interpretability positions DeferredSeg as a new state-of-the-art for trustworthy AI-in-the-loop medical image analysis. The implications extend to annotation curation, with downstream improvements in dataset quality and reduced expert labor.

Anticipated future developments include:

• Incorporation of confidence calibration and uncertainty quantification post-deferral to optimize calibration across model and expert predictions.
• Exploration of reinforcement learning for adaptive, cost-sensitive routing policies.
• Extension to semi-supervised and federated learning regimes with sparse or non-overlapping expert availability.

  Figure 5: Interactive annotation UI demonstrating DeferredSeg’s workflow: model prediction and routing, expert annotation by region, and fused mask visualization.

Conclusion

DeferredSeg (2604.12411) introduces a principled, scalable, and effective framework for pixel-wise expert triage in medical image segmentation. By extending learning-to-defer to dense predictions and supporting multi-expert collaboration with rigorous loss design and integration, DeferredSeg substantially improves both segmentation reliability and practicality for high-stakes domains. The approach provides a robust foundation for future research in collaborative medical AI and structured human-in-the-loop dense prediction systems.