Seeing Through the Tool: A Controlled Benchmark for Occlusion Robustness in Foundation Segmentation Models

Abstract: Occlusion, where target structures are partially hidden by surgical instruments or overlapping tissues, remains a critical yet underexplored challenge for foundation segmentation models in clinical endoscopy. We introduce OccSAM-Bench, a benchmark designed to systematically evaluate SAM-family models under controlled, synthesized surgical occlusion. Our framework simulates two occlusion types (i.e., surgical tool overlay and cutout) across three calibrated severity levels on three public polyp datasets. We propose a novel three-region evaluation protocol that decomposes segmentation performance into full, visible-only, and invisible targets. This metric exposes behaviors that standard amodal evaluation obscures, revealing two distinct model archetypes: Occluder-Aware models (SAM, SAM 2, SAM 3, MedSAM3), which prioritize visible tissue delineation and reject instruments, and Occluder-Agnostic models (MedSAM, MedSAM2), which confidently predict into occluded regions. SAM-Med2D aligns with neither and underperforms across all conditions. Ultimately, our results demonstrate that occlusion robustness is not uniform across architectures, and model selection must be driven by specific clinical intent-whether prioritizing conservative visible-tissue segmentation or the amodal inference of hidden anatomy.

• The paper demonstrates that full-mask evaluation metrics are insufficient for assessing clinically safe segmentation under occlusion.
• It employs a dual occlusion synthesis strategy and a three-region evaluation protocol to analyze model performance across visible, invisible, and full masks.
• Results reveal that clinical fine-tuning can lead to unsafe predictions by leaking into occluded regions, underscoring the need for tailored evaluation methods.

Benchmarking Occlusion Robustness in Foundation Segmentation Models: Insights from OccSAM-Bench

Introduction and Motivation

Robustness to occlusion is a critical obstacle for foundation segmentation models deployed in clinical endoscopy, where target anatomies are often partially or wholly obscured by surgical instruments. While the Segment Anything Model (SAM) family and its medical adaptations have shown strong zero-shot generalization, their behavior under clinically-relevant occlusion remains uncharacterized. This work introduces OccSAM-Bench—a synthetic, controlled benchmark explicitly targeting the occlusion regime in medical segmentation—systematically probing the limits of current foundation model architectures.

Figure 1: OccSAM-Bench evaluates foundation model segmentation under controlled surgical occlusions, decomposing predictions into visible, invisible, and full masks.

Benchmark Design and Methodology

OccSAM-Bench employs a dual occlusion synthesis strategy:

• Surgical-tool overlay: Realistic binary masks of surgical instruments are stochastically pasted onto polyp images, simulating clinical endoscopic occlusion.
• Cutout occlusion: Inspired by CutOut augmentation, rectangular regions are excised from the target, isolating information loss without visual confusion.

Each occlusion type is parameterized across three severity bins (0–20%, 20–40%, 40–60% occluded area), resulting in balanced, mathematically strict occlusion distributions.

A core contribution is the three-region evaluation protocol, which decomposes segmentation accuracy as follows:

1. Visible mask: Ground truth tissue not occluded by instruments (prioritizing clinical safety).
2. Invisible mask: Occluded tissue region (testing amodal reasoning).
3. Full mask: Standard overlap, confounding clinical adequacy under occlusion.

Seven segmentation models are benchmarked, including:

• General-purpose SAM, SAM 2, and SAM 3
• Medical adaptations MedSAM, MedSAM2, MedSAM3, and SAM-Med2D

Models are evaluated zero-shot on three standard colonoscopy polyp datasets (CVC-300, CVC-ColonDB, ETIS-LaribPolypDB), using both bounding box and point prompts.

Figure 2: Overview of the synthetic occlusion pipeline, revealing two model behaviors—Occluder-Aware (conservative, rejects tool overlap) and Occluder-Agnostic (predicts into occluded regions). Similar amodal scores can conceal radically different error modes.

Experimental Results

Standard (Full-Mask) Evaluation Is Inadequate

Full-mask Dice scores are unreliable indicators of safe segmentation in the presence of occluders. Architectures such as MedSAM and MedSAM2 retain or even inflate full-mask DSC under severe occlusion, not by accurately inferring hidden tissue, but by erroneously leaking predictions into the instrument footprint. Thus, they are rewarded for unsafe boundary overflow rather than correct clinical interpretation.

Behavioral Archetypes and Clinical Implications

Visible-region evaluation exposes two sharply distinct model behaviors:

• Occluder-Aware (SAM, SAM 2, SAM 3, MedSAM3): Suppress predictions in the tool region, prioritizing conservative segmentation of visible anatomy; experience less performance degradation under high occlusion.
• Occluder-Agnostic (MedSAM, MedSAM2): Predict into occluded regions, achieving artificially high invisible DSC. MedSAM2, notably, exhibits competitive visible DSC and high invisible DSC, suggesting video-based fine-tuning imparts a degree of context-aware amodal reasoning.

SAM-Med2D is an outlier, failing to clearly manifest either archetype and performing poorly under all regimes.

Figure 3: Performance gap between Cutout and Surgical Tool occlusion is negligible at mild occlusions but diverges at high severity; Surgical-tools induce more severe failure than information loss alone, especially for medical-finetuned models.

Effect of Occlusion Type

At low and medium severity, segmentation degradation is similar for both Cutout and Surgical Tool overlays. At high severity, Surgical Tool occlusion causes substantially larger failures, highlighting the model's inability to distinguish instrument edges from anatomy—particularly among medical adaptations heavily tuned on clean data.

Prompting Strategies

Bounding box prompts consistently yield higher DSC across all conditions. Some models, such as MedSAM, fail completely when only interior point prompts are used, exposing limitations imposed by prompt encoder training schemes.

Figure 4: Box prompts maintain superior visible-mask performance compared to single-point prompts across all occlusion levels—prompting robustness is non-uniform across model families.

Invisible Region (Amodal) Evaluation

Invisible-region scores must be interpreted jointly with visible DSC. High invisible DSC with low visible DSC is almost always due to uncontrolled overflow, not genuine structure completion. MedSAM2 is unique in preserving both, aligning with its design for video sequences and possible learned context integration.

Practical and Theoretical Implications

The study demonstrates that domain-specific fine-tuning can degrade, not improve, compositional robustness of foundation models under real-world perturbations. Overfitting to clean benchmarks causes medical-adapted models to conflate instrument edges with anatomical boundaries, failing to operationalize surgical safety constraints. The three-region protocol is essential for any evaluation in environments where occlusion by foreign objects is common and potentially dangerous.

Implications extend both to model selection (conservative vs. amodal boundaries must be tailored to clinical task) and to architectural training (robustness to non-anatomical occluders should be an explicit pretraining or adaptation objective).

Limitations and Future Work

OccSAM-Bench is currently limited to 2D RGB colonoscopy images and synthetic occlusions. While this provides full ground truth for decomposed evaluation, it does not capture optical artifacts (reflections, deformation) present in vivo. Future benchmarks should expand to naturalistic sequence-based occlusion, higher-fidelity rendering, and other imaging modalities (e.g., laparoscopy, radiology). Additionally, training procedures that explicitly regularize against instrument-induced out-of-distribution perturbations may further improve clinical safety.

Conclusion

OccSAM-Bench exposes deficiencies in current foundation segmentation models, especially regarding occlusion robustness in surgical environments. By moving beyond full-mask metrics and evaluating decomposed regions, the work clarifies model selection criteria for practical use and motivates a new research agenda toward both clinically safe conservative segmentation and context-aware amodal reasoning (2604.11711).