UniSurgSAM: A Unified Promptable Model for Reliable Surgical Video Segmentation

Abstract: Surgical video segmentation is fundamental to computer-assisted surgery. In practice, surgeons need to dynamically specify targets throughout extended procedures, using heterogeneous cues such as visual selections, textual expressions, or audio instructions. However, existing Promptable Video Object Segmentation (PVOS) methods are typically restricted to a single prompt modality and rely on coupled frameworks that cause optimization interference between target initialization and tracking. Moreover, these methods produce hallucinated predictions when the target is absent and suffer from accumulated mask drift without failure recovery. To address these challenges, we present UniSurgSAM, a unified PVOS model enabling reliable surgical video segmentation through visual, textual, or audio prompts. Specifically, UniSurgSAM employs a decoupled two-stage framework that independently optimizes initialization and tracking to resolve the optimization interference. Within this framework, we introduce three key designs for reliability: presence-aware decoding that models target absence to suppress hallucinations; boundary-aware long-term tracking that prevents mask drift over extended sequences; and adaptive state transition that closes the loop between stages for failure recovery. Furthermore, we establish a multi-modal and multi-granular benchmark from four public surgical datasets with precise instance-level masklets. Extensive experiments demonstrate that UniSurgSAM achieves state-of-the-art performance in real time across all prompt modalities and granularities, providing a practical foundation for computer-assisted surgery. Code and datasets will be available at https://jinlab-imvr.github.io/UniSurgSAM.

• The paper presents a decoupled two-stage framework that separates initialization and tracking to reduce mask hallucination and drift.
• It leverages multi-modal prompts—visual, textual, and audio—to enhance presence-aware decoding and boundary delineation, achieving superior performance.
• Experimental validation demonstrates significant improvements on multiple benchmarks, ensuring reliable, real-time performance in surgical video segmentation.

A Unified Multi-Modal Framework for Robust Surgical Video Segmentation: An Essay on UniSurgSAM

Introduction

The paper "UniSurgSAM: A Unified Promptable Model for Reliable Surgical Video Segmentation" (2604.03645) presents a comprehensive framework for Promptable Video Object Segmentation (PVOS) in surgical settings. It addresses the core limitation of modality-constrained segmentation systems in real-world operating room scenarios by developing a unified model capable of robust segmentation with visual, textual, and audio prompts. The architecture introduces substantial technical advancements to tackle fundamental challenges in both initialization and temporal segmentation, including explicit modeling of target presence, geometric boundary maintenance, and resilient state management.

Motivation and Technical Challenges

Conventional PVOS systems in the surgical domain are predominantly restricted to single prompt modalities and often utilize architectures where target initialization and tracking are intertwined. This coupled framework introduces "optimization interference," in which the demands of high-level semantic discrimination during initialization conflict with the requirements for spatiotemporal consistency during tracking, resulting in suboptimal performance in both stages.

Moreover, the lack of explicit models for handling absent or occluded targets leads to mask hallucinations — a critical failure mode in the high-stakes context of surgery — while mask drift degrades the usability of predictions over extended video sequences. There is a stringent need for a unified, multi-modal framework that offers evidence-based presence awareness, geometric precision during tracking, and adaptive recovery from failures.

Figure 1: Comparison between single-modal PVOS systems and UniSurgSAM; highlights the pitfalls of coupled decoders and the improved robustness of the UniSurgSAM decoupled pipeline.

Architecture: Decoupled Two-Stage Multi-Modal Framework

UniSurgSAM comprises a decoupled two-stage architecture:

• Stage I: Unified Promptable Initialization leverages modality-specific encoders and decoders to produce consistent, presence-aware initial target masks. For visual prompts, spatial anchoring is direct, while for linguistic and audio prompts, Vision-Language fusion is performed via Cross-Modal Spatial-Temporal Mamba (CSTMamba). Reliable Presence-Aware Decoding (RPAD) explicitly supervises target presence using negative sampling strategies, mitigating hallucination risk.
• Stage II: Boundary-Aware Long-Term Tracking maintains target segmentation through an augmented tracking decoder. A novel Boundary Head predicts object boundaries, and the resultant boundary features are stored in the memory bank alongside learned mask and appearance representations. The memory architecture employs Diversity-Driven Long-Term Memory (DLM) with diversity-based eviction, maximizing spatial-temporal representational coverage, effectively suppressing mask drift.

Transition between stages is governed by Adaptive State Transition (AST), which enforces credible tracking activation using temporal consistency checks and enables automatic fallback to re-initialization when consensus-based checks signal tracking failure.

Figure 2: Architectural overview depicting modality-specific prompt processing and coordination between the initialization detector and boundary-aware tracker.

Key Reliability Mechanisms

Presence-Aware Decoding

RPAD introduces explicit supervision for absence cases, through both category-level and spatial hard negative samples, and a dedicated presence-head per decoder. This design achieves substantial reduction in False Positive Rate (FPR): for instance, on Uni-EndoVis17, FPR declines from 45.6 to 18.0 with a corresponding enhancement in presence detection precision (from 86.5 to 94.0).

Figure 3: Empirical analysis of presence-aware decoding; demonstrates FPR suppression and precision increases resulting from negative sampling supervision.

Boundary-Aware Tracking and Memory

The inclusion of a boundary prediction head, boundary-feature memory integration, and diversity-driven memory selection addresses both geometric inconsistency and error propagation. The geometric memory encoding enables the system to discriminate between the target and visually similar distractors, counteracting the common failure mode of drift toward adjacent structures.

Figure 4: Qualitative visual comparison highlighting boundary retention and improved tracking stability under challenging conditions.

Adaptive State Management

AST's logic gates (Entry and Exit) ensure reliable stage transitions. Entry restricts tracker activation to high-confidence detections, validated over a temporal window, while Exit enables closed-loop recovery by periodical semantic consensus between initialization and tracking outputs. These mechanisms are critical for long-duration tracking, preventing irreversible error propagation.

Figure 5: Example of consensus-based fallback correcting catastrophic tracker drift and re-anchoring to the semantic reference.

Experimental Validation

Benchmark and Evaluation Suite

A new multi-modal, multi-granular PVOS benchmark was established, aggregating and normalizing four public surgical video datasets. The evaluation spans visual, textual, and audio prompting, with both whole-object and part-level granularity.

Quantitative Performance

UniSurgSAM achieves state-of-the-art results in all evaluated modalities and settings:

• In visual PVOS, it outperforms prior art (SurgicalSAM2, MA-SAM2) by +6.3 $\mathcal{J}{%%%%0%%%%}\mathcal{F}$ on Uni-EndoVis17 and +4.2 on Uni-EndoVis18-I.
• In textual PVOS, it exceeds ReSurgSAM2 by over +4.3 on Uni-EndoVis17 and +5.6 on Uni-RARP50.
• Real-time efficiency is retained (53–68 FPS, including audio pipeline overhead).
• For part-level segmentation, the framework improves $\mathcal{J}{%%%%1%%%%}\mathcal{F}$ by upto +9.7 compared to the strongest text-based baseline.

  Figure 6: Qualitative results for textual PVOS, highlighting robust handling of occlusions, absence, and mask drift compared to previous frameworks.

Ablation Studies

Ablation studies dissect the contributions of architectural and loss components. Decoupled decoders yield consistent increases in segmentation accuracy, while boundary-head and geometric memory integration each provide substantial independent improvements. The synergy of AST (credible activation plus fallback) and boundary-aware tracking is essential for long-sequence robustness.

Figure 7: Analysis of gradient dynamics and loss convergence: decoupled architecture eliminates optimization interference, yielding stable, monotonic optimization.

Theoretical and Practical Implications

From a methodological perspective, UniSurgSAM validates the hypothesis that architectural decoupling of disparate objectives is optimal in PVOS, countering the conventional wisdom of joint optimization. The strong ablation evidence for the design of explicit presence modeling, geometric auxiliary supervision, and semantic-conditioned state management extends directly to the general study of sequential decision networks.

Practically, this framework lowers the barrier for deployment of flexible, real-time human-in-the-loop AI in surgery, providing robust control via diverse interaction modalities. The high segmentation reliability under prompt and scene heterogeneity suggests direct applicability for real-world intraoperative assistance, with minimal risk from hallucination or drift.

The system’s architectural choices — especially the decoupled pipeline and closed-loop error recovery — signal design principles for future interactive AI agents where safety, explainability, and multi-modal integration are paramount.

Speculation on Future Developments

Future work will need to expand domain generalization across broader sets of procedures, integrate noise-robust ASR modules for practical audio prompting, and develop further coordination between perception and downstream decision/action modules (e.g., integration with autonomous robotic systems). The paradigm of closed-loop promptable segmentation with rigorous reliability control is likely to extend to other safety-critical video analysis contexts beyond surgery.

Conclusion

UniSurgSAM establishes a new state-of-the-art paradigm for surgical PVOS by resolving the core technical issues of modality restriction and optimization interference through a decoupled, multi-modal, presence- and boundary-aware architecture. The proposed framework sets a benchmark in unifying flexible prompting with real-time, reliable long-term video segmentation, and offers significant implications for interactive, safety-critical AI deployment in the surgical domain.