SAM2S: Surgical iVOS Model

• SAM2S is a specialized foundation model for interactive surgical video segmentation, integrating long-term memory, semantic learning, and ambiguity-resilient strategies.
• It is trained on the extensive SA-SV benchmark with over 61,000 annotated frames, achieving significant improvements in tracking and real-time performance.
• The model's modular design effectively addresses occlusion, instrument semantics, and annotation inconsistencies, making it suitable for intraoperative deployment at around 68 FPS.

SAM2S is a specialized foundation model for interactive Video Object Segmentation (iVOS) in surgical videos, designed to address limitations of general-purpose prompt-based models such as SAM2 when deployed in clinical contexts. SAM2S is trained on the SA-SV benchmark, which is the largest available set of instance-level spatio-temporal segmentations ("masklets") for the surgical domain, and integrates three purpose-built modules: DiveMem (trainable diverse long-term memory), Temporal Semantic Learning (TSL), and Ambiguity-Resilient Learning (ARL). The model achieves substantial improvements in long-term, real-time tracking and semantic understanding of both instruments and tissues, as assessed by the composite $\mathcal{J}$–$\mathcal{F}$ metric, while maintaining robust zero-shot generalization and operational speed suitable for intraoperative deployment (Liu et al., 20 Nov 2025).

1. Scientific Motivation and Domain Challenges

Surgical video analysis presents distinct challenges absent in natural video segmentation benchmarks. These videos exhibit complex domain gaps, including variable lighting, blood or smoke occlusions, highly reflective surgical instruments, extended durations ranging from tens of minutes to hours, frequent camera movements, and pronounced zooming. Semantic structure is also unique: while there is a limited, well-defined set of instruments, tissue boundaries are often ambiguous, and existing annotations across datasets are inconsistent. Prompt-driven iVOS models like SAM2—where a user initializes tracking with simple spatial cues, then relies on a memory bank and temporal attention—struggle in this setting: they fail to maintain long-term tracking through occlusions, cannot leverage known instrument semantics, and are prone to overfitting on imprecise labels at ambiguous boundaries. SAM2S is motivated by the need to close these gaps for surgical video segmentation.

2. The SA-SV Benchmark

The SA-SV benchmark aggregates and refines annotations from 17 open-source surgical video and image datasets, providing a large-scale resource for developing and evaluating iVOS methods in the surgical context. Key features of SA-SV include:

• Scale: 61,000 annotated frames, 1,600 masklets (instance tracks), with 67,000 instrument masks and 20,000 tissue masks across eight procedure types (cholecystectomy, colonoscopy, gynecology, hysterectomy, myotomy, nephrectomy, prostatectomy, multi-procedural robotic tool use).
• Splits: 473 training videos (41,593 frames, 1,368 masklets) and 99 test videos (19,804 frames, 244 masklets). Long-term test subsets are included, such as CIS-Test (1,807 s), RARP50 (325 s), Hyst-YT (329 s), EndoVis17 (300 s).
• Zero-shot generalization: Nephrectomy datasets (EV17, EV18) are excluded from training to enable cross-procedure generalization evaluation. Distinct instrument/tissue splits are also defined (EV18-I, EV18-T).
• Annotations: Single-frame semantic masks were converted into masklets by unique instance ID assignment, instrument class standardization, and expert-corrected boundary refinement, enabling direct evaluation of tracking and generalization.

3. SAM2S Model Architecture

SAM2S builds on the promptable encoder–decoder backbone of SAM2, retaining short-term memory over the six latest frames, but introduces three new modules:

3.1 DiveMem: Trainable Diverse Long-Term Memory

DiveMem addresses long-term consistency by dynamically managing a memory bank with short- and long-term components. During training, clips are sampled with hybrid strategies: from each eight-frame segment, one conditional frame, two long-term frames (with learned temporal embeddings), and five short-term consecutive frames simulate varied occlusion and reappearance conditions. At inference, a "diverse filter" maintains a buffer $B$ of frames with high IoU confidence ($iou_t > \gamma_{iou}$ for $\Delta = 5$), and selects the next long-term memory $b^*$ as the frame most dissimilar in the embedding space from the previous long-term key:

$$b^* = \underset{b_i \in B}{\arg\min}\; \frac{E(b_i) \cdot E(\ell_k)}{\|E(b_i)\| \;\|E(\ell_k)\|}$$

where $E(\cdot)$ is the frozen image encoder and $\ell_k$ is the last selected long-term frame. This design improves robustness to viewpoint shifts and occlusion, always retaining the initial frame for memory stability.

3.2 Temporal Semantic Learning (TSL)

TSL leverages the fact that surgical instruments belong to a limited set of semantic classes that recur across procedures. A learnable CLS token $x_c$ aggregates semantic context from the memory and cross-attends to the current frame, producing $\mathcal{F}$0. This is supervised by contrastive loss against CLIP text embeddings $\mathcal{F}$1 for instrument classes:

$\mathcal{F}$2

using temperature $\mathcal{F}$3; samples without instrument class labels are excluded from this loss.

3.3 Ambiguity-Resilient Learning (ARL)

To address inter-annotator inconsistencies in tissue boundaries, ARL convolves each ground-truth mask $\mathcal{F}$4 with a Gaussian kernel $\mathcal{F}$5 ($\mathcal{F}$6, $\mathcal{F}$7), yielding softened labels $\mathcal{F}$8. A focal loss is then applied between the model's predicted mask probability $\mathcal{F}$9 and $B$0:

$B$1

3.4 Training Objective and Protocol

The total training loss is:

$B$2

with $B$3 and $B$4. Training starts from SAM2 Hiera-B+ weights; uses 30 epochs, a learning rate $B$5, an image:video ratio of 1:4, and batch parameters tuned for NVIDIA A6000 GPUs. DiveMem parameters are set as $B$6, $B$7, $B$8. Prompting uses three clicks per initialization.

4. Quantitative Evaluation and Comparative Performance

SAM2S is evaluated primarily using the average $B$9–$iou_t > \gamma_{iou}$0 metric, where $iou_t > \gamma_{iou}$1 indicates region overlap (IoU) and $iou_t > \gamma_{iou}$2 denotes boundary F-score, with the final score as their mean. At 512 px input, the results are:

Model	Avg $iou_t > \gamma_{iou}$3–$iou_t > \gamma_{iou}$4	FPS
SAM2 (vanilla)	63.32	26 (1024 px) / 69 (512 px)
SAM2 (FT on SA-SV)	76.31	69
SAM2S	80.42	68

Ablation experiments show additive benefits:

Configuration	Avg $iou_t > \gamma_{iou}$5–$iou_t > \gamma_{iou}$6
SAM2 vanilla	59.44
+ Fine-tune on SA-SV only	77.16
+ DiveMem	80.10 (+2.94)
+ TSL	80.30 (+3.14)
+ DiveMem + TSL	81.52 (+4.36)
+ DiveMem + TSL + ARL (full SAM2S)	82.76 (+5.60)

Key observations:

• SAM2S outperforms vanilla SAM2 by 17.10 points and fine-tuned SAM2 by 4.11 points on average $iou_t > \gamma_{iou}$7–$iou_t > \gamma_{iou}$8.
• Zero-shot generalization tests (nephrectomy EV17, EV18-I) indicate SAM2S achieves approximately 87%/82% $iou_t > \gamma_{iou}$9–$\Delta = 5$0, exceeding fine-tuned SAM2 by 5–6 points.
• On long-duration videos, DiveMem produces a gain of 2.96–9.56 $\Delta = 5$1–$\Delta = 5$2 over fine-tuned SAM2, whereas training-free memory strategies lose context in long sequences.

5. Implications, Limitations, and Prospective Directions

SAM2S is the first foundation iVOS architecture explicitly tailored for surgical videos, achieving both prompt flexibility and robust long-term segmentation across diverse procedures. Its approximate 68 FPS performance is suitable for real-time, intraoperative contexts. The integration of DiveMem, TSL, and ARL modules directly addresses core domain challenges: maintaining instance correspondence through occlusion and reappearance, leveraging a limited set of instrument semantics, and mitigating label ambiguity at tissue boundaries.

Limitations include persistent challenges in tissue segmentation where semantic cues are weak; ARL offers partial mitigation but does not fully resolve boundary ambiguity. Very long procedures (over one hour) or rare instrument classes outside the SA-SV corpus may still degrade performance.

Future directions proposed include extending the SA-SV benchmark to new domains (orthopedic, cardiac), refining outputs to cover instrument poses or parts, developing adaptive memory mechanisms for scene shift detection, and integrating SAM2S with downstream tasks such as skill assessment, anomaly detection, or augmented reality overlays (Liu et al., 20 Nov 2025).