TSMS-SAM2 for Surgical Video Segmentation

• The paper introduces TSMS-SAM2, a SAM2-based framework that incorporates multi-temporal-scale video sampling and memory splitting with pruning to enhance segmentation in surgical videos.
• TSMS-SAM2 fine-tunes key modules while preserving SAM2’s pre-trained features, adapting the model specifically for the rapid, irregular motion and clutter challenges in surgical environments.
• Experimental results on EndoVis benchmarks show TSMS-SAM2 achieving mean Dice scores of up to 95.24, significantly outperforming competing methods in handling occlusions and redundancy.

TSMS-SAM2 is a SAM2-based framework for promptable video object segmentation and tracking in surgical videos, proposed to address two failure modes that become pronounced in this domain: limited robustness to rapid and irregular instrument motion, and redundancy in SAM2’s streaming memory bank. Built on the Segment Anything Model 2, which the literature positions as a promptable segmentation model for images and videos with streaming memory and memory attention (Jiaxing et al., 17 Mar 2025), TSMS-SAM2 adds two surgical-domain adaptations: multi-temporal-scale video sampling augmentation and memory splitting with similarity-based pruning. On EndoVis2017 and EndoVis2018, the paper reports mean Dice scores of 95.24 and 86.73, respectively (Xu et al., 7 Aug 2025).

1. Problem setting and relation to SAM2

TSMS-SAM2 targets promptable video object segmentation and tracking (VOST) in surgical videos. In this setting, a user gives a prompt on the first frame—mask, point, or box—and the model must track and segment the target instrument or object throughout the sequence (Xu et al., 7 Aug 2025). The task is difficult because surgical videos exhibit rapid and irregular motion, occlusions, sudden disappearance and reappearance, cluttered backgrounds, and domain shift relative to the natural-image data on which general foundation models are typically trained.

Within the broader SAM2 literature, SAM2 is characterized as a promptable segmentation model that extends SAM with a streaming memory component, supporting video segmentation and object tracking through a memory encoder, memory attention, and a memory bank (Jiaxing et al., 17 Mar 2025). TSMS-SAM2 does not replace that foundation; it adapts the memory-centric video mechanism to a surgical regime in which temporal redundancy and abrupt motion are especially consequential.

The central claim of the TSMS-SAM2 paper is that vanilla SAM2 is not sufficiently tailored to surgical video dynamics. The authors identify two issues: first, a single temporal sampling pattern does not adequately simulate the range of motion speeds and discontinuities seen in surgical instrument trajectories; second, the fixed FIFO memory bank can accumulate highly similar adjacent frames, which wastes computation and may reduce the quality of temporal information propagation (Xu et al., 7 Aug 2025).

2. Architecture and operating principle

TSMS-SAM2 preserves the basic SAM2 inference structure. The image encoder extracts frame features, the prompt encoder encodes the initial prompt, the memory attention module conditions current-frame prediction on prior memory, the mask decoder outputs segmentation masks, and the memory encoder converts selected outputs into memory entries stored in the memory bank (Xu et al., 7 Aug 2025).

The framework modifies this baseline in two places. During training, it applies multi-temporal-scale video sampling augmentation so that the same surgical content is seen at different temporal strides. During inference-time memory management, it applies memory splitting and pruning, dividing the bank into short-term and long-term groups and pruning redundancies within each group rather than relying on undifferentiated FIFO retention (Xu et al., 7 Aug 2025).

A practical point is that TSMS-SAM2 is a fine-tuned adaptation, not a training-free wrapper. The paper reports fine-tuning the mask decoder, memory attention module, and memory encoder, while keeping the image encoder and prompt encoder frozen. The reported training configuration uses a SAM2-Small backbone, $512 \times 512$ input resolution, 30 epochs, batch size 2, AdamW, and a learning rate of $5 \times 10^{-5}$ on a single NVIDIA RTX 4090 under Ubuntu (Xu et al., 7 Aug 2025).

This design implies a specific adaptation philosophy: preserve SAM2’s pretrained visual representation and prompting interface, while specializing the temporal and memory subsystems for surgical video dynamics. That interpretation is directly consistent with the modules the paper chooses to update.

3. Multi-temporal-scale video sampling augmentation

The first core component is multi-temporal-scale video sampling augmentation. Its purpose is to improve robustness to different apparent motion speeds by sampling the same video at multiple temporal strides during training (Xu et al., 7 Aug 2025). The paper describes this as a way to simulate faster motion, slower motion, abrupt temporal changes, and missing-object scenarios.

Conceptually, stride-1 sampling preserves the original temporal continuity, while larger strides such as 2 or 3 create wider temporal gaps, making objects appear to move faster or more abruptly. The framework therefore exposes the model to a broader distribution of motion patterns than a single-rate training pipeline. The paper explicitly evaluates sampling combinations (1, 2), (1, 3), and (1, 2, 3), and reports that (1, 2) performs best (Xu et al., 7 Aug 2025).

The reported interpretation is narrow and practical. The (1, 2) configuration provides a useful balance between variation and correspondence preservation. By contrast, (1, 3) introduces larger gaps that hurt feature correspondence, and (1, 2, 3) is more diverse but not as effective as the simpler two-scale variant (Xu et al., 7 Aug 2025).

This augmentation strategy is explicitly motivated by the observation that surgical instrument motion is not uniform. The paper connects the idea to multi-rate video modeling and uses it to make SAM2 less dependent on a single notion of temporal smoothness. A plausible implication is that the method regularizes the memory-conditioned decoder against overfitting to short-range continuity alone, but the paper’s concrete evidence is the ablation improvement rather than a separate theoretical derivation.

4. Memory splitting and pruning

The second core component is memory splitting and pruning, which addresses redundancy in SAM2’s memory bank. Vanilla SAM2 stores past frame features in a fixed-capacity FIFO memory bank. TSMS-SAM2 argues that, in long surgical sequences, many adjacent frames are highly similar, so the memory can become redundant and overly focused on recent content (Xu et al., 7 Aug 2025).

To counter this, the method divides the memory bank into short-term memory and long-term memory. With a maximum memory size of 7 past frames, the implementation uses up to 4 short-term frames and up to 3 long-term frames (Xu et al., 7 Aug 2025). The paper states that the splitting is inspired by the Atkinson-Shiffrin memory model, distinguishing immediate local context from older context that can still be useful after occlusion, disappearance, or viewpoint change.

Pruning is then applied within each group independently. Similarity is computed using frame features, and the most redundant frame in each group is removed when the memory bank is full. The short-term group measures similarity relative to the latest frame $f_t$; the long-term group measures similarity relative to the oldest frame in the group, described as $f_{t-(n-1)}$ (Xu et al., 7 Aug 2025). If there are fewer than 7 past frames, no pruning is applied. When the bank reaches 7 frames, one redundant frame is pruned from each group, leaving 5 retained frames for updating current features (Xu et al., 7 Aug 2025).

The similarity metric itself is an ablated design choice. The paper compares cosine similarity, Manhattan (L1), Euclidean (L2), dot product, Spearman correlation, and Pearson correlation, reporting that cosine similarity gives the best Dice and best overall performance, with Spearman correlation as a close second (Xu et al., 7 Aug 2025).

The conceptual distinction from pruning solely against the latest frame is important. Older frames may be dissimilar to the current frame yet still encode useful appearance, geometry, or viewpoint information for recovery after reappearance. This suggests that the split-memory design is intended not merely to compress the bank, but to preserve temporal diversity across different horizons.

5. Experimental protocol, datasets, and metrics

The experiments are conducted on two surgical instrument benchmarks: EndoVis2017 and EndoVis2018 (Xu et al., 7 Aug 2025). For EndoVis2017, the paper reports 8 training sequences, 225 frames each; 8 test sequences, 75 frames each; and 2 additional hold-out test sequences (9 and 10), 600 frames total, for 3,000 annotated frames overall. The instrument classes are Bipolar Forceps, Prograsp Forceps, Large Needle Driver, Vessel Sealer, Grasping Retractor, and Monopolar Curved Scissors. A notable protocol decision is that the authors use only the two hold-out test sequences (9 and 10) “to avoid leakage and better evaluate out-of-distribution performance” (Xu et al., 7 Aug 2025).

For EndoVis2018, the paper reports 15 videos, 149 frames each, for 2,235 annotated frames. The classes are Bipolar Forceps, Prograsp Forceps, Large Needle Driver, Monopolar Curved Scissors, Ultrasound Probe, Suction Instrument, and Clip Applier. Following ISINet and SurgSAM2, the evaluation uses sequences 2, 5, 9, 15 as test videos and the remaining 11 videos as training data (Xu et al., 7 Aug 2025).

The metric suite combines regional, boundary, and sequence-level measures. The paper uses Jaccard Index / IoU, Boundary F1, J&F, Dice, and Challenge IoU (CIoU). The reported formulas are the standard ones:

$$J=\frac{|P\cap G|}{|P\cup G|}, \qquad F=\frac{2\cdot Precision\cdot Recall}{Precision+Recall},$$

$$JDice=\frac{2|P\cap G|}{|P|+|G|}.$$

CIoU is described as an object-level metric that accumulates predicted and ground-truth regions over the full video sequence (Xu et al., 7 Aug 2025).

6. Results and ablation evidence

The paper reports that TSMS-SAM2 achieves the strongest mean Dice scores among the compared methods on both surgical benchmarks (Xu et al., 7 Aug 2025).

Dataset	TSMS-SAM2	Comparison highlights
EndoVis2017	Dice 95.24 ± 0.96	SurgSAM2 94.29; SAM2 89.35; MedSAM2 87.37
EndoVis2018	Dice 86.73 ± 15.46	SurgSAM2 81.55; SAM2 72.79; MedSAM2 63.83
EndoVis2018 CIoU	85.1	SurgSAM2 84.4; MedSAM2 83.7; SAM2 82.2

On EndoVis2017, TSMS-SAM2 reports J&F 93.41 ± 1.68, J 91.89 ± 1.43, F 94.94 ± 3.01, and Dice 95.24 ± 0.96. On EndoVis2018, it reports J&F 84.48 ± 15.37, J 84.20 ± 15.17, F 84.77 ± 15.74, and Dice 86.73 ± 15.46 (Xu et al., 7 Aug 2025). The paper emphasizes that the gains on EndoVis2018 are particularly meaningful because that dataset is described as more difficult due to clutter and domain shift.

The ablation study isolates the two main contributions. With neither temporal sampling nor memory splitting, the baseline yields J&F 76.28 and Dice 78.44. Adding only memory splitting raises this to J&F 84.36 and Dice 86.69. Adding only temporal sampling yields J&F 83.13 and Dice 85.35. Combining both reaches J&F 84.48 and Dice 86.73 (Xu et al., 7 Aug 2025). The reported pattern indicates that both components contribute materially, with memory splitting slightly stronger than temporal sampling in this ablation, and the combination best overall.

The paper also reports qualitative improvements, stating that TSMS-SAM2 better handles instruments missed by SurgSAM2, including Large Needle Driver, Prograsp Forceps, and Ultrasound Probe (Xu et al., 7 Aug 2025). The qualitative interpretation offered is improved tracking stability, better recovery from missed detections, and stronger segmentation in challenging scenes.

7. Relation to adjacent SAM2 adaptations, misconceptions, and limitations

TSMS-SAM2 belongs to a broader class of SAM2 adaptations that intervene in memory behavior rather than discarding SAM2’s promptable segmentation backbone. The survey literature frames SAM2’s key distinction from SAM as the addition of streaming memory for video and tracking (Jiaxing et al., 17 Mar 2025). In surgical video specifically, MA-SAM2 proposes a different memory intervention: a training-free strategy with Context-Aware Memory (CAM) and Occlusion-Resilient Memory (ORM) to replace greedy FIFO behavior, reporting improvements on EndoVis2017 and EndoVis2018 without additional parameters or further training (Yin et al., 13 Jul 2025). This suggests that surgical adaptation of SAM2 has converged on memory management as a principal design axis, although the concrete mechanisms differ.

A common misconception would be to treat TSMS-SAM2 as a wholly new segmentation architecture. The paper does not support that reading. It is more precisely a SAM2 adaptation that modifies training-time temporal exposure and inference-time memory selection while retaining the promptable VOST pipeline (Xu et al., 7 Aug 2025). Another misconception would be to classify it as training-free; unlike MA-SAM2, TSMS-SAM2 explicitly fine-tunes selected modules.

The paper also states two limitations. First, the current multi-scale augmentation uses fixed sampling rates, such as 1 and 2. The authors suggest future adaptive sampling strategies based on scene dynamics, object presence or absence, keypoint-based motion cues such as SIFT, FAST, or ORB, shape and motion characteristics, or optical-flow-like information (Xu et al., 7 Aug 2025). Second, the short-term/long-term memory split is static, fixed at 4 and 3 frames. The paper proposes future dynamic or learnable memory separation strategies based on similarity, relevance scores, or attention-based updates, potentially closer in spirit to more advanced memory modules in XMem or AOT (Xu et al., 7 Aug 2025).

Taken together, these details place TSMS-SAM2 as a domain-specialized refinement of SAM2 for surgical promptable VOST: it does not alter the foundation model’s role as a promptable video segmenter, but restructures temporal sampling and memory retention so that surgical motion variability and memory redundancy are handled more explicitly.