---
title: TSMS-SAM2 for Surgical Video Segmentation
url: https://www.emergentmind.com/topics/tsms-sam2
type: topic
---

# TSMS-SAM2 for Surgical Video Segmentation

TSMS-SAM2 is a SAM2-based framework for promptable video object segmentation and tracking in surgical videos, proposed to address two failure modes that become pronounced in this domain: limited robustness to rapid and irregular instrument motion, and redundancy in SAM2’s streaming memory bank. Built on the Segment Anything Model 2, which the literature positions as a promptable segmentation model for images and videos with streaming memory and memory attention [2503.12781], TSMS-SAM2 adds two surgical-domain adaptations: multi-temporal-scale video sampling augmentation and memory splitting with similarity-based pruning. On EndoVis2017 and EndoVis2018, the paper reports mean Dice scores of 95.24 and 86.73, respectively [2508.05829].

## 1. Problem setting and relation to SAM2

TSMS-SAM2 targets **promptable video object segmentation and tracking (VOST)** in **surgical videos**. In this setting, a user gives a prompt on the first frame—mask, point, or box—and the model must track and segment the target instrument or object throughout the sequence [2508.05829]. The task is difficult because surgical videos exhibit rapid and irregular motion, occlusions, sudden disappearance and reappearance, cluttered backgrounds, and domain shift relative to the natural-image data on which general foundation models are typically trained.

Within the broader SAM2 literature, SAM2 is characterized as a promptable segmentation model that extends SAM with a **streaming memory component**, supporting video segmentation and object tracking through a memory encoder, memory attention, and a memory bank [2503.12781]. TSMS-SAM2 does not replace that foundation; it adapts the memory-centric video mechanism to a surgical regime in which temporal redundancy and abrupt motion are especially consequential.

The central claim of the TSMS-SAM2 paper is that vanilla SAM2 is not sufficiently tailored to surgical video dynamics. The authors identify two issues: first, a single temporal sampling pattern does not adequately simulate the range of motion speeds and discontinuities seen in surgical instrument trajectories; second, the fixed FIFO memory bank can accumulate highly similar adjacent frames, which wastes computation and may reduce the quality of temporal information propagation [2508.05829].

## 2. Architecture and operating principle

TSMS-SAM2 preserves the basic SAM2 inference structure. The image encoder extracts frame features, the prompt encoder encodes the initial prompt, the memory attention module conditions current-frame prediction on prior memory, the mask decoder outputs segmentation masks, and the memory encoder converts selected outputs into memory entries stored in the memory bank [2508.05829].

The framework modifies this baseline in two places. During training, it applies **multi-temporal-scale video sampling augmentation** so that the same surgical content is seen at different temporal strides. During inference-time memory management, it applies **memory splitting and pruning**, dividing the bank into short-term and long-term groups and pruning redundancies within each group rather than relying on undifferentiated FIFO retention [2508.05829].

A practical point is that TSMS-SAM2 is a **fine-tuned adaptation**, not a training-free wrapper. The paper reports fine-tuning the **mask decoder**, **memory attention module**, and **memory encoder**, while keeping the **image encoder** and **prompt encoder** frozen. The reported training configuration uses a **SAM2-Small** backbone, **\(512 \times 512\)** input resolution, **30 epochs**, **batch size 2**, **AdamW**, and a learning rate of **\(5 \times 10^{-5}\)** on a **single NVIDIA RTX 4090** under Ubuntu [2508.05829].

This design implies a specific adaptation philosophy: preserve SAM2’s pretrained visual representation and prompting interface, while specializing the temporal and memory subsystems for surgical video dynamics. That interpretation is directly consistent with the modules the paper chooses to update.

## 3. Multi-temporal-scale video sampling augmentation

The first core component is **multi-temporal-scale video sampling augmentation**. Its purpose is to improve robustness to different apparent motion speeds by sampling the same video at multiple temporal strides during training [2508.05829]. The paper describes this as a way to simulate faster motion, slower motion, abrupt temporal changes, and missing-object scenarios.

Conceptually, stride-1 sampling preserves the original temporal continuity, while larger strides such as 2 or 3 create wider temporal gaps, making objects appear to move faster or more abruptly. The framework therefore exposes the model to a broader distribution of motion patterns than a single-rate training pipeline. The paper explicitly evaluates sampling combinations **(1, 2)**, **(1, 3)**, and **(1, 2, 3)**, and reports that **(1, 2)** performs best [2508.05829].

The reported interpretation is narrow and practical. The **(1, 2)** configuration provides a useful balance between variation and correspondence preservation. By contrast, **(1, 3)** introduces larger gaps that hurt feature correspondence, and **(1, 2, 3)** is more diverse but not as effective as the simpler two-scale variant [2508.05829].

This augmentation strategy is explicitly motivated by the observation that surgical instrument motion is not uniform. The paper connects the idea to multi-rate video modeling and uses it to make SAM2 less dependent on a single notion of temporal smoothness. A plausible implication is that the method regularizes the memory-conditioned decoder against overfitting to short-range continuity alone, but the paper’s concrete evidence is the ablation improvement rather than a separate theoretical derivation.

## 4. Memory splitting and pruning

The second core component is **memory splitting and pruning**, which addresses redundancy in SAM2’s memory bank. Vanilla SAM2 stores past frame features in a fixed-capacity FIFO memory bank. TSMS-SAM2 argues that, in long surgical sequences, many adjacent frames are highly similar, so the memory can become redundant and overly focused on recent content [2508.05829].

To counter this, the method divides the memory bank into **short-term memory** and **long-term memory**. With a maximum memory size of **7 past frames**, the implementation uses up to **4 short-term frames** and up to **3 long-term frames** [2508.05829]. The paper states that the splitting is inspired by the **Atkinson-Shiffrin memory model**, distinguishing immediate local context from older context that can still be useful after occlusion, disappearance, or viewpoint change.

Pruning is then applied **within each group independently**. Similarity is computed using frame features, and the most redundant frame in each group is removed when the memory bank is full. The short-term group measures similarity relative to the **latest frame** \(f_t\); the long-term group measures similarity relative to the **oldest frame in the group**, described as \(f_{t-(n-1)}\) [2508.05829]. If there are fewer than **7** past frames, no pruning is applied. When the bank reaches **7** frames, one redundant frame is pruned from each group, leaving **5 retained frames** for updating current features [2508.05829].

The similarity metric itself is an ablated design choice. The paper compares **cosine similarity**, **Manhattan (L1)**, **Euclidean (L2)**, **dot product**, **Spearman correlation**, and **Pearson correlation**, reporting that **cosine similarity** gives the best Dice and best overall performance, with **Spearman correlation** as a close second [2508.05829].

The conceptual distinction from pruning solely against the latest frame is important. Older frames may be dissimilar to the current frame yet still encode useful appearance, geometry, or viewpoint information for recovery after reappearance. This suggests that the split-memory design is intended not merely to compress the bank, but to preserve temporal diversity across different horizons.

## 5. Experimental protocol, datasets, and metrics

The experiments are conducted on two surgical instrument benchmarks: **EndoVis2017** and **EndoVis2018** [2508.05829]. For **EndoVis2017**, the paper reports **8 training sequences, 225 frames each**; **8 test sequences, 75 frames each**; and **2 additional hold-out test sequences (9 and 10), 600 frames total**, for **3,000 annotated frames** overall. The instrument classes are **Bipolar Forceps**, **Prograsp Forceps**, **Large Needle Driver**, **Vessel Sealer**, **Grasping Retractor**, and **Monopolar Curved Scissors**. A notable protocol decision is that the authors use **only the two hold-out test sequences (9 and 10)** “to avoid leakage and better evaluate out-of-distribution performance” [2508.05829].

For **EndoVis2018**, the paper reports **15 videos**, **149 frames each**, for **2,235 annotated frames**. The classes are **Bipolar Forceps**, **Prograsp Forceps**, **Large Needle Driver**, **Monopolar Curved Scissors**, **Ultrasound Probe**, **Suction Instrument**, and **Clip Applier**. Following **ISINet** and **SurgSAM2**, the evaluation uses sequences **2, 5, 9, 15** as test videos and the remaining **11 videos** as training data [2508.05829].

The metric suite combines regional, boundary, and sequence-level measures. The paper uses **Jaccard Index / IoU**, **Boundary F1**, **J&F**, **Dice**, and **Challenge IoU (CIoU)**. The reported formulas are the standard ones:
$$
J=\frac{|P\cap G|}{|P\cup G|}, \qquad
F=\frac{2\cdot Precision\cdot Recall}{Precision+Recall},
$$
$$
J\&F=\frac{J+F}{2}, \qquad
Dice=\frac{2|P\cap G|}{|P|+|G|}.
$$
CIoU is described as an object-level metric that accumulates predicted and ground-truth regions over the full video sequence [2508.05829].

## 6. Results and ablation evidence

The paper reports that TSMS-SAM2 achieves the strongest mean Dice scores among the compared methods on both surgical benchmarks [2508.05829].

| Dataset | TSMS-SAM2 | Comparison highlights |
|---|---:|---|
| EndoVis2017 | Dice **95.24 ± 0.96** | SurgSAM2 94.29; SAM2 89.35; MedSAM2 87.37 |
| EndoVis2018 | Dice **86.73 ± 15.46** | SurgSAM2 81.55; SAM2 72.79; MedSAM2 63.83 |
| EndoVis2018 CIoU | **85.1** | SurgSAM2 84.4; MedSAM2 83.7; SAM2 82.2 |

On **EndoVis2017**, TSMS-SAM2 reports **J&F 93.41 ± 1.68**, **J 91.89 ± 1.43**, **F 94.94 ± 3.01**, and **Dice 95.24 ± 0.96**. On **EndoVis2018**, it reports **J&F 84.48 ± 15.37**, **J 84.20 ± 15.17**, **F 84.77 ± 15.74**, and **Dice 86.73 ± 15.46** [2508.05829]. The paper emphasizes that the gains on EndoVis2018 are particularly meaningful because that dataset is described as more difficult due to clutter and domain shift.

The ablation study isolates the two main contributions. With neither temporal sampling nor memory splitting, the baseline yields **J&F 76.28** and **Dice 78.44**. Adding only **memory splitting** raises this to **J&F 84.36** and **Dice 86.69**. Adding only **temporal sampling** yields **J&F 83.13** and **Dice 85.35**. Combining both reaches **J&F 84.48** and **Dice 86.73** [2508.05829]. The reported pattern indicates that both components contribute materially, with memory splitting slightly stronger than temporal sampling in this ablation, and the combination best overall.

The paper also reports qualitative improvements, stating that TSMS-SAM2 better handles instruments missed by SurgSAM2, including **Large Needle Driver**, **Prograsp Forceps**, and **Ultrasound Probe** [2508.05829]. The qualitative interpretation offered is improved tracking stability, better recovery from missed detections, and stronger segmentation in challenging scenes.

## 7. Relation to adjacent SAM2 adaptations, misconceptions, and limitations

TSMS-SAM2 belongs to a broader class of SAM2 adaptations that intervene in memory behavior rather than discarding SAM2’s promptable segmentation backbone. The survey literature frames SAM2’s key distinction from SAM as the addition of streaming memory for video and tracking [2503.12781]. In surgical video specifically, **MA-SAM2** proposes a different memory intervention: a **training-free** strategy with **Context-Aware Memory (CAM)** and **Occlusion-Resilient Memory (ORM)** to replace greedy FIFO behavior, reporting improvements on EndoVis2017 and EndoVis2018 without additional parameters or further training [2507.09577]. This suggests that surgical adaptation of SAM2 has converged on memory management as a principal design axis, although the concrete mechanisms differ.

A common misconception would be to treat TSMS-SAM2 as a wholly new segmentation architecture. The paper does not support that reading. It is more precisely a **SAM2 adaptation** that modifies training-time temporal exposure and inference-time memory selection while retaining the promptable VOST pipeline [2508.05829]. Another misconception would be to classify it as training-free; unlike MA-SAM2, TSMS-SAM2 explicitly fine-tunes selected modules.

The paper also states two limitations. First, the current multi-scale augmentation uses **fixed sampling rates**, such as **1** and **2**. The authors suggest future adaptive sampling strategies based on scene dynamics, object presence or absence, keypoint-based motion cues such as **SIFT**, **FAST**, or **ORB**, shape and motion characteristics, or optical-flow-like information [2508.05829]. Second, the **short-term/long-term memory split is static**, fixed at **4** and **3** frames. The paper proposes future dynamic or learnable memory separation strategies based on similarity, relevance scores, or attention-based updates, potentially closer in spirit to more advanced memory modules in **XMem** or **AOT** [2508.05829].

Taken together, these details place TSMS-SAM2 as a domain-specialized refinement of SAM2 for surgical promptable VOST: it does not alter the foundation model’s role as a promptable video segmenter, but restructures temporal sampling and memory retention so that surgical motion variability and memory redundancy are handled more explicitly.

Source: https://www.emergentmind.com/topics/tsms-sam2