DD-SAM2: Efficient Medical Video Segmentation

• The paper presents DD-SAM2 as an efficient adaptation that fine-tunes SAM2's image encoder and mask decoder with lightweight Depthwise-Dilated Adapters to enhance multi-scale segmentation in medical videos.
• DD-SAM2 leverages a frozen streaming memory module to ensure temporal consistency in challenging cine MRI and echocardiography settings while managing limited training data.
• Evaluations demonstrate that DD-SAM2 outperforms baseline SAM2 with improved Dice scores and minimal additional parameters, highlighting its practicality in medical object tracking.

DD-SAM2 is an efficient adaptation framework for Segment Anything Model 2 (SAM2) for medical object tracking and segmentation, centered on a Depthwise-Dilated Adapter (DD-Adapter) that enhances multi-scale feature extraction with minimal parameter overhead (Xu et al., 19 Jul 2025). It is designed for video-like medical imaging settings, such as cine MRI and echocardiography, where segmentation must remain temporally consistent across frames while preserving SAM2’s prompt-based interaction model and streaming memory mechanism. In the formulation reported by the paper, DD-SAM2 does not replace SAM2’s core video pipeline; rather, it fine-tunes the image encoder and mask decoder while keeping the streaming memory frozen, thereby adapting a general-purpose video segmentation foundation model to limited-data medical video scenarios (Xu et al., 19 Jul 2025).

1. Origin within the SAM-to-SAM2 lineage

The technical context of DD-SAM2 begins with the evolution from SAM to SAM2. SAM established a promptable foundation model for image segmentation, using an image encoder, a prompt encoder, and a mask decoder to generate high-quality masks from prompts such as points, bounding boxes, rough masks, and text (Geetha et al., 2024). Its zero-shot behavior was enabled by the SA-1B dataset, which the comparison paper describes as containing 11 million images and 1.1 billion masks, with 94% of masks having IoU greater than 90% (Geetha et al., 2024).

SAM2 extended this paradigm from single images to videos by introducing temporal reasoning through a memory encoder, a memory bank, and memory attention, together with a Hiera-based multiscale image backbone and spatio-temporal mask propagation (Geetha et al., 2024). This shift is particularly relevant in medical video, where anatomical structures move, deform, disappear under occlusion or low contrast, and reappear later. DD-SAM2 is situated precisely at this intersection: it assumes that SAM2’s streaming memory is already a useful substrate for temporally coherent segmentation, but argues that the image features feeding that memory remain insufficiently specialized for medical motion patterns and fine local boundaries (Xu et al., 19 Jul 2025).

The paper motivates DD-SAM2 by identifying two gaps in prior adaptation work. First, existing adapter-based SAM methods were mainly designed for static images or 3D volume segmentation rather than streaming video. Second, their adapters generally relied on MLP bottlenecks, simple convolutional bottlenecks, or single standard convolution or 3D convolution, and therefore did not explicitly encode multi-scale local context, which the paper considers crucial for small tumors, vessel boundaries, thin anatomical structures, and subtle motion contours in ultrasound and cine MRI (Xu et al., 19 Jul 2025).

2. Architectural composition of DD-SAM2

The DD-SAM2 framework inherits the four-component SAM2 organization: image encoder, streaming memory, prompt encoder, and mask decoder (Xu et al., 19 Jul 2025). Its central architectural decision is selective adaptation. The streaming memory module is preserved rather than redesigned, while DD-Adapters are inserted into the SAM2 image encoder and the mask decoder is fine-tuned (Xu et al., 19 Jul 2025). Frozen components are the image encoder backbone weights except adapter insertions, the prompt encoder, and the streaming memory module (Xu et al., 19 Jul 2025).

The paper describes the streaming memory as comprising a memory encoder, memory attention, and a memory bank stored in a FIFO queue (Xu et al., 19 Jul 2025). In DD-SAM2, this temporal subsystem continues to integrate past frames into current predictions. The adaptation target is therefore upstream of memory attention: improve the quality of encoded frame features without disrupting SAM2’s built-in temporal propagation.

DD-Adapters are inserted after each Transformer block in the SAM2 image encoder, and the default experimental configuration places six DD-Adapters after the first six Transformer blocks (Xu et al., 19 Jul 2025). The authors report that this configuration balances accuracy, computational cost, and avoidance of overfitting (Xu et al., 19 Jul 2025). During testing, the workflow is explicitly video-oriented: the first frame of a video and its annotation are provided; the user prompt is a bounding box on the first frame; SAM2 predicts segmentation for later frames; predicted masks and features are stored in the memory bank; and the memory bank supplies historical context to segment subsequent frames (Xu et al., 19 Jul 2025).

A common misconception is to treat DD-SAM2 as a prompt-free SAM2 variant. The reported method does not do that. It retains bounding-box prompting on the first frame and relies on SAM2’s existing prompt-driven propagation mechanism (Xu et al., 19 Jul 2025). A second misconception is to read “DD” as referring to distractor-aware tracking or detector-driven prompting. In this paper, “DD” refers specifically to “Depthwise-Dilated” adapters (Xu et al., 19 Jul 2025).

3. Depthwise-Dilated Adapter design

The DD-Adapter is the defining technical contribution of the framework. It follows the parameter-efficient fine-tuning bottleneck pattern—compress channels, transform features, restore channels—but replaces a simple bottleneck transform with parallel depthwise-dilated convolution branches intended to capture multi-scale spatial context (Xu et al., 19 Jul 2025).

The adapter is described in three stages. Channel reduction is performed by a point-wise convolution followed by GELU:

$f_a = \mathrm{GELU}(\mathrm{PWConv}(f_i))$

Multi-scale local feature extraction is then carried out through multiple parallel depthwise-dilated convolution branches with different dilation rates:

$$f_m = f_a + \sum_{i=1}^{n} \mathrm{GELU}(\mathrm{DW\text{-}DiConv}_i(f_a))$$

Finally, channel restoration projects the fused features back to the original dimension:

$f_o = f_i + \mathrm{GELU}(\mathrm{PWConv}(f_m))$

These equations are given as the adapter pipeline in the paper (Xu et al., 19 Jul 2025).

The rationale for this design is twofold. Depthwise convolution applies one spatial filter per channel and therefore introduces much fewer parameters and lower compute than standard convolution, while preserving channel-specific structure. Dilated convolution enlarges receptive field without increasing kernel size; the paper gives the example that a $3 \times 3$ kernel with dilation rate 2 behaves like a $5 \times 5$ receptive field (Xu et al., 19 Jul 2025). By combining multiple dilation rates in parallel, the adapter is intended to capture both fine details and broader anatomical context.

The training objective is simple and conventional:

$$\mathcal{L} = \mathcal{L}_{Dice} + \mathcal{L}_{CE}$$

This loss is applied across both evaluation datasets (Xu et al., 19 Jul 2025). The simplicity of the objective is notable because the performance gains are attributed primarily to representation adaptation rather than to task-specific loss engineering.

4. Data, training protocol, and evaluation setting

The framework is evaluated on two medical video datasets with distinct motion and appearance characteristics (Xu et al., 19 Jul 2025). TrackRAD2025 addresses tumor tracking and segmentation in time-resolved sagittal 2D cine-MRI. The dataset includes 477 unlabeled cases and 108 manually labeled cases, and the reported experiments use the 50 publicly available labeled cases, split into 34 train, 4 validation, and 12 test, with splits performed within three cohorts (Xu et al., 19 Jul 2025). EchoNet-Dynamic addresses left ventricle tracking and segmentation in echocardiography and contains 10,030 labeled videos collected at Stanford University Hospital, with an official split of 7,465 train, 1,288 validation, and 1,277 test (Xu et al., 19 Jul 2025).

All SAM-based methods in the comparison use bounding boxes as prompts, and at test time only the first-frame bounding box is provided (Xu et al., 19 Jul 2025). Input size is 1024 × 1024 for SAM2 and DD-SAM2, and 512 × 512 for MedSAM2 and DD-MedSAM2 (Xu et al., 19 Jul 2025). Optimization uses AdamW, with learning rate 1e-4 for adapter modules and 1e-5 for the mask decoder (Xu et al., 19 Jul 2025). On TrackRAD2025, training runs for 30 epochs with the learning rate halved at epoch 15, two videos per step, eight-frame subsequences sampled randomly, and 64 consecutive steps used for augmentation (Xu et al., 19 Jul 2025). On EchoNet-Dynamic, training runs for 10 epochs with the learning rate halved at epoch 5, two videos per step, eight-frame subsequences, and no extra augmentation (Xu et al., 19 Jul 2025).

Evaluation uses Dice, Normalized Surface Dice (NSD), 95th percentile Hausdorff Distance (HD95), and Average Surface Distance (ASD) (Xu et al., 19 Jul 2025). The paper compares DD-SAM2 not only against SAM2-family baselines such as SAM2, MedSAM2, and Medical SAM2, but also against registration-based baselines using ANTs and ANTs SyN, deformable-tracking baselines such as VoxelMorph-unet, VoxelMorph-swinunetr, and TransMorph, and memory-based segmentation baseline XMem (Xu et al., 19 Jul 2025).

5. Reported results, ablations, and efficiency profile

On TrackRAD2025, DD-SAM2 achieves Dice $0.93 \pm 0.04$, NSD $0.45 \pm 0.08$, HD95 $2.37 \pm 0.54$, and ASD $0.74 \pm 0.18$ (Xu et al., 19 Jul 2025). On the same benchmark, SAM2 records Dice $$f_m = f_a + \sum_{i=1}^{n} \mathrm{GELU}(\mathrm{DW\text{-}DiConv}_i(f_a))$$0, NSD $$f_m = f_a + \sum_{i=1}^{n} \mathrm{GELU}(\mathrm{DW\text{-}DiConv}_i(f_a))$$1, HD95 $$f_m = f_a + \sum_{i=1}^{n} \mathrm{GELU}(\mathrm{DW\text{-}DiConv}_i(f_a))$$2, and ASD $$f_m = f_a + \sum_{i=1}^{n} \mathrm{GELU}(\mathrm{DW\text{-}DiConv}_i(f_a))$$3 (Xu et al., 19 Jul 2025). The paper therefore characterizes DD-SAM2 as improving by about $$f_m = f_a + \sum_{i=1}^{n} \mathrm{GELU}(\mathrm{DW\text{-}DiConv}_i(f_a))$$4 Dice over SAM2 on this dataset (Xu et al., 19 Jul 2025).

On EchoNet-Dynamic, DD-SAM2 achieves Dice $$f_m = f_a + \sum_{i=1}^{n} \mathrm{GELU}(\mathrm{DW\text{-}DiConv}_i(f_a))$$5, NSD $$f_m = f_a + \sum_{i=1}^{n} \mathrm{GELU}(\mathrm{DW\text{-}DiConv}_i(f_a))$$6, HD95 $$f_m = f_a + \sum_{i=1}^{n} \mathrm{GELU}(\mathrm{DW\text{-}DiConv}_i(f_a))$$7, and ASD $$f_m = f_a + \sum_{i=1}^{n} \mathrm{GELU}(\mathrm{DW\text{-}DiConv}_i(f_a))$$8 (Xu et al., 19 Jul 2025). The strongest overall result in the table is DD-MedSAM2, which reaches Dice $$f_m = f_a + \sum_{i=1}^{n} \mathrm{GELU}(\mathrm{DW\text{-}DiConv}_i(f_a))$$9, NSD $f_o = f_i + \mathrm{GELU}(\mathrm{PWConv}(f_m))$0, HD95 $f_o = f_i + \mathrm{GELU}(\mathrm{PWConv}(f_m))$1, and ASD $f_o = f_i + \mathrm{GELU}(\mathrm{PWConv}(f_m))$2 (Xu et al., 19 Jul 2025). This suggests that the DD-Adapter is not tied to a single SAM2 initialization, but can function as a general adapter template across SAM2-derived backbones.

The ablation studies identify three particularly important findings. First, the number of adapters matters: performance improves as the number of DD-Adapters increases from 1 to 6, but too many adapters can hurt performance, indicating overfitting or unnecessary complexity; the best reported setting uses 6 adapters (Xu et al., 19 Jul 2025). Second, dilation-rate choice matters: among the tested configurations $f_o = f_i + \mathrm{GELU}(\mathrm{PWConv}(f_m))$3, $f_o = f_i + \mathrm{GELU}(\mathrm{PWConv}(f_m))$4, $f_o = f_i + \mathrm{GELU}(\mathrm{PWConv}(f_m))$5, $f_o = f_i + \mathrm{GELU}(\mathrm{PWConv}(f_m))$6, and $f_o = f_i + \mathrm{GELU}(\mathrm{PWConv}(f_m))$7, the best result is $f_o = f_i + \mathrm{GELU}(\mathrm{PWConv}(f_m))$8 (Xu et al., 19 Jul 2025). Third, the paper attributes degradation under larger dilation sets to a possible grid effect and less stable feature fusion (Xu et al., 19 Jul 2025).

The adapter-variant comparison places DD-Adapter above Adp-a, Adp-b, Adp-c, Adp-d, and LoRA on TrackRAD2025 (Xu et al., 19 Jul 2025). For SAM2 with different adapters, the reported Dice scores are 0.89, 0.90, 0.91, 0.91, 0.92, and 0.93 for Adp-a, Adp-b, Adp-c, Adp-d, LoRA, and DD-SAM2, respectively (Xu et al., 19 Jul 2025). This is one of the clearest pieces of evidence that the gains arise from the depthwise-dilated multi-scale design rather than from generic PEFT alone.

The efficiency claim is based on minimal parameter increase. The paper reports 38.99M parameters for SAM2 and 39.53M for DD-SAM2, corresponding to an increase of 0.54M parameters (Xu et al., 19 Jul 2025). It also reports FPS as slightly lower than SAM2, while emphasizing the accuracy gains and describing the overhead as minimal (Xu et al., 19 Jul 2025). The paper lists FLOPs for SAM2 as 68.47 GMac and for DD-SAM2 as 30.58 GMac (Xu et al., 19 Jul 2025). A plausible implication is that the authors regard the adapter insertion and selective fine-tuning strategy as practically deployable even in settings where full retraining would be unattractive.

Dataset	Method	Reported metrics
TrackRAD2025	DD-SAM2	Dice 0.93±0.04; NSD 0.45±0.08; HD95 2.37±0.54; ASD 0.74±0.18
TrackRAD2025	SAM2	Dice 0.89±0.10; NSD 0.39±0.13; HD95 4.97±5.12; ASD 1.25±0.95
EchoNet-Dynamic	DD-SAM2	Dice 0.96±0.04; NSD 0.58±0.07; HD95 1.72±5.10; ASD 0.43±0.20
EchoNet-Dynamic	DD-MedSAM2	Dice 0.97±0.01; NSD 0.62±0.06; HD95 1.37±1.61; ASD 0.39±0.12

6. Position among SAM2-derived methods and current limitations

DD-SAM2 belongs to a broader class of SAM2 adaptations that preserve the foundation model while altering the pathway by which domain knowledge enters the system. Several neighboring methods illustrate the diversity of this design space. “3DTeethSAM: Taming SAM2 for 3D Teeth Segmentation” adapts SAM2 to 3D dental segmentation through multi-view rendering, a Prompt Embedding Generator, a Mask Refiner, a Mask Classifier, and Deformable Global Attention Plugins, while keeping SAM2 frozen (Lu et al., 12 Dec 2025). “SAM-DAQ: Segment Anything Model with Depth-guided Adaptive Queries for RGB-D Video Salient Object Detection” replaces manual prompts and bulky memory attention with a prompt-free pipeline built from depth-guided parallel adapters and a query-driven temporal memory (Lin et al., 13 Nov 2025). “SSS: Semi-Supervised SAM-2 with Efficient Prompting for Medical Imaging Segmentation” integrates weak-to-strong consistency, Discriminative Feature Enhancement, and a prompt generator based on Physical Constraints with Sliding Window (Zhu et al., 10 Jun 2025). “A Distractor-Aware Memory for Visual Object Tracking with SAM2” modifies SAM2’s memory policy rather than its encoder through a distractor-aware memory split into recent-appearance and distractor-resolving components (Videnovic et al., 2024). “Det-SAM2: Technical Report on the Self-Prompting Segmentation Framework Based on Segment Anything Model 2” adds automatic prompt generation and systems-level optimizations for infinitely long video streams with constant VRAM and RAM usage, but does not fine-tune SAM2 itself (Wang et al., 2024).

Within this landscape, DD-SAM2 is distinct in three respects. First, it is explicitly medical-video-oriented rather than prompt-automation-oriented, self-prompting, or memory-redesign-oriented (Xu et al., 19 Jul 2025). Second, it treats multi-scale local feature modeling as the main unresolved problem in SAM2 medical adaptation, rather than prompt absence, distractor management, or depth fusion (Xu et al., 19 Jul 2025). Third, it preserves SAM2’s streaming memory rather than adapting it, which both simplifies the method and defines one of its limitations (Xu et al., 19 Jul 2025).

The paper states several limitations directly. The current setup is supervised and needs labeled data. The memory module is frozen and not jointly optimized. The adapter introduces no explicit anatomical prior. The method may benefit from future semi-supervised extensions that leverage unlabeled medical video data. Finally, excessive dilation branching can introduce a grid effect, so multi-branch dilation must be chosen carefully (Xu et al., 19 Jul 2025). These limitations also clarify the scope of the contribution: DD-SAM2 is not a universal re-foundation of medical video segmentation, but a parameter-efficient method for injecting multi-scale locality into SAM2 while preserving its temporal machinery.

Taken together, DD-SAM2 shows a specific path for adapting a prompt-based video foundation model to medical tracking and segmentation: retain the streaming memory that already supports temporal coherence, insert lightweight adapters into the encoder to recover medical-domain locality, and fine-tune only the components most directly responsible for feature quality and mask production (Xu et al., 19 Jul 2025). This suggests a broader research direction in which SAM2-based medical systems are differentiated less by wholesale retraining than by the design of tightly scoped, domain-specific adaptation modules.