DD-Adapter for Medical Video Segmentation

• DD-Adapter is a parameter-efficient, multi-scale adapter that employs depthwise-dilated convolutions to enhance SAM2 for medical video object tracking and segmentation.
• It uses point-wise convolutions followed by parallel depthwise-dilated branches to capture local context at multiple scales with minimal additional parameters.
• Empirical results on TrackRAD2025 and EchoNet-Dynamic demonstrate improved Dice scores, NSD, and reduced HD95 and ASD, highlighting its efficacy in challenging medical imaging scenarios.

Searching arXiv for the specified paper and closely related work on SAM2 and adapter-based medical segmentation. arXiv query: (Xu et al., 19 Jul 2025) The Depthwise-Dilated Adapter (DD-Adapter) is a bottleneck-style adapter introduced to adapt the Segment Anything Model 2 (SAM2) to medical video object tracking and segmentation with a very small number of trainable parameters while preserving SAM2’s pre-trained generalization (Xu et al., 19 Jul 2025). In the reported framework, termed DD-SAM2, the adapter is inserted into the SAM2 image encoder and combines point-wise convolutions with parallel depthwise-dilated convolution branches to enhance multi-scale feature extraction with minimal parameter overhead. The stated motivation is that existing SAM and SAM2 adapters are mostly designed for static images and typically use plain bottleneck MLPs or simple convolutional designs that do not explicitly model multi-scale local context, which is especially important for small, deformable, or low-contrast medical targets such as tumors and the left ventricle.

1. Problem setting and rationale

The DD-Adapter is motivated by medical-imaging scenarios in which the input is inherently temporal, such as cine MRI and echocardiography, and the task requires tracking the same object across frames (Xu et al., 19 Jul 2025). In this setting, full fine-tuning of SAM2 is described as expensive and as posing a risk of catastrophic forgetting of the pre-trained model’s broad segmentation ability. The reported alternative is parameter-efficient fine-tuning in which most of SAM2 remains frozen.

The practical difficulty arises from the characteristics of medical targets emphasized in the source paper: small size, fine boundaries, low contrast, shape changes over time, and motion-dependent appearance variation. The authors state that existing adapter methods do not sufficiently incorporate multi-scale local features, even though such features are crucial for these structures. DD-Adapter is therefore designed to inject local, multi-scale medical-domain information into SAM2 while keeping most of the model frozen. This is meant to support efficient domain adaptation with minimal parameter growth and reduced forgetting.

A common misconception is that prompt-based segmentation models can be transferred to medical video simply by reusing image-oriented adapters. The reported position is more specific: adapter designs developed and evaluated mainly for static images are not, by themselves, an adequate answer for SAM2’s streaming-memory-based video segmentation. The contribution is framed accordingly as an adaptation strategy for medical video object tracking and segmentation rather than as a general replacement for all prior adapter families.

2. Architectural formulation of the DD-Adapter

The DD-Adapter is described as a bottleneck-style adapter, but unlike a standard MLP adapter it uses point-wise convolutions and depthwise dilated convolutions (Xu et al., 19 Jul 2025). Its pipeline is organized into three stages: channel dimension reduction, multi-scale local feature extraction, and dimension recovery.

If the input feature map is $f_i$, the first transformation is

$f_a = \mathrm{GELU}(\mathrm{PWConv}(f_i))$

which reduces channels while preserving semantic content. The reduced feature map is then processed by several parallel depthwise-dilated convolution branches with different dilation rates, and the outputs are fused with a residual connection:

$$f_m = f_a + \sum_{n} \mathrm{GELU}(\mathrm{DW\text{-}DiConv}(f_a))$$

where $n$ is the number of branches and each branch uses a different dilation rate to capture context at different receptive fields. Finally, the representation is projected back to the original channel size:

$f_o = f_i + \mathrm{GELU}(\mathrm{PWConv}(f_m))$

In this formulation, the adapter learns a residual correction to the frozen representation. The use of depthwise convolution is justified as lightweight compared with standard convolution, while dilation enlarges the receptive field without a proportional increase in parameters. The paper explicitly notes that a $3 \times 3$ kernel with dilation rate 2 expands the effective receptive field to $5 \times 5$. This mechanism is the basis for the claim that DD-Adapter captures multi-scale local structure without heavy computation.

The stated significance of this design is that the depthwise dilated branches can detect patterns at multiple spatial scales. This is especially relevant in medical videos, where the object may change size, shape, or boundary clarity across frames. The adapter therefore complements SAM2’s hierarchical image encoder by adding a local multi-scale inductive bias that the vanilla transformer blocks do not enforce explicitly. This suggests that the method is aimed less at replacing SAM2’s global representation and more at correcting its local feature sensitivity under domain shift.

3. Placement within SAM2 and interaction with streaming memory

SAM2 is described as containing four main parts: image encoder, streaming memory, prompt encoder, and mask decoder (Xu et al., 19 Jul 2025). The streaming memory module includes a memory encoder, memory attention, and memory bank. It stores information from prior frames in a FIFO queue and uses self-/cross-attention to refine the current frame embedding using temporal context.

Within this architecture, DD-Adapters are inserted into the image encoder, specifically after each Transformer block in the Hiera-based image encoder. Figure 1(b), as summarized in the source material, shows the adapter placed after Transformer block 1, block 2, block 3, and so forth. In the default configuration, adapters are not inserted into every block. Instead, the model uses six DD-Adapters with dilation rates 1 and 3, attached after the first six Transformer blocks. The paper reports that this is a good tradeoff between accuracy and efficiency and that putting adapters into all blocks is not optimal.

The DD-Adapter is not inserted into the streaming memory module. Rather, the streaming memory parameters are frozen, and fine-tuning is done on the DD-Adapters and the mask decoder. The image encoder backbone is frozen, and the prompt encoder is frozen as well. In functional terms, this means that DD-Adapter modifies the frame-level image features before they flow into the memory, prompt, and mask-decoding pathway, while temporal memory behavior remains fixed. A plausible implication is that the method treats temporal propagation as a stable pre-trained prior and concentrates adaptation capacity on the spatial representation supplied to that prior.

For efficiency analysis, the input video shape is defined as $1024 \times 1024 \times 8 \times 3$ for SAM2 and DD-SAM2, and $512 \times 512 \times 8 \times 3$ for MedSAM2 and DD-MedSAM2. This corresponds to 8 frames with 3 channels at the stated spatial resolution.

4. Fine-tuning regime and parameter-efficient adaptation

The training strategy is explicitly parameter-efficient (Xu et al., 19 Jul 2025). During fine-tuning, the DD-Adapters are trainable and the mask decoder is trainable, while the image encoder backbone, prompt encoder, and streaming memory are frozen. The paper presents this as the main PEFT strategy for reducing compute and avoiding catastrophic forgetting.

Optimization uses an equal-weight combination of Dice loss and cross-entropy loss. The optimizer is AdamW. The learning rate is $1 \times 10^{-4}$ for adapter modules and $f_a = \mathrm{GELU}(\mathrm{PWConv}(f_i))$0 for the mask decoder. The training schedule is dataset-specific: TrackRAD2025 is trained for 30 epochs with the learning rate halved at epoch 15, whereas EchoNet-Dynamic is trained for 10 epochs with the learning rate halved at epoch 5.

The sampling strategy also depends on the dataset. For TrackRAD2025, the procedure uses 2 videos per step, 8-frame subsequences sampled with random starting frames, and repetition for 64 consecutive training steps for augmentation. For EchoNet-Dynamic, the setup uses 2 videos per step, 8-frame subsequences per video, effective batch size 2, and no extra augmentation because the dataset is already large.

Prompting and evaluation follow a constrained protocol. Bounding boxes are used as prompts, following MedSAM2 practice, because boxes are described as clearer and more reliable than clicks for organs and lesions. At test time, only the first frame’s bounding box annotation is provided, and the model tracks and segments the target through the rest of the video using the memory bank. The source characterizes this as clinically realistic and as a test of whether the streaming memory can maintain object identity over time.

5. Empirical behavior on medical video benchmarks

The experimental evidence is reported on two tasks: tumor tracking and segmentation on TrackRAD2025, and left ventricle tracking and segmentation on EchoNet-Dynamic (Xu et al., 19 Jul 2025). In both cases, DD-Adapter is evaluated within SAM2 and MedSAM2 variants.

On TrackRAD2025, the reported Dice scores are 0.89 ± 0.10 for SAM2, 0.93 ± 0.04 for DD-SAM2, 0.90 ± 0.05 for MedSAM2, and 0.93 ± 0.04 for DD-MedSAM2. Additional reported metrics for DD-SAM2 are NSD 0.45 ± 0.08, HD95 2.37 ± 0.54, and ASD 0.74 ± 0.18. Compared with MedSAM2, DD-MedSAM2 improves Dice by about 0.03 and also reduces HD95 and ASD.

On EchoNet-Dynamic, the gains are larger. The reported Dice scores are 0.86 ± 0.06 for SAM2, 0.91 ± 0.06 for MedSAM2, 0.96 ± 0.04 for DD-SAM2, and 0.97 ± 0.01 for DD-MedSAM2. For DD-MedSAM2, the reported NSD is 0.62 ± 0.06, HD95 is 1.37 ± 1.61, and ASD is 0.39 ± 0.12. The paper emphasizes that DD-MedSAM2 more than doubles XMem’s NSD and clearly outperforms the non-DD versions, which is presented as evidence that the adapter is especially beneficial in difficult ultrasound motion conditions.

Dataset / task	Model	Reported Dice
TrackRAD2025 tumor tracking and segmentation	SAM2	0.89 ± 0.10
TrackRAD2025 tumor tracking and segmentation	DD-SAM2	0.93 ± 0.04
TrackRAD2025 tumor tracking and segmentation	MedSAM2	0.90 ± 0.05
TrackRAD2025 tumor tracking and segmentation	DD-MedSAM2	0.93 ± 0.04
EchoNet-Dynamic left ventricle tracking and segmentation	SAM2	0.86 ± 0.06
EchoNet-Dynamic left ventricle tracking and segmentation	MedSAM2	0.91 ± 0.06
EchoNet-Dynamic left ventricle tracking and segmentation	DD-SAM2	0.96 ± 0.04
EchoNet-Dynamic left ventricle tracking and segmentation	DD-MedSAM2	0.97 ± 0.01

The two datasets also define the intended operating regime. TrackRAD2025 is described as time-resolved sagittal 2D cine-MRI for MRI-guided radiotherapy, with 50 labeled cases used in the paper and split into train, validation, and test within cohorts. EchoNet-Dynamic is described as cardiac ultrasound video with 10,030 labeled videos and an official train, validation, and test split. The stated interpretation is that DD-Adapter helps on tumors because they can be small, deformable, and weakly contrasted, and helps on echocardiography because of rapid motion, variable appearance, and low-resolution ultrasound. This suggests that the core benefit is not tied to a single modality but to a recurring combination of local ambiguity and temporal variation.

6. Ablations, comparisons, and reported efficiency

The ablation studies examine the number of adapters, the dilation rates, and the adapter family used (Xu et al., 19 Jul 2025). For the number of DD-Adapters, the default is 6. Performance generally improves as the number increases from 1 to 6, but adding more than 6 degrades results. The reported conclusion is that inserting adapters into all transformer blocks can cause overfitting or unnecessary complexity, so 6 adapters is selected as the best balance.

For dilation configurations on TrackRAD2025, the paper tests $f_a = \mathrm{GELU}(\mathrm{PWConv}(f_i))$1, $f_a = \mathrm{GELU}(\mathrm{PWConv}(f_i))$2, $f_a = \mathrm{GELU}(\mathrm{PWConv}(f_i))$3, $f_a = \mathrm{GELU}(\mathrm{PWConv}(f_i))$4, and $f_a = \mathrm{GELU}(\mathrm{PWConv}(f_i))$5. The best result is reported for $f_a = \mathrm{GELU}(\mathrm{PWConv}(f_i))$6, which is then used in the final model. The accompanying interpretation is that too small a dilation range may limit multi-scale context, whereas too many dilation branches can hurt performance because of the grid effect and noise in fusion.

The comparative adapter analysis is organized around several prior families: Adp-a, the standard MLP bottleneck adapter from Med-SA; Adp-b, the SAM-Med2D-style adapter; Adp-c, the MA-SAM-style adapter with 3D conv; Adp-d, the 3DSAM-Adapter-style depthwise 3D conv adapter; and LoRA. Table 4 is summarized as showing DD-Adapter outperforming all compared adapter variants and LoRA on both SAM2 and MedSAM2. The standard MLP adapter is reported as the weakest, convolutional adapters improve performance, and DD-Adapter is best overall. The paper uses this pattern to support the claim that explicit multi-scale locality matters.

The reported efficiency figures quantify the parameter and compute overhead. SAM2 is listed at 38.99M parameters and 68.47 GMac, while DD-SAM2 is listed at 39.53M parameters and 99.05 GMac. MedSAM2 is listed at 38.99M parameters and 17.43 GMac, while DD-MedSAM2 is listed at 39.53M parameters and 25.07 GMac. The reported increase is therefore about 0.54M parameters, accompanied by a small FPS drop that the authors consider acceptable given the gains.

Model	Parameters	GMac
SAM2	38.99M	68.47
DD-SAM2	39.53M	99.05
MedSAM2	38.99M	17.43
DD-MedSAM2	39.53M	25.07

These ablations are central to distinguishing DD-Adapter from simpler PEFT mechanisms. The paper’s comparison is not merely between “adapter” and “no adapter,” but between locality-agnostic bottlenecks, single-path convolutional alternatives, 3D-convolution-based designs, and a depthwise-dilated multi-branch design targeted to SAM2. A plausible implication is that the reported advantage derives from the combination of locality, multi-scale receptive fields, and low parameter cost rather than from the presence of an adapter module in the abstract.

7. Scope, claims, and interpretive boundaries

The paper presents DD-SAM2 as an efficient adaptation framework for SAM2 that fully exploits SAM2’s streaming memory for medical video object tracking and segmentation (Xu et al., 19 Jul 2025). It further states that, to the best of the authors’ knowledge, the work provides an initial attempt at systematically exploring adapter-based SAM2 fine-tuning for medical video segmentation and tracking. This is a priority claim and should be read as the authors’ characterization of the literature rather than as a settled taxonomic fact.

Within the boundaries of the reported experiments, the central claim is that a lightweight, multi-scale, convolutional adapter can improve medical video adaptation without full retraining of SAM2. The article’s evidence is strongest where the data are explicit: six DD-Adapters after the first six transformer blocks, dilation rates 1 and 3, frozen streaming memory and frozen encoder, trainable adapters and mask decoder, box prompting, and first-frame-only annotation at test time. The consistency of improvements across TrackRAD2025 and EchoNet-Dynamic supports the narrower conclusion that parameter-efficient, multi-scale adaptation is effective for the two examined medical video tasks.

At the same time, some broader interpretations remain inferential. The source data do not establish that DD-Adapter is optimal for every medical modality, every SAM2 scale, or every prompting regime. They do, however, support a more focused reading: when the target exhibits small size, fine boundaries, low contrast, temporal deformation, or motion-dependent appearance variation, explicitly modeling multi-scale local context inside a frozen-pretrained SAM2 pipeline can be advantageous. In that sense, DD-Adapter occupies a specific methodological niche at the intersection of PEFT, prompt-based segmentation, and streaming-memory video modeling.