DD-Adapters for Medical Video Segmentation

• DD-Adapters are lightweight modules integrated into SAM2’s ViT encoder to enable multi-scale local feature extraction for medical video segmentation.
• They adopt a three-stage design—channel reduction, parallel depthwise-dilated convolutions, and channel restoration—ensuring high accuracy with minimal overhead.
• Empirical tests on datasets like TrackRAD2025 and EchoNet-Dynamic demonstrate notable improvements in Dice scores and temporal consistency.

The Depthwise-Dilated Adapter (DD-Adapter) is a parameter-efficient architectural module developed for integration into large-scale segmentation models, specifically enhancing the Segment Anything Model 2 (SAM2) for medical object tracking and segmentation tasks in video. Its design addresses the challenges of modality adaptability, catastrophic forgetting, and resource constraints that arise when fine-tuning prompt-based segmentation models on domain-specific medical imaging datasets. Notably, DD-Adapters introduce channel-efficient, multi-scale local feature extraction capabilities into the frozen image encoder of SAM2 while preserving compatibility with its streaming memory mechanism for real-time, temporally consistent segmentation and tracking performance (Xu et al., 19 Jul 2025).

1. Architecture and Integration

The DD-Adapter is implemented as a lightweight bottleneck interposed after each of the first six transformer blocks within the hierarchical Vision Transformer (ViT) image encoder of SAM2. Each adapter consists of three logical stages: (1) channel reduction via a point-wise convolution (PWConv) followed by a GELU activation; (2) multi-scale local feature extraction through parallel depthwise-dilated convolutions (DW-DiConv), each branch applying a distinct dilation rate for receptive field diversity; (3) channel restoration via a second PWConv with GELU and a residual skip path. Formally, for feature tensor $f_i \in \mathbb{R}^{H \times W \times C}$, the adapter executes:

• $f_a = \mathrm{GELU}(\mathrm{PWConv}_1(f_i))$ (reduce to $C_r$ channels)
• $y_l = \mathrm{GELU}(\mathrm{DW-DiConv}_{r_l}(f_a))$, $l = 1,...,n$ (parallel branches)
• Fuse: $f_m = f_a + \sum_{l=1}^n y_l$
• $f_o' = \mathrm{GELU}(\mathrm{PWConv}_2(f_m))$ (restore channels)
• Output: $f_o = f_i + f_o'$

A single DD-Adapter is inserted after each of the first six transformer blocks in the encoder. During fine-tuning, only the parameters of the adapters and the mask decoder are updated; the base SAM2 encoder and streaming memory remain fixed.

2. Mathematical Formulation

The core operation within DD-Adapters is the depthwise-dilated convolution applied separately in each channel. For a kernel $W$ (size $k \times k$), dilation $f_a = \mathrm{GELU}(\mathrm{PWConv}_1(f_i))$0, and input $f_a = \mathrm{GELU}(\mathrm{PWConv}_1(f_i))$1, the convolution is:

$f_a = \mathrm{GELU}(\mathrm{PWConv}_1(f_i))$2

The full DD-Adapter pipeline per block is mathematically characterized as:

• Channel reduction: $f_a = \mathrm{GELU}(\mathrm{PWConv}_1(f_i))$3
• Multi-scale feature fusion: $f_a = \mathrm{GELU}(\mathrm{PWConv}_1(f_i))$4
• Restoration and residual: $f_a = \mathrm{GELU}(\mathrm{PWConv}_1(f_i))$5

This configuration enables effective probing of local context at multiple spatial resolutions while minimizing parameter and computational overhead.

3. Parameter and Computational Efficiency

The DD-Adapter introduces minimal parameter growth relative to the full backbone+decoder. For the baseline SAM2 "Tiny" variant:

• SAM2 (encoder+mask-decoder): 38.99M parameters
• DD-SAM2 (six DD-Adapters): 39.53M parameters (an overhead of ~0.54M, or ∼1.4%)
• Adapter parameter count: $f_a = \mathrm{GELU}(\mathrm{PWConv}_1(f_i))$6

In terms of inference throughput (batch of eight 1024×1024 frames):

• SAM2: 68.47 GMac, ~33 FPS
• DD-SAM2: 99.05 GMac, ~30 FPS

Analogous efficiency holds for MedSAM2 and DD-MedSAM2. The modest FLOPs increase is counterbalanced by substantial accuracy improvements, especially in medical video contexts.

4. Streaming Memory Synergy

DD-Adapters are designed to function within the unique streaming memory system of SAM2. This mechanism comprises:

1. A memory encoder for fusing previous mask embeddings with current features.
2. Memory attention, incorporating both self- and cross-attention over historical embeddings.
3. A FIFO memory bank maintaining a running context from prior frames.

For each frame, the dataflow proceeds as:

1. The image is encoded (including DD-Adapters), producing features.
2. Prompt encoder processes bounding-box guidance.
3. Memory attention fuses features with historical context.
4. Mask decoder predicts segmentation, and embeddings are appended to the memory bank.

The DD-Adapters enhance the encoder output by injecting learnable, multi-scale local information. This improves the richness of signal available to temporal memory, benefiting both tracking and segmentation quality under shifting appearance and motion dynamics.

5. Training Protocols and Optimization

Training schedules and optimization strategies are dataset-specific. Key parameters include:

• Datasets: TrackRAD2025 (cine-MRI, tumor tracking), EchoNet-Dynamic (echocardiography, left ventricle)
• Loss Functions: Dice loss, cross-entropy; total loss is their sum:

$f_a = \mathrm{GELU}(\mathrm{PWConv}_1(f_i))$7

$f_a = \mathrm{GELU}(\mathrm{PWConv}_1(f_i))$8

$f_a = \mathrm{GELU}(\mathrm{PWConv}_1(f_i))$9

• Optimization: AdamW optimizer, learning rates of $C_r$0 (adapters), $C_r$1 (mask decoder), with step-down schedules.
• Augmentation: Temporal sampling of random 8-frame clips for video consistency. No additional spatial augmentation for large-scale datasets (EchoNet-Dynamic).

Fine-tuning is restricted to adapter modules and the mask decoder, facilitating rapid convergence and minimizing risk of catastrophic forgetting.

6. Quantitative Performance and Ablations

Empirical benchmarking was conducted on TrackRAD2025 and EchoNet-Dynamic. Key results (Dice score, Normalized Surface Dice [NSD], 95th percentile Hausdorff Distance [HD95], Average Surface Distance [ASD]):

Model	TrackRAD2025 Dice	EchoNet-Dynamic Dice
Rigid	0.80 ± 0.13	0.86 ± 0.05
SAM2	0.89 ± 0.10	0.86 ± 0.06
MedSAM2	0.90 ± 0.07	0.91 ± 0.06
DD-SAM2	0.93 ± 0.04	0.96 ± 0.04
DD-MedSAM2	0.93 ± 0.04	0.97 ± 0.01

Ablation experiments reveal:

• Performance increases with adapter count, peaking at six (aligned with the first six transformer blocks).
• The multi-branch dilation set $C_r$2 offers superior trade-offs versus alternatives, balancing effective receptive field growth and avoiding grid artifacts.
• The DD-Adapter architecture outperforms standard MLP or LoRA-based adapters and prior MedSAM2-style model adaptation methods in all major metrics.

7. Position within Medical Video Segmentation

DD-Adapters represent a systematic approach to unlocking domain specialization in prompt-based segmentation models for dynamic, temporally evolving medical image data, with minimal retraining and strong preservation of generalist model knowledge. By leveraging local and multi-scale processing within a frozen high-capacity encoder and augmenting only a thin adaptation layer, they address the limitations of prior modality-specific or static-image-focused adapters. This allows high-fidelity tracking and segmentation without extensive data, computational budget, or risk of catastrophic forgetting. The introduction of DD-SAM2 is, to current knowledge, the first systematic exploration of adapter-based SAM2 fine-tuning tailored to medical video segmentation and tracking domains (Xu et al., 19 Jul 2025).