---
title: DD-Adapters for Medical Video Segmentation
url: https://www.emergentmind.com/topics/dd-adapters
type: topic
---

# DD-Adapters for Medical Video Segmentation

The Depthwise-Dilated Adapter (DD-Adapter) is a parameter-efficient architectural module developed for integration into large-scale segmentation models, specifically enhancing the Segment Anything Model 2 (SAM2) for medical object tracking and segmentation tasks in video. Its design addresses the challenges of modality adaptability, catastrophic forgetting, and resource constraints that arise when fine-tuning prompt-based segmentation models on domain-specific medical imaging datasets. Notably, DD-Adapters introduce channel-efficient, multi-scale local feature extraction capabilities into the frozen image encoder of SAM2 while preserving compatibility with its streaming memory mechanism for real-time, temporally consistent segmentation and tracking performance [2507.14613].

## 1. Architecture and Integration

The DD-Adapter is implemented as a lightweight bottleneck interposed after each of the first six transformer blocks within the hierarchical Vision Transformer (ViT) image encoder of SAM2. Each adapter consists of three logical stages: (1) channel reduction via a point-wise convolution (PWConv) followed by a GELU activation; (2) multi-scale local feature extraction through parallel depthwise-dilated convolutions (DW-DiConv), each branch applying a distinct dilation rate for receptive field diversity; (3) channel restoration via a second PWConv with GELU and a residual skip path. Formally, for feature tensor $f_i \in \mathbb{R}^{H \times W \times C}$, the adapter executes:

- $f_a = \mathrm{GELU}(\mathrm{PWConv}_1(f_i))$  (reduce to $C_r$ channels)
- $y_l = \mathrm{GELU}(\mathrm{DW-DiConv}_{r_l}(f_a))$, $l = 1,...,n$ (parallel branches)
- Fuse: $f_m = f_a + \sum_{l=1}^n y_l$
- $f_o' = \mathrm{GELU}(\mathrm{PWConv}_2(f_m))$ (restore channels)
- Output: $f_o = f_i + f_o'$

A single DD-Adapter is inserted after each of the first six transformer blocks in the encoder. During fine-tuning, only the parameters of the adapters and the mask decoder are updated; the base SAM2 encoder and streaming memory remain fixed.

## 2. Mathematical Formulation

The core operation within DD-Adapters is the depthwise-dilated convolution applied separately in each channel. For a kernel $W$ (size $k \times k$), dilation $d$, and input $f$, the convolution is:

$$(DW\!-\!DiConv(f))_{i,j,c} = \sum_{u=0}^{k-1}\sum_{v=0}^{k-1} W_{u,v,c} \cdot f_{i+u \cdot d, j+v \cdot d, c}$$

The full DD-Adapter pipeline per block is mathematically characterized as:
- Channel reduction: $f_a = \mathrm{GELU}(\mathrm{PWConv}_1(f_i))$
- Multi-scale feature fusion: $f_m = f_a + \sum_{l=1}^n \mathrm{GELU}(\mathrm{DW-DiConv}_{r_l}(f_a))$
- Restoration and residual: $f_o = f_i + \mathrm{GELU}(\mathrm{PWConv}_2(f_m))$

This configuration enables effective probing of local context at multiple spatial resolutions while minimizing parameter and computational overhead.

## 3. Parameter and Computational Efficiency

The DD-Adapter introduces minimal parameter growth relative to the full backbone+decoder. For the baseline SAM2 "Tiny" variant:
- SAM2 (encoder+mask-decoder): 38.99M parameters
- DD-SAM2 (six DD-Adapters): 39.53M parameters (an overhead of ~0.54M, or ∼1.4%)
- Adapter parameter count: $P_\text{adapter} \approx 2(C \times C_r) + n(k^2 \times C_r)$

In terms of inference throughput (batch of eight 1024×1024 frames):
- SAM2: 68.47 GMac, ~33 FPS
- DD-SAM2: 99.05 GMac, ~30 FPS

Analogous efficiency holds for MedSAM2 and DD-MedSAM2. The modest FLOPs increase is counterbalanced by substantial accuracy improvements, especially in medical video contexts.

## 4. Streaming Memory Synergy

DD-Adapters are designed to function within the unique streaming memory system of SAM2. This mechanism comprises:
1. A memory encoder for fusing previous mask embeddings with current features.
2. Memory attention, incorporating both self- and cross-attention over historical embeddings.
3. A FIFO memory bank maintaining a running context from prior frames.

For each frame, the dataflow proceeds as:
1. The image is encoded (including DD-Adapters), producing features.
2. Prompt encoder processes bounding-box guidance.
3. Memory attention fuses features with historical context.
4. Mask decoder predicts segmentation, and embeddings are appended to the memory bank.

The DD-Adapters enhance the encoder output by injecting learnable, multi-scale local information. This improves the richness of signal available to temporal memory, benefiting both tracking and segmentation quality under shifting appearance and motion dynamics.

## 5. Training Protocols and Optimization

Training schedules and optimization strategies are dataset-specific. Key parameters include:
- **Datasets**: TrackRAD2025 (cine-MRI, tumor tracking), EchoNet-Dynamic (echocardiography, left ventricle)
- **Loss Functions**: Dice loss, cross-entropy; total loss is their sum:
  $$\mathcal{L}_{\mathrm{Dice}} = 1 - \frac{2\sum_i p_i g_i}{\sum_i p_i + \sum_i g_i}$$
  $$\mathcal{L}_{\mathrm{CE}} = -\sum_i [g_i \log p_i + (1-g_i)\log(1-p_i)]$$
  $$\mathcal{L} = \mathcal{L}_{\mathrm{Dice}} + \mathcal{L}_{\mathrm{CE}}$$
- **Optimization**: AdamW optimizer, learning rates of $1 \times 10^{-4}$ (adapters), $1 \times 10^{-5}$ (mask decoder), with step-down schedules.
- **Augmentation**: Temporal sampling of random 8-frame clips for video consistency. No additional spatial augmentation for large-scale datasets (EchoNet-Dynamic).

Fine-tuning is restricted to adapter modules and the mask decoder, facilitating rapid convergence and minimizing risk of catastrophic forgetting.

## 6. Quantitative Performance and Ablations

Empirical benchmarking was conducted on TrackRAD2025 and EchoNet-Dynamic. Key results (Dice score, Normalized Surface Dice [NSD], 95th percentile Hausdorff Distance [HD95], Average Surface Distance [ASD]):

| Model         | TrackRAD2025 Dice | EchoNet-Dynamic Dice |
|---------------|-------------------|----------------------|
| Rigid         | 0.80 ± 0.13       | 0.86 ± 0.05          |
| SAM2          | 0.89 ± 0.10       | 0.86 ± 0.06          |
| MedSAM2       | 0.90 ± 0.07       | 0.91 ± 0.06          |
| DD-SAM2       | 0.93 ± 0.04       | 0.96 ± 0.04          |
| DD-MedSAM2    | 0.93 ± 0.04       | 0.97 ± 0.01          |

Ablation experiments reveal:
- Performance increases with adapter count, peaking at six (aligned with the first six transformer blocks).
- The multi-branch dilation set $\{1,3\}$ offers superior trade-offs versus alternatives, balancing effective receptive field growth and avoiding grid artifacts.
- The DD-Adapter architecture outperforms standard MLP or LoRA-based adapters and prior MedSAM2-style model adaptation methods in all major metrics.

## 7. Position within Medical Video Segmentation

DD-Adapters represent a systematic approach to unlocking domain specialization in prompt-based segmentation models for dynamic, temporally evolving medical image data, with minimal retraining and strong preservation of generalist model knowledge. By leveraging local and multi-scale processing within a frozen high-capacity encoder and augmenting only a thin adaptation layer, they address the limitations of prior modality-specific or static-image-focused adapters. This allows high-fidelity tracking and segmentation without extensive data, computational budget, or risk of catastrophic forgetting. The introduction of DD-SAM2 is, to current knowledge, the first systematic exploration of adapter-based SAM2 fine-tuning tailored to medical video segmentation and tracking domains [2507.14613].

Source: https://www.emergentmind.com/topics/dd-adapters