---
title: GroundingDINO-V1.5 with SAM-2
url: https://www.emergentmind.com/topics/groundingdino-v1-5-with-sam-2
type: topic
---

# GroundingDINO-V1.5 with SAM-2

GroundingDINO v1.5 integrated with SAM-2, as instantiated in the GroundingDINO-US-SAM framework, is a prompt-driven vision-language architecture for text-prompted multi-organ segmentation in ultrasound imaging. The system combines an open-vocabulary object detector (GroundingDINO v1.5), adapted to the medical domain with Low-Rank Adaptation (LoRA), and a frozen, high-fidelity mask segmentation model (SAM-2), operating with free-form text prompts to delineate diverse anatomical regions with high generalizability. The approach is motivated by the challenge of anatomical heterogeneity, diverse imaging protocols, and scarcity of annotated datasets in medical ultrasound, enabling scalable, prompt-driven, zero-shot segmentation across six organ systems using only a small fraction of parameter updates for adaptation [2506.23903].

## 1. Model Architecture and Data Flow

The pipeline processes each image-text pair $(I, T)$ as follows:

1. **Input Preparation**: Images are resized to $800 \times 800$ and paired with an unrestricted free-form text prompt (e.g., "segment renal capsule", "malignant lesion"). 
2. **Text Encoding**: The text $T$ is tokenized and mapped into a sequence of embeddings $(\mathbf{t}_i)_{i=1}^L$ using a frozen BERT-style encoder, producing $d=512$-dimensional representations with context-aware LoRA adapters.
3. **Visual Encoding**: The image $I$ is processed by a frozen backbone (ResNet-50 or Swin-T from GroundingDINO v1.5), outputting multi-scale visual features further refined by a transformer “feature enhancer.”
4. **Cross-Modal Grounding Decoder**: A set of learnable object queries $\{\mathbf{q}_i\}_{i=1}^N$ interact via cross-attention with both visual features and text embeddings. Each decoder layer outputs bounding-box parameters $b_i = (x, y, w, h)$ and open-vocabulary classification logits.
5. **Bounding-Box Head**: A small MLP predicts $\hat b_i \in \mathbb{R}^4$ and logits $\hat y_i \in \mathbb{R}^{L+1}$ (token-by-token match plus "no object").
6. **Mask Prompting to SAM-2**: The top-$K$ predicted bounding-boxes are used as spatial prompts for the frozen SAM-2 mask decoder, which returns high-resolution segmentation masks $\hat M$.

Modifications to the base architectures are as follows:

- **GroundingDINO v1.5**: All pretrained weights are frozen except $\sim$1.7% (LoRA adapters) in feature enhancer, cross-modal decoder, and text encoder modules; the bounding-box regression head remains fully trainable.
- **SAM-2**: Used entirely off-the-shelf, with no fine-tuning; relies on predicted boxes as spatial prompts.

## 2. LoRA Fine-Tuning and Optimization

GroundingDINO adaptation is performed using LoRA [2], injecting low-rank adapters into selected self-attention and feed-forward modules. For any adapted matrix $W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$, the output is
$$
\mathbf{y} = W_0 \mathbf{x} + \Delta W \mathbf{x} \quad \text{where} \quad \Delta W = B A
$$
with $A \in \mathbb{R}^{r \times d_{\text{in}}}$, $B \in \mathbb{R}^{d_{\text{out}} \times r}$, and $r=4$ for all adapted layers.

Attention and FFN calculations are modified as:
- **Attention**: 
  $$
  \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V W_0 + \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V (B A)
  $$
- **FFN**:
  $$
  \mathrm{FFN}(x) = \sigma(x W_{1,0}) W_{2,0} + \sigma(x W_{1,0}) B_2 A_2 + x B_1 A_1
  $$

Only the LoRA adapters and box head are updated during fine-tuning. Training uses the AdamW optimizer (learning rate $5 \times 10^{-4}$, weight decay $1 \times 10^{-4}$), batch size 4, early stopping with patience of 20 epochs, and data augmentation including random flips, cropping, erasing, and scale jittering.

The overall loss combines bounding-box regression, GIoU [5], and focal loss [6] for classification:
- $\lambda_\mathrm{L1}=5$, $\lambda_\mathrm{GIoU}=2$, $\lambda_\mathrm{focal}=2$, $\gamma=2$, $\alpha=0.25$.

## 3. Text Prompt Design and Encoding

The system supports unbounded free-form prompts, facilitating segmentation of arbitrary organ subregions or pathological findings. Example prompts include "segment kidney cortex," "delineate renal capsule," "malignant lesion," "liver parenchyma," and "thyroid nodule." Prompt encoding consists of:

1. Tokenization using GroundingDINO's BERT vocabulary.
2. Embedding to 512-dimensional vectors.
3. Processing through frozen self-attention layers, with selective LoRA adaptation.
4. Cross-attention interaction with decoder queries, enabling open-vocabulary localization.

Sensitivity analyses demonstrate that appropriately worded prompts meaningfully affect segmentation performance. For kidney structures, "segment renal cortex" yields $\mathrm{DSC}=71.95\pm10\%$, $\mathrm{IoU}=57.14\pm12\%$, while "segment kidney capsule" achieves $\mathrm{DSC}=91.74\pm5\%$, $\mathrm{IoU}=85.10\pm8\%$.

## 4. Datasets, Training, and Evaluation

The system is trained and validated on 15 datasets spanning six anatomies (breast, thyroid, liver, prostate, kidney, muscle), with a total of 16,805 images, and is tested on three held-out datasets ("BUSBRA," "TNSCUI," "Luminous") for out-of-distribution generalization (2,808 images). 

Evaluation metrics are:
- **Intersection over Union (IoU)**: $\mathrm{IoU} = \frac{|P \cap G|}{|P \cup G|}$
- **Dice Similarity Coefficient (DSC)**: $\mathrm{DSC} = \frac{2\,|P \cap G|}{|P| + |G|}$

Detection AP is not reported: the primary focus is final mask quality.

## 5. Quantitative Results

Comprehensive experiments demonstrate the following segmentation performance (DSC % / IoU %) on representative datasets (all values from [2506.23903]):

### Ablation of Segmentation Back-End

| Organ          | Dataset   | DINO+SAM2 (no FT) | FT DINO+MedSAM | FT DINO+SAM2 |
|----------------|-----------|-------------------|----------------|--------------|
| Breast         | BrEaST    | 23.9 / 18.1       | 49.6 / 37.3    | 76.7 / 66.2  |
| Breast         | BUID      | 44.1 / 36.1       | 61.4 / 49.4    | 82.9 / 75.8  |
| Prostate       | MicroSeg  | 61.4 / 47.5       | 56.8 / 42.8    | 88.6 / 81.2  |
| Average        |           | 34.5 ±26 / 26.6±24| 50.4±27/38.5±23| 74.0±24/65.7±24 |

Fine-tuning GroundingDINO with LoRA and coupling to frozen SAM-2 is optimal across almost all anatomies.

### Comparison With SOTA on Seen Datasets

| Method            | DSC Mean | IoU Mean   |
|-------------------|----------|------------|
| UniverSeg         | 47.3     | 35.8       |
| BiomedParse       | 62.0     | 53.4       |
| SAMUS             | 60.0     | 48.0       |
| MedCLIP-SAM       | 62.8     | 50.5       |
| MedCLIP-SAMv2     | 68.9     | 59.1       |
| Ours              | 79.0     | 71.1       |

### Generalization to Unseen Datasets

| Method            | DSC Mean | IoU Mean   |
|-------------------|----------|------------|
| UniverSeg         | 26.6     | 18.3       |
| BiomedParse       | 44.0     | 36.8       |
| SAMUS             | 68.0     | 57.4       |
| MedCLIP-SAM       | 56.6     | 42.5       |
| Ours              | 72.8     | 68.5       |

The segmentation system achieves strong generalization, maintaining robust performance on unseen distributions without further model updates.

## 6. Discussion, Strengths, and Limitations

### Strengths

- Supports zero-click, zero-point, text-prompted segmentation across six organ systems.
- Only $\sim$1.7% of GroundingDINO parameters updated; all other weights, including SAM-2, are frozen.
- Efficient, real-time inference ($\sim$0.33 s/image on Titan V).
- Outperforms state-of-the-art segmentation methods (UniverSeg, MedSAM, MedCLIP-SAM, BiomedParse, SAMUS) on most seen and unseen datasets.

### Weaknesses

- Performance is contingent on the quality of the image-text alignment learned during pretraining; rare or highly specialized prompts may degrade performance.
- Segmentation mask fidelity is tied to bounding-box proposal accuracy; if predicted boxes are poor, SAM-2 outputs degrade.
- Absence of explicit shape priors or local refinement modules affects complex topology handling.

## 7. Future Directions and Broader Applications

Fine-tuning of LoRA adapters alone offers a lightweight pathway to adapt GroundingDINO-US-SAM for non-ultrasound modalities:
- **CT/MRI**: Segmentation by prompt (e.g., "segment liver tumor") with minimal LoRA adaptation.
- **X-ray**: Text prompts such as "rib," "lung nodule" possible with small-scale LoRA fine-tuning.
- **Digital Pathology**: Prompts for "tumor region," "immune cell aggregate."
- **Clinical Integration**: Incorporation with voice-driven prompts, PACS systems, and real-time procedural guidance.

These extensions leverage the parameter efficiency and open-vocabulary design of the core system, facilitating scalable model deployment with minimal dependence on large annotated datasets [2506.23903].

Source: https://www.emergentmind.com/topics/groundingdino-v1-5-with-sam-2