---
title: 'Grounded-SAM: Open-World Text-to-Mask Model'
url: https://www.emergentmind.com/topics/grounded-sam
type: topic
---

# Grounded-SAM: Open-World Text-to-Mask Model

Grounded-SAM is a model assembly paradigm integrating open-vocabulary grounding and segmentation capabilities by composing Grounding DINO, an open-set language-driven object detector, and the Segment Anything Model (SAM), a promptable segmentation network based on Vision Transformers. This architecture is designed to realize direct text-to-mask functionality, enabling detection and segmentation of arbitrary user-specified objects from natural-language prompts with robust zero-shot generalization. Grounded-SAM has been validated for open-world segmentation, data-frugal self-supervised learning, and as a modular core for diverse downstream tasks, including automatic dense annotation, controllable image editing, and prompt-based 3D pose reconstruction [2401.14159][2406.05776].

## 1. Architectural Composition and Text-to-Mask Pipeline

Grounded-SAM explicitly composes two state-of-the-art open-world models:

- **Grounding DINO**: Accepts image $I$ and natural-language text $T$ (e.g., “soldier”), generating set $\{(b_i, p_i)\}$ of object bounding boxes $b_i$ and associated confidence scores $p_i$ by cross-modal attention between image and text features.
- **Segment Anything Model (SAM)**: Receives $I$ and prompt(s) (points, boxes, or polygons), then outputs high-resolution segmentation mask(s) $m_i$. The mask decoder combines an image tokenization via ViT with the prompt embedding $e_{\text{prompt}}$ for each region.

The operational pipeline is strictly modular:

1. The text prompt $T$ is processed by Grounding DINO to yield up to $N$ regions $b_i$ with confidence $p_i$.
2. Proposals are filtered by $p_i\geq t_c$ (with $t_c$ typically in $[0.15, 0.3]$); at least one candidate per image is guaranteed by fall-back to $\operatorname{argmax}_{i} p_i$.
3. Each $b_i$ is transformed into a box prompt and fed, together with $I$, to SAM, yielding segmentation masks $M_i$.
4. The selected $M_i$ (highest-confidence or union) serve as output or as “pseudo-labels” for downstream model training.

Notably, both Grounding DINO and SAM are used in their frozen, pre-trained state; no parameter finetuning is performed during assembly. Adaptation for new scenarios is limited to prompt selection and confidence thresholding [2401.14159][2406.05776].

## 2. Mathematical Formalisms and Evaluation

Grounded-SAM’s architecture and learning are analytically characterized as follows:

- **Detection loss (Grounding DINO)**:

  $$
  \mathrm{cost}(i,j) = -\log p_i(c_j) + \lambda_1 \|b_i - g_j\|_1 + \lambda_2 (1-\mathrm{IoU}(b_i,g_j))
  $$
  $$
  \mathcal{L}_\mathrm{det} = \sum_{j=1}^N \bigl[ \mathrm{FocalLoss}(p_{\sigma(j)},c_j) + \lambda_1\|b_{\sigma(j)}-g_j\|_1 + \lambda_2(1-\mathrm{IoU}(b_{\sigma(j)},g_j)) \bigr]
  $$

- **Mask loss (SAM)**:
  $$
  \mathcal{L}_\mathrm{mask}(m_i, M_i^*) = \mathrm{BCE}(m_i, M_i^*) + \alpha\,\mathrm{Dice}(m_i, M_i^*)
  $$

- **Self-supervised pseudo-label loss**:
  $$
  \mathcal{L}_\mathrm{pseudo}(\theta) = \frac{1}{N} \sum_{i=1}^N \ell\left(p_i(\theta), \hat{y}_i\right)
  $$
  where $\hat{y}_i$ are GSAM-derived masks, $p_i(\theta)$ is the foreground probability from downstream detector, and $\ell$ is pixel-wise cross-entropy.

- **Evaluation (e.g., Segmentation in the Wild, CPD1K)**:
  - Mean Average Precision ($\mathrm{mAP}$) over open-vocabulary classes
  - Weighted F-measure $F^w_\beta$ (with $\beta^2=1$)
  - Mean Absolute Error (MAE), S-measure ($S$), E-measure ($E_\phi$)
  - False-Positive Rate (FPR), True-Negative Rate (TNR) on background scenes

GSAM achieves competitive performance, e.g., $48.7\,\mathrm{mAP}$ on SGinW (Grounding DINO-base + SAM-huge) [2401.14159], and, for data-efficient camouflaged human detection, optimized pseudo-labeling allows the downstream HitNet model (with 30 images) to match “30-shot” supervised baselines within 10% of full training on CPD1K [2406.05776].

## 3. Pseudo-Labeling, Self-Supervised Frugal Learning, and Post-Processing

Pseudo-labeling in Grounded-SAM is central for label-efficient downstream transfer:

- **GSAM Pseudo-Labeling Algorithm**:
  ```python
  Input: image I, text prompt T, threshold t_c
  B, p = GroundingDINO(I, prompt=T)
  keep = {i | p_i ≥ t_c} if keep else {argmax_i p_i}
  Masks = [SAM(I, box=B_i) for i in keep]
  Output: choose highest-confidence M* or union(Masks)
  ```

- **Confidence Thresholding**: Only masks above detector confidence $t_c$ are retained.
- **Loss Re-weighting**: Each pseudo-label loss is scaled by normalized $p_i$; $L'_i=p'_i L_i$.

Comparative experiments on various pseudo-label strategies (GSAM, MAT-inpainting error, HitNet-animal) establish that GSAM supplies higher-quality, high-recall pseudo-labels for previously unseen classes (e.g., camouflaged humans) than both inpainting-based and naive class transfer approaches.

Optimized GSAM pseudo-labels (with selection/filtering and reweighting) allow transformer-based COD architectures (e.g., HitNet) to close the performance gap to fully supervised learning using minimal or zero manual mask annotation [2406.05776].

## 4. Downstream Applications and Extensions

The modular design of Grounded-SAM enables rapid adaptation for a variety of open-world vision tasks:

- **Automatic Dense Annotation**: Integration with image tagging (e.g., Recognize Anything) or BLIP image captioning allows automatic extraction of either coarse tags or fine-grained noun phrases, which are then grounded and segmented, producing dense instance-level annotations without human labeling.
- **Promptable Image Editing**: By localizing target objects (e.g., “the dog’s fur”) through Grounded SAM, instance masks $m_{\mathrm{edit},T'}$ are provided to inpainting models such as Stable Diffusion, supporting precise, text-driven image manipulation.
- **Prompt-based 3D Human Analysis**: Cropping the localized detected region and passing it to models like OSX enables instance-specific, promptable full-body 3D mesh reconstruction and motion capture.

A summary of core application modules is given below:

| Application Area              | Upstream Model           | Downstream Utility                             |
|-------------------------------|-------------------------|-----------------------------------------------|
| Dense annotation              | BLIP, Recognize Anything| Rapid mask generation per user/entity         |
| Image editing                 | Stable Diffusion        | Precise inpainting guided by grounded mask    |
| 3D human motion analysis      | OSX                     | Prompt-to-mesh for user-referred individuals  |

## 5. Benchmarking, Ablation Studies, and Performance Insights

Comprehensive benchmarking on open-vocabulary segmentation and camouflaged object detection supports several key conclusions:

- **Open-vocabulary segmentation (SGinW):**
  - Grounded-SAM with Grounding DINO-base+SAM-Huge: $48.7\,\mathrm{mAP}$ zero-shot, outperforming prior unified approaches.
  - Upgrading to HQ-SAM: $49.6\,\mathrm{mAP}$.
  - Prompting SAM with DINO-derived boxes yields masks of significantly higher quality for open-set phrases than alternative SAM prompts (points, scribbles).

- **Frugal Self-supervised COD (CPD1K, HitNet):**
  - Baseline: Zero-shot $F^w_\beta=0.564$; full supervised ($k=600$): $F^w_\beta=0.828$.
  - GSAM zero-shot: $F^w_\beta=0.722$; fine-tuned on 30 GSAM pseudo-labels: $F^w_\beta=0.748$.
  - GSAM pseudo-labels cleanly bridge the label-efficiency gap with 30-shot supervised approaches [2406.05776].

- **Module ablations reveal**:
  - Using SAM-Large in place of SAM-Huge degrades mAP by 2.7.
  - Removing the text encoder from Grounding DINO reduces mAP by >5.
  - Using suboptimal SAM prompts causes up to $10\,\mathrm{mAP}$ loss, establishing box-prompting as critical for segmentation precision [2401.14159].

A notable limitation: direct GSAM pseudo-labels hallucinate foregrounds on empty images (FPR $=0.68$), but frugal downstream finetuning corrects the background prior (FPR $=0.02$ after HitNet adaptation).

## 6. Limitations, Systematic Weaknesses, and Comparative Analysis

Although Grounded-SAM exhibits compelling zero-shot and frugal learning performance, it manifests structural and practical limitations:

- **Over-segmentation and Hard Label-Mask Assignment**: As a prompt-driven pipeline, Grounded-SAM binds each DINO proposal box (and its corresponding segmentation mask) to a single class or phrase, and cannot refine instances when boundary coherence is poor or when multiple fragments are produced.
- **No End-to-End Optimization**: Model assembly uses completely frozen weights, limiting adaptation to new domains or granularity beyond what DINO/SAM were pre-trained for.
- **Computational Cost**: Each mask requires a full forward pass through SAM’s decoder; use of dense or coarse prompts can induce significant latency.

Compared to advanced mask-injection approaches (e.g., SAM-MI [2511.20027]), Grounded-SAM remains susceptible to SAM’s over-segmentation and rigid class-assignment weaknesses. SAM-MI introduces sparse point prompting (reducing decoder calls by 96%), shallow mask aggregation, and decoupled mask injection, achieving a $16.7\%$ relative mIoU improvement and $1.6\times$ speedup over Grounded-SAM in open-vocabulary benchmarks. This suggests that “mask injection” paradigms address critical structural weaknesses inherent to the prompt-driven box+sandbox model assembly strategy [2511.20027].

## 7. Research Impact and Prospective Directions

Grounded-SAM establishes a foundational paradigm for open-world vision by demonstrating that modular assembly of strong pretrained detectors and segmentation models enables broad, zero-shot text-to-mask capability with robust performance on real-world and open-vocabulary challenges [2401.14159][2406.05776]. Its success in frugal self-supervised learning and annotation automation has spurred follow-up studies focusing on:

- **Efficient Prompting and Aggregation**: Mask aggregation (e.g., to correct over-segmentation) and sparse adaptive prompting (e.g., TSPP in SAM-MI) are active areas for reducing computational load and improving mask quality.
- **Soft Mask-Label Integration**: Moving beyond 1:1 mask-label assignment via decoupled mask fusion or injection into segmentation heads.
- **Domain Specialization**: While generalist, the architecture’s performance on highly non-canonical images (e.g., camouflaged, highly occluded, or compositionally complex objects) is contingent on effective pseudo-label filtering and postprocessing.

A plausible implication is that future open-world segmentation systems will increasingly favor tightly integrated, end-to-end trainable modules with mask-injection mechanisms, rather than modular prompt-driven pipelines, particularly as annotation and computational efficiency become central constraints [2511.20027].

Source: https://www.emergentmind.com/topics/grounded-sam