---
title: 'Grounded SAM: Open-Vocab Vision Pipeline'
url: https://www.emergentmind.com/topics/groundedsam
type: topic
---

# Grounded SAM: Open-Vocab Vision Pipeline

Grounded SAM is an integrated vision pipeline that enables open-vocabulary detection and segmentation of image regions conditioned on arbitrary text inputs. By assembling Grounding DINO—a text-driven, open-set object detector—and the Segment Anything Model (SAM)—a promptable segmentation backbone—Grounded SAM achieves diverse visual tasks without monolithic retraining, providing modularity and extensibility for applications such as automated annotation, controllable image editing, and promptable 3D human motion analysis. This architecture shows state-of-the-art zero-shot performance on segmentation benchmarks, highlighting its utility as a plug-and-play assembly for open-world visual modeling [2401.14159].

## 1. Component Architecture and Workflow

Grounded SAM operates as a composition rather than a single end-to-end network. The main components are:

- **Grounding DINO** [2303.05499]: An open-set object detector that accepts an image $I\in\mathbb{R}^{H\times W \times 3}$ and a text prompt $T$ and outputs $N$ bounding boxes $\mathcal{B} = \{(b_i, s_i)\}_{i=1}^N$ where $b_i$ describes spatial coordinates and $s_i$ is a grounding score.
- **Segment Anything Model (SAM)** [2304.02643]: For each detected box $b_i$, SAM predicts a binary mask $M_i \in \{0,1\}^{H \times W}$, with the box serving as the prompt.

The process is illustrated as follows:

```
Input: (I, T)
    ├─→ [Grounding DINO] ──→ Boxes {(b_i, s_i)}
    │
    └─→ For each b_i:
         └─→ [SAM mask-predictor ◂ box b_i] ──→ Mask M_i
```

Pseudocode:

```python
def GroundedSAM(I, T):
    Boxes = GroundingDINO.detect(I, T)  # ℬ = {b_i, s_i}
    Masks = []
    for (b, s) in Boxes:
        M = SAM.predict_mask(I, box_prompt=b)
        Masks.append((b, s, M))
    return Masks
```

This architecture supports modular plug-ins and expert components for extended pipelines (e.g., BLIP, RAM, Stable-Diffusion, OSX).

## 2. Mathematical Formulation

The detection and segmentation stages are mathematically defined as follows:

**Open-set Detection (Grounding DINO):**
- Text embedding: $E_{\text{text}}(T) \in \mathbb{R}^d$
- Visual features: $E_{\text{img}}(I) \in \mathbb{R}^{H'\times W'\times C}$
- Learned object queries $\{q_j\}_{j=1}^{N_q}$ attend to $(E_{\text{img}}, E_{\text{text}})$.
- For each query, classification score $\hat p_{ij}$ and bounding box $\hat b_{ij}$.
- Grounding score:
  $$
  s_i = \sigma(f_{\text{cls}}(q_i, E_{\text{img}}, E_{\text{text}}))
  $$
  where $\sigma$ denotes sigmoid.
- Training loss:
  $$
  \mathcal{L}_{\text{det}} = \sum_{i=1}^{N_q}\Bigl[
      \mathcal{L}_{\text{cls}}(p_i, p_i^*)
      + \mathbb{1}_{\{p_i^*>0\}}\bigl(
          \lambda_1 \|\hat b_i - b_i^*\|_1
          + \lambda_2\,\mathrm{GIoU}(\hat b_i, b_i^*)
      \bigr)
  \Bigr]
  $$

**Promptable Segmentation (SAM):**
- Segmentation mask predicted for each box:
  $$
  \hat M = \sigma\bigl(f_{\text{mask}}(E_{\text{img}}, b)\bigr)
  $$
- Mask loss:
  $$
  \mathcal{L}_{\text{mask}} = -\frac{1}{HW}\sum_{u,v}\bigl[
      M^*_{u,v}\log\hat M_{u,v} +
      (1 - M^*_{u,v})\log(1-\hat M_{u,v})
  \bigr]
  $$

**Overall Objective:** No joint fine-tuning is performed; both models are used off-the-shelf. For end-to-end training, the objective becomes:
$$
\mathcal{L} = \underbrace{\mathcal{L}_{\text{cls}} + \mathcal{L}_{\text{bbox}}}_{\text{detection}} + \underbrace{\mathcal{L}_{\text{mask}}}_{\text{segmentation}}
$$

## 3. Task Pipelines and Representative Use Cases

Grounded SAM serves as a foundation for a variety of task-specific pipelines:

### 3.1 Automatic Dense Annotation
- **BLIP-Grounded-SAM:** BLIP captioner produces captions from $I$; each noun phrase in caption $C$ prompts GroundedSAM, yielding $b_i, M_i$ for object instance annotation.
- **RAM-Grounded-SAM:** Recognize Anything Model (RAM) generates tags $\{t_k\}$; each tag serves as a prompt for instance mask generation.

Example:

| Input              | Output                 |
|--------------------|-----------------------|
| I: cow/rainbow     | Box: (100, 30, 320, 260) Mask: cow |

### 3.2 Controllable Image Editing
User supplies prompt $T$ (e.g. “remove dog”). GroundedSAM produces relevant mask $M_i$, fed into Stable Diffusion inpainting for content manipulation.

### 3.3 Promptable 3D Human Motion Analysis
Prompt $T$ (“person in pink shirt”) yields instance mask and box, cropped to run OSX mesh recovery for obtaining parametric 3D body meshes.

## 4. Performance on Open-Vocabulary Benchmarks

On the SegInW zero-shot segmentation benchmark spanning 25 datasets, Grounded SAM demonstrates strong results:

| Method                                 | mean SGinW |
|---------------------------------------- | ---------- |
| X-Decoder-L                            | 32.2       |
| OpenSeeD-L                             | 36.7       |
| ODISE-L                                | 38.7       |
| SAN-CLIP-ViT-L                         | 41.4       |
| UNINEXT-Huge                           | 42.1       |
| Grounded-HQ-SAM (Base+DINO + HQ-SAM)   | 49.6       |
| Grounded-SAM (Base+DINO + SAM-Huge)    | 48.7       |

Grounded SAM (DINO-Base + SAM-Huge) achieves mean AP **48.7**, a significant improvement (+6 points) over the next best competitor. Integration of SAM-HQ mask backbone further raises performance to **49.6**.

## 5. Modularity, Extensibility, and Limitations

**Strengths**:
- **Open vocabulary**: Directly localizes arbitrary text queries ("Gazania linearis," "Zale Horrida") without fine-tuning.
- **Modularity**: Each step (detection, segmentation, downstream application) is independently inspectable.
- **Plug-and-play**: Easily replace or augment components (e.g., FastSAM for speed, LLM controllers for prompt routing).

**Limitations**:
- Detection bias and coverage depend on pre-trained detectors; exotic or diminutive entities may elude detection.
- Inference latency arises from cascading two large backbones; lighter alternatives (FastSAM, MobileSAM) offer remediation.

*A plausible implication is that modularity enables rapid assembly of tailored pipelines while maintaining interpretability of intermediate outputs, although limitations in coverage and computational cost persist.*

## 6. Prospective Directions

Proposed future extensions include:
- **End-to-end fine-tuning:** To improve alignment across detection and segmentation stages.
- **Vision–language large model controllers:** Automating expert selection and pipeline routing.
- **Video and 3D extension:** Incorporation of transformer-tracking (e.g., DEVA), NeRFs, or depth estimation modules for temporal and spatial modeling.

These directions suggest that the framework can be adapted for more complex modalities and enhanced automation, leveraging its inherent modular architecture.

Source: https://www.emergentmind.com/topics/groundedsam