---
title: 'SA-Co Benchmark: Open-Vocabulary Concept Segmentation'
url: https://www.emergentmind.com/topics/segment-anything-with-concepts-sa-co-benchmark
type: topic
---

# SA-Co Benchmark: Open-Vocabulary Concept Segmentation

Segment Anything with Concepts (SA-Co) is a large-scale, open-vocabulary benchmark for Promptable Concept Segmentation (PCS) in images and videos, introduced in conjunction with SAM 3 ("Segment Anything Model 3"). PCS tasks require models to detect, segment, and track all instances of user-specified concepts, defined via short noun phrases, image exemplars, or their combination, assigning consistent identities to these instances across frames. SA-Co enables rigorous evaluation of open-vocabulary recognition, instance segmentation quality, and tracking accuracy, supporting noun-phrase, exemplar, and mixed prompt formats.

## 1. Benchmark Definition and Scope

SA-Co is designed to quantify model performance on PCS tasks, where the input prompt can be a noun phrase (e.g., "striped cat"), an image exemplar (positive/negative crop), or both. The benchmark mandates segmentation of every matching instance in the provided media and unique ID assignment for temporal consistency in video. Supported objectives comprise:

- Open-vocabulary recognition (presence detection)
- Instance segmentation boundary quality
- Instance tracking accuracy in video settings

SA-Co supports prompt types: (1) noun phrase, (2) exemplar, (3) mixed.

## 2. Dataset Construction and Statistics

The SA-Co dataset is produced via a multi-phase, scalable data engine pipeline:

- **Phase 1:** Mining image-caption-derived noun phrases → mask proposals (SAM 2 + OWLv2) → human Mask Verification (MV) and Exhaustivity Verification (EV) → manual corrections, yielding the HQ image set (4.3M image-NPs).
- **Phase 2:** Replacement of human MV/EV by Llama-based AI verifiers; introduction of hard negative generator (ontology + MLLM sourcing) filtered by spurious mask triggers, adding 122M image-NPs.
- **Phase 3:** Expansion to 15 visual domains, adoption of a fully AI synthetic verification mode ("SYN"), contributing 19.5M image-NPs.
- **Phase 4 (video):** Scene/motion-based mining, SAM 3 pseudo-masks, and manual annotation to form 52.5K videos, 134K video-NP pairs, and 467K masklets.

Final splits are:

| Split   | Images/Videos | Concept Labels | Masks      |
|---------|--------------|---------------|------------|
| HQ      | 5.2M images  | 4M unique NPs | 52M        |
| SYN     | 39M images   | 1.7B image-NPs| 1.4B       |
| EXT     | 9.3M images  | 15 datasets   | 70M        |
| VID     | 52.5K videos | 24.8K NPs     | 467K       |

Evaluation subsets encompass ~207K unique NPs, 121K media, and 3M media-NPs, with specific splits for images (Gold, Silver, Bronze, Bio) and videos (SA-V, YT-Temporal-1B, SmartGlasses). Ontologically, 22.4M Wikidata nodes are mapped to 17 top-level categories (e.g., animals, human, tools), with split distributions documented in SAM 3.

## 3. Annotation Procedures and Evaluation Protocols

### Image Annotation

- **Mask Verification (MV):** Accept/reject candidate triplet (image, NP, mask).
- **Exhaustivity Verification (EV):** Verify that accepted masks exhaustively cover all instances.
- **Correction:** Manual refinement for exhaustive mask annotation.
- Gold splits are labeled by three independent experts to estimate human upper-bound accuracy.

### Video Annotation

- Detection per frame, propagation through SAM 3 tracker, and matching of new detections.
- Masklets unconfirmed for a delay (T=15 frames) are discarded based on Masklet Detection Score (MDS).
- Masklet suppression is performed if MDS falls below zero over the lifespan.
- Tracker re-anchoring is enforced at regular intervals (N=16 frames).

### Metrics

- **Localization:** Micro-F1 across IoU thresholds $\tau = 0.5, ..., 0.95$
  $$ \text{pmF}_1 = \frac{1}{10} \sum_{\tau=0.5:0.05:0.95} \frac{2\ TP^\tau}{2\ TP^\tau + FP^\tau + FN^\tau} $$
  $$ \text{IoU}(m, \hat{g}) = \frac{|m \wedge \hat{g}|}{|m \vee \hat{g}|} $$
- **Image-level Classification:** Matthews Corr. Coeff. (IL_MCC)
  $$ \text{IL\_MCC} = \frac{TP \times TN - FP \times FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}} $$
- **Combined:** $cgF_1 = 100 \times pmF_1 \times IL\_MCC$
- **Video:** cgF₁, volume-IoU, plus pHOTA (HOTA on video-NP pairs), TETA
- **Standard benchmarks:** AP for box/mask detection (COCO, LVIS), MAE/Accuracy for counting, J/G for VOS.

### Split Protocols

- Image QA splits are zero-shot (no training overlap).
- Few-shot adaptation possible (1–10 examples) for select datasets.
- Video evaluation is performed without fine-tuning on evaluation sets.

## 4. Comparative Baseline Performance and Ablation Studies

### Image-Level

- On SA-Co Gold: Best prior baseline achieves $1 \approx 24.6$, SAM 3 achieves $1 = 54.1$ (+2× increase, $\sim$74% of human).
- Similar performance gains on Silver/Bronze/Bio splits.
- Exemplar prompt on COCO/LVIS/ODinW13: T-Rex2 best prior $1^+ = 58.5$; SAM 3+exemplar $1^+ = 76.8$ (+18.3).

### Video-Level

- Text-prompted tracking (SA-V, YT-Temporal, SmartGlasses): SAM 3 $cgF_1 = 30-50\%$ (>$3\times$ baseline).
- On LVVIS, BURST, YTVIS, OVIS: SAM 3 $36.3$ mAP vs. $20.8$ for baselines (+75% rel).

### Other Visual Segmentation Tasks

- VOS (MOSEv2): SAM 3 $J=83.5$ vs. prior best $J=75.3$ ($+6.5$ improvement).
- Interactive image segmentation: 5-click $mIoU = 85.1$ (SAM 3) vs. $84.3$ (SAM 2.1).

### Counting Tasks

- CountBench/PixMo-Count: SAM 3 with IoM-NMS, MAE $=0.12$, accuracy $=93.8\%$; prior best accuracy $=92.4\%$.

### Ablations

- Presence head increases $1$ by $+1.5$ ($IL: 0.77 \rightarrow 0.82$).
- Hard negatives unlock $+15$ (IL $0.44 \rightarrow 0.68$).
- Data subset analysis: EXT+SYN+HQ $1=47.4$; without HQ $1=23.7$; with SYN only $1=32.8$.
- AI verifier improvements: EV AI $+7.2$, MV AI $+1.1$, closing half gap to human annotation.

### Domain Adaptation

Holding out "Food&Drink" domain and fine-tuning on synthetic only (SYN-Food) matches HQ-Food performance at large scale without human annotation.

## 5. Data Access and Benchmark Usage

Annotations and evaluation code are publicly available at https://github.com/facebookresearch/sam3. Annotation format is federated JSON containing media_id, phrase, masks, and hard_negatives.

**Benchmarking Process:**
1. Run model on each (media, phrase) pair, outputting mask polygons and confidence scores.
2. Pack results into SAM 3 JSON format.
3. Evaluate results using provided scripts:
   ```bash
   python eval_image.py --pred sampreds.json --gt saco_gold.json --metric cgF1 --threshold 0.5
   python eval_video.py --pred vidpreds.json --gt sacovideo.json --metrics cgF1 pHOTA TETA --iou_thresh 0.5
   ```

**Video tracking pseudocode:**
```python
prev_masklets = []
for t, frame in enumerate(video):
   dets = Detector(frame, prompt)
   props = Tracker.propagate(prev_masklets)
   matched, unmatched = match(props, dets, iou_thresh=0.3)
   masklets = suppress_unconfirmed(matched) + new_masklets(unmatched)
   if t % 16==0: masklets = re_prompt(masklets, dets)
   output_after_delay(masklets, delay=15)
   prev_masklets = masklets
```

## 6. Category Coverage and Ontological Structure

SA-Co employs an ontology comprising 22.4M Wikidata nodes, collapsed to 17 top-level categories, ensuring broad concept coverage. The training distribution spans all domains, including fine-grained and long-tail NPs, and evaluation subsets span external domains and medical data.

A plausible implication is that categorical breadth and fine-grained prompt coverage synergistically support robust open-vocabulary evaluation and realistic assessment of semantic generalization across domains.

## 7. Significance and Prospects

SA-Co establishes a standardized, rigorous, and scalable framework for evaluating promptable concept segmentation in both images and videos, with extensive coverage of object categories and ontological granularity. By supporting zero-shot, few-shot, and synthetic domain adaptation evaluation, the benchmark enables assessment of PCS model generality and performance ceiling relative to human-level segmentation and recognition. The methodological advances in dataset generation, annotation protocols, metric design, and open-source tooling collectively position SA-Co as a central resource for advancing open-vocabulary segmentation research in computer vision [2511.16719].

Source: https://www.emergentmind.com/topics/segment-anything-with-concepts-sa-co-benchmark