---
title: Segment Anything Model (SAM)
url: https://www.emergentmind.com/topics/segment-anything-model-sam-a3ef8598-8bcc-4668-bf0b-1f7e8540cdbd
type: topic
---

# Segment Anything Model (SAM)

The Segment Anything Model (SAM) is a promptable foundation model for image segmentation, introduced by Meta AI Research in 2023. It is designed to segment arbitrary objects in images using prompts such as points, bounding boxes, or masks, exhibiting impressive zero-shot generalization to new data distributions and tasks. SAM was trained on the largest segmentation dataset to date—SA-1B, comprising over 1.1 billion masks on 11 million images—dramatically broadening its domain applicability. SAM and its second-generation version, SAM2, have catalyzed the vision foundation model movement, spurring rapid research, specialized adaptation, and deployment in heterogeneous domains such as autonomous driving, medical imaging, remote sensing, and more.

## 1. Core Architecture: Promptable Segmentation Foundation Model

SAM’s architecture consists of three modular components: an image encoder, a prompt encoder, and a mask decoder. The image encoder is a Vision Transformer (ViT) pretrained with masked autoencoding, producing a dense feature map for high-resolution inputs. The prompt encoder processes user-provided prompts—points, bounding boxes, free-form masks, and, in later versions, text—using learned positional encodings and lightweight MLPs or CLIP-based text embeddings. The mask decoder is a compact Transformer-based module that fuses the image and prompt embeddings, outputs multiple candidate masks, and estimates their Intersection over Union (IoU) confidence.

Mathematically, the core inference step can be written as:

\[
\hat{m} = \sigma\left(w^\top E_{\text{up}}(:, i, j)\right)
\]

where \(E_{\text{up}}\) is the upsampled feature embedding, \(w\) is the dynamic mask classifier for the output token, and \(\sigma\) is the sigmoid activation. For each prompted segmentation, SAM predicts three candidate masks accompanied by their predicted IoUs ("ambiguity-aware" output) [2304.02643, 2306.06211].

The architecture of SAM2 introduces multi-scale feature integration in the encoder, dynamic prompt weighting, and boundary-aware loss objectives, improving boundary sharpness and contextual granularity, while extending prompt support to text for semantic-driven segmentation [2306.06211, 2408.06305].

## 2. Training Regimen and Large-Scale Data Engine

SAM's generalization is underpinned by its pretraining on SA-1B using a three-stage annotation engine: initial manual masking, semi-automatic curation augmented by SAM predictions, and fully automatic large-scale mask generation. The loss function combines focal loss, dice loss, and mask IoU regression:

\[
L_\text{mask} = L_\text{focal}(m, \hat{m}) + \lambda L_\text{dice}(m, \hat{m}), \quad
L_\text{IoU} = (\hat{\text{IoU}} - \text{IoU}(m, \hat{m}))^2
\]

with total loss \(L = L_\text{mask} + L_\text{IoU}\) and hyperparameters such as \(\lambda = 1/20\) [2304.02643].

Image inputs are standardized (e.g., cropping, normalization, resizing to 1024×1024), and the prompting protocol is simulated interactively, varying prompt types (points, boxes) and their spatial diversity throughout pretraining [2304.02643, 2306.06211].

In SAM2, video adaptation is facilitated by the memory encoder and attention over both past and (for offline modes) future frames, supporting real-time or near-real-time segmentation and annotation at video scale [2408.06305].

## 3. Prompt Engineering and Zero-Shot Transfer

SAM's promptability is central to its universality. Supported prompt modalities are summarized as follows:

| Prompt Type  | Internal Representation              | Supported In      |
|--------------|-------------------------------------|-------------------|
| Point        | Sin-cos positional + FG/BG token    | SAM, SAM2         |
| Bounding Box | Two corner tokens + type embedding  | SAM, SAM2         |
| Dense Mask   | Downsampled mask → conv proj        | SAM, SAM2         |
| Text         | CLIP text embedding (256d)          | SAM2              |

Prompt combinations (e.g., point+box) provide synergistic gains, particularly under domain shift or image perturbations [2305.06422]. Empirically, box prompts outperform points by 0.03–0.10 Dice (e.g., brain tumor segmentation [2309.08434]), and prompt design is critical for maximizing zero-shot transfer across challenging domains—including medical imaging, remote sensing, agricultural phenotyping, and eye feature analysis [2306.11730, 2306.16623, 2305.09418, 2311.08077].

Zero-shot protocols involve applying SAM off-the-shelf to new datasets using an appropriate prompt strategy and evaluating spatial overlap (IoU, Dice) and boundary metrics (Hausdorff, ASSD) without fine-tuning any model weights.

## 4. Quantitative Performance, Robustness, and Foundational Properties

SAM frequently approaches or matches state-of-the-art (SOTA) supervised segmentation models in out-of-distribution settings, especially under simple box or combined prompts. Example findings:

- **Brain Tumor Segmentation (BraTS2019):**
  - Box-prompted SAM Dice: WT 0.70, TC 0.78, ET 0.67 vs. SOTA mmformer: WT ≈0.89, TC ≈0.85, ET ≈0.80. Box prompts outperform points by up to 0.25 Dice (WT) [2309.08434].
- **Radiation Oncology (CT OARs):**
  - Box-prompts boost Dice by 0.1–0.5. Large, high-contrast organs achieve Dice >0.85 without prompts; small or low-contrast organs remain challenging without domain adaptation [2306.11730].
- **Autonomous Driving:**
  - Cityscapes val, zero-shot: SAM+OneFormer mIoU ≈80.0% (clean), ≈75.5% (Gaussian blur, moderate corruption); retains 50–70 mIoU under severe corruptions, exceeding CNN baselines by ∼20–40 points [2408.09839].
- **Remote Sensing:**
  - UAV trees, Dice ≈0.922–0.950 (text or box/prompted, with or without automated one-shot fine-tuning) [2306.16623].
- **Plant Phenotyping:**
  - Potato leaf, AP_75 ≈60% for postprocessed “Leaf Only SAM” pipeline, compared to ≈75% for fine-tuned Mask R-CNN [2305.09418].
- **Eye Feature Segmentation:**
  - Pupil IoU up to ≈0.93 with box plus point prompts, matching fully supervised models for high-contrast features [2311.08077].

Robustness evaluations reveal that SAM exhibits marked resilience to moderate noise, digital corruptions, and even adversarial examples, attributed to model scale, diverse pretraining, and prompt/multimodal conditioning. For example, under white-box PGD-10 attacks (\(\epsilon=8/255\)), SAM+OneFormer only dropped from 80% to 53% mIoU, outperforming conventional segmentation networks [2408.09839, 2305.06422].

However, the model is notably susceptible to motion blur, chromatic aberration, and heavy noise (IoU drops of 14–17%), while maintaining relative immunity to mild brightness/saturation shifts [2305.06422].

## 5. Model Biases, Limitations, and Domain Transfer

SAM displays a strong empirical bias toward local texture cues rather than global shape, as demonstrated by synthetic experiments where shape–texture cues are pitted against each other. In conditions with pure silhouette (shape-only) or conflicting internal patterning, SAM's predicted mask aligns much more with texture, diverging from human contour-adherence in segmentation [2311.11465]. This foundational bias partly explains performance drops in low-contrast, boundary-ambiguous, or highly non-Euclidean domains (e.g., cloudy, SAR, or complex medical imagery).

Model limitations and recurring failure cases include:

- Poor segmentation of ambiguous, low-contrast, or small/thin structures.
- Loss of 3D or multi-frame context (SAM1 is 2D only; SAM2 partially addresses video context [2408.06305]).
- Instance-centric objectness, causing failure in class-level or amorphous objects (e.g., roads, clouds) [2304.13000].
- No built-in semantic label prediction; requiring downstream fusion with vision–language models for full panoptic tasks [2312.03628].
- High computational cost for ViT-H backbone variants; slow runtimes restrict edge deployment [2410.04960].

Domain shift (e.g., from natural images to medical MRI, PolSAR, multispectral aerial) causes accuracy gaps, correctable to some extent via lightweight adapters, prompt redesign, or token-level fine-tuning [2412.12737, 2306.11730].

## 6. Extensions: SAM2, Open-Vocabulary, Captioning, and Efficient Variants

Recent research has rapidly extended SAM and its paradigm:

- **SAM2** incorporates multi-scale feature hierarchies, temporally-aware memory encoding, and expanded text-prompt conditioning, achieving real-time video segmentation and substantially faster annotation workflows [2408.06305, 2306.06211].
- **Open-Vocabulary Segmentation** integrates CLIP-derived text conditioning, SideFormer adapters, and open-set region proposal networks, enabling zero-shot detection and segmentation of arbitrary textual categories [2312.03628].
- **Regional Captioning**: Lightweight feature mixer modules are used to align SAM’s sparse region tokens with language models, yielding state-of-the-art dense region captions on Visual Genome. Weak supervision using category labels and parameter-efficient learning enables rapid scaling [2312.00869].
- **Efficient Variants**: Distillation, pruning, quantization, and knowledge transfer from mainline SAM to CNN, tiny ViT, or lightweight transformer backbones (e.g., MobileSAM, EdgeSAM, EfficientViT-SAM, NanoSAM, FastSAM) have realized >10–30x speedups with <2% mean IoU degradation in standard segmentation tasks, facilitating deployment on edge hardware [2410.04960].

| Model/Variant              | Params (M) | Inference FPS | mIoU (COCO val, box) | Notable Attribute                      |
|---------------------------|------------|---------------|----------------------|----------------------------------------|
| SAM-H                     | 641        | 2.0           | 77.4                 | Baseline, high accuracy                |
| MobileSAM/EdgeSAM         | 10/9.6     | ~17           | 73.9/76.2            | TinyViT/RepViT distilled               |
| EfficientViT-SAM-L0       | 34.8       | 20.9          | 78.5                 | Linear attention                       |
| NanoSAM                   | 0.94       | 27.9          | 70.2                 | CNN backbone, Jetson deployment        |
| FastSAM                   | –          | >10           | –                    | YOLO instance segmentation pipeline    |


## 7. Prospects and Future Directions

Research identifies multiple high-potential directions for the Segment Anything family:

- Unified joint mask+label models combining segmentation and recognition.
- Continual, domain-adaptive prompt encoders and mask decoders.
- Robustness enhancements through adversarial and corruption-augmented pretraining.
- Cross-modal (multi-sensor) and multi-task foundation vision models, e.g., incorporating audio, depth, language, or tactile prompts [2306.06211].
- 3D and volumetric segmentation support, especially for biomedical and remote sensing domains [2412.12737, 2309.08434].
- Real-time and edge deployment via efficient backbones, smart prompt selection, and structured sparsity [2410.04960, 2306.06211].
- Deeper exploration of task-specific adapters, LoRA modules, and parameter-efficient fine-tuning to bridge domain gaps.

Ongoing work continues to address model biases (e.g., texture over shape), instance-class segmentation limits, and efficient integration with downstream fusion models for full panoptic and open-world understanding.

## References

- Original Segment Anything Model (SAM): [2304.02643]
- Unified survey and SAM2: [2306.06211], [2408.06305]
- Robustness (adversarial, corruption): [2408.09839], [2305.06422]
- Medical imaging and domain transfer: [2309.08434], [2306.11730], [2311.08077]
- Remote sensing and overhead imagery: [2306.16623], [2304.13000]
- Texture vs. shape bias: [2311.11465]
- Plant phenotyping: [2305.09418]
- Efficient model variants: [2410.04960]
- Captioning and open-vocabulary extensions: [2312.03628], [2312.00869]
- SAR and cross-domain fusion: [2412.12737]

Source: https://www.emergentmind.com/topics/segment-anything-model-sam-a3ef8598-8bcc-4668-bf0b-1f7e8540cdbd