Segment Anything Model (SAM)

Updated 12 May 2026

Segment Anything Model (SAM) is a promptable image segmentation model that segments arbitrary objects using diverse prompts and demonstrates zero-shot generalization.
SAM leverages a modular architecture—with an image encoder, prompt encoder, and mask decoder—trained on the vast SA-1B dataset to ensure broad domain applicability.
Its extensions, including SAM2 with multi-scale integration and video support, drive rapid advances in autonomous driving, medical imaging, remote sensing, and more.

The Segment Anything Model (SAM) is a promptable foundation model for image segmentation, introduced by Meta AI Research in 2023. It is designed to segment arbitrary objects in images using prompts such as points, bounding boxes, or masks, exhibiting impressive zero-shot generalization to new data distributions and tasks. SAM was trained on the largest segmentation dataset to date—SA-1B, comprising over 1.1 billion masks on 11 million images—dramatically broadening its domain applicability. SAM and its second-generation version, SAM2, have catalyzed the vision foundation model movement, spurring rapid research, specialized adaptation, and deployment in heterogeneous domains such as autonomous driving, medical imaging, remote sensing, and more.

1. Core Architecture: Promptable Segmentation Foundation Model

SAM’s architecture consists of three modular components: an image encoder, a prompt encoder, and a mask decoder. The image encoder is a Vision Transformer (ViT) pretrained with masked autoencoding, producing a dense feature map for high-resolution inputs. The prompt encoder processes user-provided prompts—points, bounding boxes, free-form masks, and, in later versions, text—using learned positional encodings and lightweight MLPs or CLIP-based text embeddings. The mask decoder is a compact Transformer-based module that fuses the image and prompt embeddings, outputs multiple candidate masks, and estimates their Intersection over Union (IoU) confidence.

Mathematically, the core inference step can be written as:

$\hat{m} = \sigma\left(w^\top E_{\text{up}}(:, i, j)\right)$

where $E_{\text{up}}$ is the upsampled feature embedding, $w$ is the dynamic mask classifier for the output token, and $\sigma$ is the sigmoid activation. For each prompted segmentation, SAM predicts three candidate masks accompanied by their predicted IoUs ("ambiguity-aware" output) (Kirillov et al., 2023, Zhang et al., 2023).

The architecture of SAM2 introduces multi-scale feature integration in the encoder, dynamic prompt weighting, and boundary-aware loss objectives, improving boundary sharpness and contextual granularity, while extending prompt support to text for semantic-driven segmentation (Zhang et al., 2023, Geetha et al., 2024).

2. Training Regimen and Large-Scale Data Engine

SAM's generalization is underpinned by its pretraining on SA-1B using a three-stage annotation engine: initial manual masking, semi-automatic curation augmented by SAM predictions, and fully automatic large-scale mask generation. The loss function combines focal loss, dice loss, and mask IoU regression:

$L_\text{mask} = L_\text{focal}(m, \hat{m}) + \lambda L_\text{dice}(m, \hat{m}), \quad L_\text{IoU} = (\hat{\text{IoU}} - \text{IoU}(m, \hat{m}))^2$

with total loss $L = L_\text{mask} + L_\text{IoU}$ and hyperparameters such as $\lambda = 1/20$ (Kirillov et al., 2023).

Image inputs are standardized (e.g., cropping, normalization, resizing to 1024×1024), and the prompting protocol is simulated interactively, varying prompt types (points, boxes) and their spatial diversity throughout pretraining (Kirillov et al., 2023, Zhang et al., 2023).

In SAM2, video adaptation is facilitated by the memory encoder and attention over both past and (for offline modes) future frames, supporting real-time or near-real-time segmentation and annotation at video scale (Geetha et al., 2024).

3. Prompt Engineering and Zero-Shot Transfer

SAM's promptability is central to its universality. Supported prompt modalities are summarized as follows:

Prompt Type	Internal Representation	Supported In
Point	Sin-cos positional + FG/BG token	SAM, SAM2
Bounding Box	Two corner tokens + type embedding	SAM, SAM2
Dense Mask	Downsampled mask → conv proj	SAM, SAM2
Text	CLIP text embedding (256d)	SAM2

Prompt combinations (e.g., point+box) provide synergistic gains, particularly under domain shift or image perturbations (Wang et al., 2023). Empirically, box prompts outperform points by 0.03–0.10 Dice (e.g., brain tumor segmentation (Zhang et al., 2023)), and prompt design is critical for maximizing zero-shot transfer across challenging domains—including medical imaging, remote sensing, agricultural phenotyping, and eye feature analysis (Zhang et al., 2023, Osco et al., 2023, Williams et al., 2023, Maquiling et al., 2023).

Zero-shot protocols involve applying SAM off-the-shelf to new datasets using an appropriate prompt strategy and evaluating spatial overlap (IoU, Dice) and boundary metrics (Hausdorff, ASSD) without fine-tuning any model weights.

4. Quantitative Performance, Robustness, and Foundational Properties

SAM frequently approaches or matches state-of-the-art (SOTA) supervised segmentation models in out-of-distribution settings, especially under simple box or combined prompts. Example findings:

Brain Tumor Segmentation (BraTS2019):
- Box-prompted SAM Dice: WT 0.70, TC 0.78, ET 0.67 vs. SOTA mmformer: WT ≈0.89, TC ≈0.85, ET ≈0.80. Box prompts outperform points by up to 0.25 Dice (WT) (Zhang et al., 2023).
Radiation Oncology (CT OARs):
- Box-prompts boost Dice by 0.1–0.5. Large, high-contrast organs achieve Dice >0.85 without prompts; small or low-contrast organs remain challenging without domain adaptation (Zhang et al., 2023).
Autonomous Driving:
- Cityscapes val, zero-shot: SAM+OneFormer mIoU ≈80.0% (clean), ≈75.5% (Gaussian blur, moderate corruption); retains 50–70 mIoU under severe corruptions, exceeding CNN baselines by ∼20–40 points (Yan et al., 2024).
Remote Sensing:
- UAV trees, Dice ≈0.922–0.950 (text or box/prompted, with or without automated one-shot fine-tuning) (Osco et al., 2023).
Plant Phenotyping:
- Potato leaf, AP_75 ≈60% for postprocessed “Leaf Only SAM” pipeline, compared to ≈75% for fine-tuned Mask R-CNN (Williams et al., 2023).
Eye Feature Segmentation:
- Pupil IoU up to ≈0.93 with box plus point prompts, matching fully supervised models for high-contrast features (Maquiling et al., 2023).

Robustness evaluations reveal that SAM exhibits marked resilience to moderate noise, digital corruptions, and even adversarial examples, attributed to model scale, diverse pretraining, and prompt/multimodal conditioning. For example, under white-box PGD-10 attacks ( $\epsilon=8/255$ ), SAM+OneFormer only dropped from 80% to 53% mIoU, outperforming conventional segmentation networks (Yan et al., 2024, Wang et al., 2023).

However, the model is notably susceptible to motion blur, chromatic aberration, and heavy noise (IoU drops of 14–17%), while maintaining relative immunity to mild brightness/saturation shifts (Wang et al., 2023).

5. Model Biases, Limitations, and Domain Transfer

SAM displays a strong empirical bias toward local texture cues rather than global shape, as demonstrated by synthetic experiments where shape–texture cues are pitted against each other. In conditions with pure silhouette (shape-only) or conflicting internal patterning, SAM's predicted mask aligns much more with texture, diverging from human contour-adherence in segmentation (Zhang et al., 2023). This foundational bias partly explains performance drops in low-contrast, boundary-ambiguous, or highly non-Euclidean domains (e.g., cloudy, SAR, or complex medical imagery).

Model limitations and recurring failure cases include:

Poor segmentation of ambiguous, low-contrast, or small/thin structures.
Loss of 3D or multi-frame context (SAM1 is 2D only; SAM2 partially addresses video context (Geetha et al., 2024)).
Instance-centric objectness, causing failure in class-level or amorphous objects (e.g., roads, clouds) (Ren et al., 2023).
No built-in semantic label prediction; requiring downstream fusion with vision–LLMs for full panoptic tasks (Han et al., 2023).
High computational cost for ViT-H backbone variants; slow runtimes restrict edge deployment (Sun et al., 2024).

Domain shift (e.g., from natural images to medical MRI, PolSAR, multispectral aerial) causes accuracy gaps, correctable to some extent via lightweight adapters, prompt redesign, or token-level fine-tuning (Wang et al., 2024, Zhang et al., 2023).

6. Extensions: SAM2, Open-Vocabulary, Captioning, and Efficient Variants

Recent research has rapidly extended SAM and its paradigm:

SAM2 incorporates multi-scale feature hierarchies, temporally-aware memory encoding, and expanded text-prompt conditioning, achieving real-time video segmentation and substantially faster annotation workflows (Geetha et al., 2024, Zhang et al., 2023).
Open-Vocabulary Segmentation integrates CLIP-derived text conditioning, SideFormer adapters, and open-set region proposal networks, enabling zero-shot detection and segmentation of arbitrary textual categories (Han et al., 2023).
Regional Captioning: Lightweight feature mixer modules are used to align SAM’s sparse region tokens with LLMs, yielding state-of-the-art dense region captions on Visual Genome. Weak supervision using category labels and parameter-efficient learning enables rapid scaling (Huang et al., 2023).
Efficient Variants: Distillation, pruning, quantization, and knowledge transfer from mainline SAM to CNN, tiny ViT, or lightweight transformer backbones (e.g., MobileSAM, EdgeSAM, EfficientViT-SAM, NanoSAM, FastSAM) have realized >10–30x speedups with <2% mean IoU degradation in standard segmentation tasks, facilitating deployment on edge hardware (Sun et al., 2024).

Model/Variant	Params (M)	Inference FPS	mIoU (COCO val, box)	Notable Attribute
SAM-H	641	2.0	77.4	Baseline, high accuracy
MobileSAM/EdgeSAM	10/9.6	~17	73.9/76.2	TinyViT/RepViT distilled
EfficientViT-SAM-L0	34.8	20.9	78.5	Linear attention
NanoSAM	0.94	27.9	70.2	CNN backbone, Jetson deployment
FastSAM	–	>10	–	YOLO instance segmentation pipeline

7. Prospects and Future Directions

Research identifies multiple high-potential directions for the Segment Anything family:

Unified joint mask+label models combining segmentation and recognition.
Continual, domain-adaptive prompt encoders and mask decoders.
Robustness enhancements through adversarial and corruption-augmented pretraining.
Cross-modal (multi-sensor) and multi-task foundation vision models, e.g., incorporating audio, depth, language, or tactile prompts (Zhang et al., 2023).
3D and volumetric segmentation support, especially for biomedical and remote sensing domains (Wang et al., 2024, Zhang et al., 2023).
Real-time and edge deployment via efficient backbones, smart prompt selection, and structured sparsity (Sun et al., 2024, Zhang et al., 2023).
Deeper exploration of task-specific adapters, LoRA modules, and parameter-efficient fine-tuning to bridge domain gaps.

Ongoing work continues to address model biases (e.g., texture over shape), instance-class segmentation limits, and efficient integration with downstream fusion models for full panoptic and open-world understanding.