SAM2: Advanced Promptable Segmentation

Updated 18 October 2025

SAM2 is a unified transformer-based foundation model offering prompt-driven segmentation with innovations in architecture and data handling.
It utilizes a streaming memory mechanism and skip connections to ensure temporal coherence and high-resolution detail in dynamic image and video tasks.
SAM2 achieves state-of-the-art metrics in guided segmentation while exhibiting trade-offs in automatic mode performance, highlighting its prompt sensitivity.

Segment Anything Model 2 (SAM2) is a unified transformer-based foundation model for promptable visual segmentation in both images and videos. Building upon the original Segment Anything Model (SAM), SAM2 introduces architectural, training, and data innovations that enable strong performance in prompt-driven, interactive segmentation, with particular advances in video and real-time applications. However, SAM2 exhibits fundamentally different behaviors and performance trade-offs compared to its predecessor, especially in fully automatic (prompt-free) segmentation modes.

1. Architectural Overview and Data Engine

SAM2 possesses a modular transformer architecture explicitly designed to extend segmentation capabilities from images to video. The core components include:

Image encoder: Utilizes a pre-trained, hierarchical Hiera transformer (multi-scale, high-resolution features).
Streaming memory mechanism: Video frames are processed sequentially; a FIFO-style memory bank holds feature representations and object pointers from prior frames, enabling temporal coherence and consistent segmentation.
Memory attention module: Incorporates both self-attention across current frame features and cross-attention to memory bank feature/state vectors. Temporal context is maintained not only across prior but, in select modes, also “future” frames.
Prompt encoder and mask decoder: Support standard prompt types (points, boxes, masks), with “two-way” transformers fusing prompt and image features. In the event of ambiguity (e.g., a single click matches multiple objects), multiple masks are predicted per frame.
Skip connections: High-resolution image encoder features feed directly into the mask decoder, bypassing memory attention to preserve spatial detail.

The entire model is trained on the Segment Anything Video (SA-V) dataset, an interactive, model-in-the-loop annotated corpus comprising 50.9K videos, 196 hours, and 4.2M frames with over 640K masklets. Iteratively improved versions of SAM2 were used both for annotation and to propagate/refine masklets across time, enabling 8.4× annotation speed-up compared to SAM-based per-frame annotation (Ravi et al., 1 Aug 2024).

2. Promptable vs. Automatic Segmentation Performance

SAM2 demonstrates a clear dichotomy between prompt-driven (interactive/“promptable”) and prompt-free (auto/“automatic”) segmentation performance.

Promptable settings: When segmentation is guided by explicit user inputs (coordinate clicks, bounding boxes, ground truth annotations, or coordinates generated by multimodal LLMs), SAM2 achieves substantially better results than SAM and several prior SOTA methods. This improvement is observed across domains:
- Camouflaged object detection (COD): On tasks with guided input, such as video COD with three prompt points, SAM2 achieves weighted F-measure $F_\beta^w \approx 0.691$ , structural measure $S_\alpha \approx 0.804$ , and MAE of 0.004 on MoCA-Mask, surpassing models like SLTNet and TSP-SAM (Tang et al., 31 Jul 2024).
- General visual segmentation: Across 17 video datasets and 37 image benchmarks, SAM2 is reported to require 3× fewer interactions and is 6× faster than SAM, beneficial in real-time use cases (Ravi et al., 1 Aug 2024).
- Underwater, remote sensing, and medical imaging: Similar improvements are reported when strong prompts (especially GT bounding boxes or accurate point locations) are supplied (Lian et al., 6 Aug 2024, Rafaeli et al., 13 Aug 2024, Ma et al., 6 Aug 2024).
Auto mode limitations: In contrast, SAM2's ability to discover and segment objects in the absence of prompts is significantly diminished relative to the original SAM.
- On the CAMO COD dataset in auto mode, SAM2 predicts only 4,761 masks compared to 25,472 by SAM (a reduction by 6–10×), with $S_\alpha$ falling from approximately 0.684 (SAM) to 0.444 (SAM2); similar degradations are observed in all aggregate segmentation metrics (Tang et al., 31 Jul 2024).
- In underwater instance segmentation, a dense set of point prompts produces many redundant or low-quality masks, substantially reducing AP scores compared to strong GT bounding box prompts (Lian et al., 6 Aug 2024).
- In high-resolution or class-agnostic segmentation, fine details (e.g., thin structures in DIS) are often missed, regardless of prompt strategy (Pei et al., 4 Sep 2024).

The following table summarizes these trade-offs:

Segmentation Mode	Metric Improvement (SAM2 vs. SAM)	Limitation
Promptable (guided)	SOTA, large gains in $F_\beta^w$ , $S_\alpha$ , mAP, speed	Requires accurate guides
Auto (no prompt)	Deteriorated $S_\alpha$ , fewer masks, lower AP	Missed objects, low recall

3. Experimental Evaluation and Benchmarks

SAM2’s evaluation spans multiple public datasets and task paradigms, measuring both prompt-driven and prompt-free segmentation under challenging conditions:

Datasets: CAMO, COD10K, NC4K, MoCA-Mask (video COD), UIIS, USIS10K (underwater), multiple medical imaging sets (CT, MRI, PET, ultrasound, etc.).
Performance metrics: Structure-measure ( $S_{\alpha}$ ), mean E-measure ( $E_{\phi}$ ), (weighted/maximal) $F_\beta$ -measures, mean absolute error (MAE), mean Average Precision (mAP, AP $_{50}$ , AP $_{75}$ ), Dice similarity coefficient (DSC).
Prompt sources: Randomly sampled from ground truth, generated by multimodal LLMs (e.g., Shikra, LLaVA), manual point/box annotations, and automatic object detectors (e.g., YOLOv9, YOLOv8 for Det-SAM2 (Wang et al., 28 Nov 2024)).

Notable findings include:

In promptable video COD on MoCA-Mask, SAM2 outperformed SOTA VCOD methods in every relevant metric (Tang et al., 31 Jul 2024).
Automatic mode evaluations systematically show dramatic under-segmentation and lower structural and AP metrics across all tested datasets and domains (Tang et al., 31 Jul 2024, Lian et al., 6 Aug 2024, Pei et al., 4 Sep 2024).
SAM2 is much faster than SAM for both images and video, enabling real-time applications (e.g., ~130 FPS on images for Hiera-B+ backbone) (Ravi et al., 1 Aug 2024).

4. Trade-offs, Limitations, and Technical Implications

SAM2’s design choices—particularly its streaming memory mechanism and prompt-centric training regime—prioritize high accuracy and efficiency for guided segmentation and temporal consistency in video. This yields the following trade-offs:

Prompt sensitivity: The model’s segmentation ability is tightly coupled to the presence and quality of prompts. With accurate location cues, performance exceeds prior models, but without them, recall and spatial coverage fall sharply.
Reduced object discovery: The architecture, optimized for prompt-guided segmentation and video propagation, appears to have implicit constraints (e.g., how mask proposals are generated and filtered in auto mode) that suppress over-segmentation at the cost of missing valid targets, especially in difficult domains such as camouflage, underwater, or cluttered scenes.
Efficiency vs. recall: Accelerated inference (6× faster than SAM), lower annotation workload in data collection (8.4× reported speed-up), and scalability in video come at the expense of unsupervised object detection capabilities.
Inability to capture fine details: Both qualitative and quantitative analyses (e.g., DIS and Human Correction Efforts [H_γ] metrics) indicate that SAM2, even in GT-bbox mode, does not fully resolve thin or high-frequency structures (Pei et al., 4 Sep 2024).
Generalization boundaries: While broad improvements in promptable settings are observed across domains, adaptation to domains with fundamentally different visual statistics (e.g., medical, underwater) may demand further fine-tuning or architectural adaptation for best performance.

5. Practical Applications and Deployment Strategies

Based on performance analyses, SAM2 is best suited for scenarios in which user or algorithmic prompts can be reliably obtained. These include:

Interactive and semi-automatic segmentation: Medical imaging (physician-provided points/boxes), video editing (frame-initialized interactive masks), and data annotation tools for constructing segmentation datasets all benefit from SAM2’s improved promptable accuracy and speed (Ravi et al., 1 Aug 2024, Ma et al., 6 Aug 2024).
Real-time video applications: Streaming memory and prompt propagation enable high-quality and temporally consistent segmentation for robotics, AR/VR, and autonomous driving, provided an external system can supply object locations.
Hybrid models: Combining SAM2 as a promptable module with complementary detection/localization networks (e.g., YOLOv9 or multimodal LLMs for prompt generation) can close the gap between interactive and fully automatic segmentation. Det-SAM2 illustrates such integration, supporting nearly infinite video stream processing with bounded memory footprints by batching frames and limiting memory updates per correction interval (Wang et al., 28 Nov 2024, Rafaeli et al., 13 Aug 2024).
Benchmarking and development frameworks: The rich experimental methodology and metrics suite in the COD task provide a valuable testbed for evaluating future model extensions.

6. Future Research Directions

Several key directions emerge from the analyses:

Bridging prompt dependence: Reducing SAM2’s reliance on strong prompts requires integrating unsupervised or self-supervised objectness discovery, more robust proposal generation, or fusing with explicit detectors.
Adapter and hybrid architectures: There is a strong rationale for developing efficient adapters (e.g., for domain adaptation, multi-modal fusion, cross-view detail enhancement) or hybrid prompt-detection architectures to balance promptability with auto mode recall (Pei et al., 4 Sep 2024).
Fine detail recovery: Enhanced decoders or multi-scale refinement modules are needed to address fine structure segmentation limits observed in high-resolution tasks and class-agnostic settings.
Generalization and domain adaptation: Extending SAM2’s success in promptable segmentation to challenging domains (medical, underwater) may require specialized datasets, memory designs, or domain-aware prompt strategies (Lian et al., 6 Aug 2024, Ma et al., 6 Aug 2024).
Maintaining efficiency: All future improvements must preserve—or minimally impact—SAM2’s favorable inference speed and memory usage characteristics, essential for deployment in real-time and edge systems.

7. Summary

SAM2 represents a substantial advance in promptable image and video segmentation, with architectural and data innovations delivering SOTA performance in guided settings and unprecedented annotation and inference efficiency. Its limitations in prompt-free (auto) object discovery, however, reveal fundamental trade-offs in the current design—improved interactive segmentation comes at the expense of unsupervised capability and fine detail localization. These findings define the landscape for future research: developing models that unite the strengths of promptable efficiency/accuracy with greater autonomy and generalization in complex, real-world settings (Tang et al., 31 Jul 2024, Ravi et al., 1 Aug 2024, Lian et al., 6 Aug 2024, Pei et al., 4 Sep 2024).