---
title: 'SAM 2++: Enhanced Tracking & Segmentation'
url: https://www.emergentmind.com/topics/sam-2
type: topic
---

# SAM 2++: Enhanced Tracking & Segmentation

**SAM 2++** denotes a line of research that extends the Segment Anything Model 2 (SAM 2) beyond promptable video object segmentation toward more robust tracking and domain-adapted segmentation. In the current literature, the term has two coexisting uses. First, it functions as an umbrella label for enhanced SAM 2 systems that add automated prompting, specialized memory, biomedical adaptation, or tracking refinements. Second, it names a specific unified model for tracking masks, boxes, and points with task-specific prompt encoding, a unified decoder, task-adaptive memory, and a dedicated tri-granular dataset [2510.18822]. The dual usage is consequential: SAM 2++ is not yet a single standardized architecture, but a family of designs centered on improving prompt formation, memory matching, temporal stability, and adaptation to context-dependent concepts [2412.01240].

## 1. Terminological scope and historical usage

The expression “SAM 2++” emerged as a shorthand for improvements over vanilla SAM 2 in settings where the base model exhibits prompt sensitivity, temporal drift, weak context modeling, or domain mismatch. In this broader sense, the label has been attached to biomedical adaptation, automated prompting for medical segmentation, and memory-stabilized ultrasound tracking, while a later work formalized it as the title of a unified tracking model [2502.03501] [2408.03286] [2510.18213] [2510.18822].

| Variant | Primary modification | Domain |
|---|---|---|
| Proxy Prompt | auto-generated proxy prompt from non-target image-mask pairs | few-shot medical segmentation |
| BioSAM 2 | frozen prompt encoder with fine-tuned image encoder and mask decoder | biomedical images and videos |
| EMA-SAM | confidence-weighted exponential moving average pointer in the memory bank | PTMC segmentation in ultrasound videos |
| SAM 2++ | task-specific prompts, unified decoder, task-adaptive memory | tracking at any granularity |

A common misconception is that the name refers only to the unified tracker introduced in late 2025. The literature instead shows a broader pattern in which “SAM 2++” can denote either a specific architecture or a research agenda for strengthening SAM 2 under harder regimes, especially context-dependent concepts, medical imaging, and long-term video tracking [2412.01240].

## 2. Core architectural motif: prompt, memory, and decoder augmentation

Across the literature, SAM 2 is treated as a modular backbone with an image encoder, prompt encoder, memory mechanism, and promptable decoder. One explicit recap describes SAM-2 as comprising a frozen hierarchical ViT image encoder, a memory-augmented transformer, a promptable decoder that outputs a binary mask, an object-pointer token, and an occlusion score, and a fixed-capacity FIFO memory bank of spatial features and pointer tokens [2510.18213]. Enhancements marketed or discussed as SAM 2++ almost always intervene at one of four loci.

The first locus is **prompt construction**. Some works replace manual clicks or boxes with structured prompts derived from context, exemplars, or alternate granularities. Proxy Prompt replaces direct target prompting with contextual embeddings extracted from non-target image-mask pairs [2502.03501]. The formal SAM 2++ tracker encodes mask prompts, box prompts, and point prompts into a shared prompt space, with point tracking additionally using a Gaussian mask [2510.18822].

The second locus is **memory design**. Memory is the dominant intervention point in video and volumetric settings. EMA-SAM inserts a persistent EMA pointer that is never evicted from the bank [2510.18213]. SLM-SAM 2 splits memory into short-term and long-term banks with separate attention modules [2505.01854]. HiM2SAM distinguishes short-term and long-term memory frames under motion- and distractor-aware criteria [2507.07603]. CamSAM2 adds object prototypes distilled from high-resolution masked features of previous frames [2503.19730].

The third locus is **decoder specialization**. BioSAM 2 simplifies the mask decoder to output exactly one mask per frame or slice, dropping the mask-ranking head and replacing bidirectional cross-attention with single-mask transformer layers [2408.03286]. The unified SAM 2++ tracker retains a two-way-transformer style decoder but adds heads for masks, boxes, and points [2510.18822]. CamSAM2 appends a learnable decamouflaged token to the decoder without modifying SAM2’s original weights [2503.19730].

The fourth locus is **adaptation regime**. The spectrum ranges from trainless augmentation to full joint training. HiM2SAM is explicitly trainless [2507.07603]. Proxy Prompt fine-tunes only LoRA adapters while keeping the original SAM encoder and decoder otherwise frozen [2502.03501]. BioSAM 2 fine-tunes the image encoder and mask decoder while freezing the prompt encoder [2408.03286]. The formal SAM 2++ model is jointly trained across video object segmentation, single-object tracking, and point tracking with task-specific memory modules [2510.18822]. This suggests a shared engineering view of SAM 2 as a prompt-memory-decoder substrate rather than a fixed end model.

## 3. The formal unified model: tracking masks, boxes, and points

The paper titled **“SAM 2++: Tracking Anything at Any Granularity”** defines the most specific use of the term. Its objective is to unify three tasks that differ only in target granularity: mask-level video object segmentation, bounding-box single-object tracking, and point tracking [2510.18822]. The central claim is that all three can be cast within a memory-matching framework if prompts, decoder outputs, and memory representation are made task-adaptive rather than task-isolated.

At the input side, SAM 2++ uses a shared prompt encoder with task-specific embeddings. Mask prompts are dense binary masks, box prompts are two corner points, and point prompts are a 2D coordinate plus a Gaussian mask. In the formulation reported for the model,
$$
\mathcal P_{dense}=
\begin{cases}
\mathrm{Conv}_{dense}(I_{mask}), & \text{mask prompt},\\
\mathrm{Conv}_{dense}(G_{point}), & \text{point prompt},\\
0, & \text{otherwise}.
\end{cases}
$$
The Gaussian mask is annealed from coarse to fine during training, so point localization is first broad and later precise [2510.18822].

At the output side, the model extends SAM 2’s mask decoder into a unified decoder. Memory-conditioned image features, sparse prompt tokens, dense prompt features, mask tokens, an object-existence token, and an IoU-prediction token are fused by a Two-Way Transformer. The resulting token set can produce candidate masks, boxes via a corner head, and points via soft-argmax during training or hard argmax at inference. Candidate quality is scored by the IoU token, and the best candidate is selected for the task [2510.18822].

The memory mechanism is task-adaptive rather than fully shared. For each granularity $\rho\in\{\text{mask, box, point}\}$, the model uses a dedicated memory encoder and decoupled LoRA parameters inside memory attention:
$$
\bar{\bar F}_{img,\rho}=\mathrm{MemEn}_\rho(F_{img},M^*_\rho),\qquad
\mathcal{MB}_\rho=\mathrm{FIFO}\bigl(\mathcal{MB}_\rho\cup\{\bar{\bar F}_{img,\rho},\varepsilon_{pointer,\rho}\}\bigr),\qquad
\bar F_{img}=\mathrm{MemAttn}_\rho(F_{img},\mathcal{MB}_\rho).
$$
The rationale is that mask tracking requires fine pixel-level memory, box tracking requires coarse localization, and point tracking requires tightly localized memory around a Gaussian prompt [2510.18822].

Training is organized around a customized data engine and the **Tracking-Any-Granularity (TAG)** dataset. TAG contains **6 000 videos** and **2.2 M frames**, with **2.15 M segmentation masks**, **2.15 M bounding boxes**, and **2.64 M key-points**; it is split into **150 val + 150 test videos** [2510.18822]. A single batch comes from exactly one task, sampled with probability approximately **0.1:0.4:0.5** for mask:box:point. Each training sequence has **8 frames**, **1–2 conditional frames**, and up to **7 corrective clicks**. Losses combine focal and Dice mask supervision, IoU regression, object-existence classification, and either box or point regression depending on the task [2510.18822].

Empirically, the model is reported to set a new state of the art across multiple benchmarks. On video object segmentation it achieves **66.4 HOTA on BURST**, **82.2 / 78.7 / 85.7 on LVOS**, **74.6 / 70.6 / 78.6 on MOSE**, and **87.4 / 84.2 / 90.7 on TAG-VOS**. On single-object tracking it reports **80.7 AO on GOT-10k**, **78.0 AUC on TAG-SOT**, and **86.0 AUC on TrackingNet**. On point tracking it reaches **72.9 PCK-T on BADJA**, **66.2 AJ on TAG**, and **37.7 AJ on TAG-PT** [2510.18822]. The associated ablations indicate that decoupling only the memory encoders and LoRA adapters per task restores performance lost under naive parameter sharing, and that joint training across granularities yields positive transfer rather than task interference [2510.18822].

## 4. Biomedical and medical reinterpretations

Before the formal tri-granularity tracker, several biomedical papers used “SAM 2++” to describe domain-specific extensions of SAM 2. These systems are unified less by a canonical architecture than by a shared attempt to overcome weak prompting, poor low-level adaptation, or unstable propagation in medical imagery.

**Proxy Prompt** treats SAM and SAM 2 as promptable backbones and replaces manual target prompting with an auto-generated contextual prompt derived from non-target image-mask pairs [2502.03501]. Its **3-step context-selection strategy** begins with Vision Mamba encoding of concatenated support image-mask pairs, continues with a Bridge Unit that concatenates frozen SAM encoder features with Mamba outputs and applies CBAM plus residual blocks, and ends with a Selective Map that computes target relevance. The Selective Map is defined as
$$
\mathrm{Selective}=\bigl[2(\hat F_{\rm sup}^T\hat F_x)-\hat F_{\rm sup}^2\bigr]/\sqrt{C},
$$
after which
$$
E_{\rm ctx}=A_{\rm agg}\cdot \mathrm{Softmax}(\mathrm{Selective}).
$$
A **Contextual Colorization Module** with **$L=4$ identical blocks** then alternates feature-to-context and reversed context-to-feature cross-attention, sharpening channels aligned with user-defined mask regions and amplifying distinctive edges and textures [2502.03501]. In the SAM 2 case, the resulting prompt is injected into memory attention as high-dimensional prompt vectors. The method fine-tunes only LoRA adapters, uses **$M=16$ training support pairs per modality**, and reports few-shot Dice scores of **88.2%** for **REFUGE2 Disc**, **85.6%** for **REFUGE2 Cup**, **78.9%** for **STARE Vessel**, **81.5%** for **FPA-PS**, and **88.0%** for **FPA-FH**, for an **overall average of 84.4%**. On the **JNU-IFM** video benchmark it improves over **MedSAM 2 by +17.5 pp** and over **SAM 2-box by +2.3 pp**, with **$\sigma(\mathrm{Dice})\le 1.0\%$ across 5 random support pairs**, compared with **29.1%** for MedSAM 2 [2502.03501].

**BioSAM 2** adapts SAM 2 to biomedical images and videos by freezing the prompt encoder and fine-tuning only the image encoder and mask decoder [2408.03286]. It uses the **tiny Hiera transformer** with **12 transformer blocks** and **384-dim embeddings**, applies layer decay, keeps positional embeddings unchanged, and simplifies the decoder to output **exactly one mask per frame or volume slice**, dropping the mask-ranking head while retaining the auxiliary occlusion head [2408.03286]. Training uses point prompts only, with 1–5 automatically sampled clicks, Dice plus BCE loss, AdamW with **initial learning rate $1e{-4}$**, and cosine decay after warm-up [2408.03286]. On **EndoVis 2017 Instruments**, BioSAM 2 reports **0.6251±0.2897 DSC** and **0.6427±0.3095 NSD**; on the **NeurIPS 2022 Cell Segmentation Challenge**, it reports **0.5792±0.2666 F1** and **0.7436±0.2104 NSD**. Relative to the best zero-shot SAM 2 baseline with 5 clicks, the gains are **+0.087 on DSC** and **+0.090 on F1** [2408.03286]. The same source notes that BioSAM 2 video results were not yet reported.

**SLM-SAM 2** addresses volumetric medical annotation by replacing SAM 2’s single memory-attention branch with a dual-branch short-long memory module [2505.01854]. It preserves the image encoder, prompt encoder, and mask decoder, but conditions the current slice on a short-term bank $\mathcal M_{short}$ and a long-term bank $\mathcal M_{long}$ with separate attention modules and a lightweight fusion module. The short-term bank has capacity **$N_s=1$**, while the long-term bank keeps **$N_{long}=6$** recent plus **$M_{long}=1$** prompted encoding [2505.01854]. This design targets boundary over-propagation, especially when the target is present in a previous slice but absent in the current one. In the **5-Volume** setting it reports an average **DSC gain of approximately +0.14** over SAM 2, and in the **1-Volume** setting an average **+0.11** gain. The key ablation gives **0.69** DSC for the SAM 2 baseline, **0.78** for a recent-only bank of size 1, and **0.83** for the full short-long memory configuration [2505.01854]. On target-absent slices, SLM-SAM 2 maintains **near-perfect DSC (≈1.0)**, whereas SAM 2’s DSC drops to zero under complete over-segmentation [2505.01854].

Taken together, these biomedical variants show that “SAM 2++” in medicine usually refers not to a broader general tracker, but to methods that re-engineer prompting, decoder specialization, or propagation while keeping the SAM 2 backbone recognizable.

## 5. Memory-centric and scene-specific video extensions

A second major branch of SAM 2++ research emphasizes temporal robustness under occlusion, drift, camouflage, and long-term reappearance. These works often aim for minimal perturbation of SAM 2 while modifying the memory pathway or adding small task-specific modules.

**EMA-SAM** inserts a persistent **EMA pointer** into SAM-2’s memory bank [2510.18213]. If $p_t$ is the instantaneous pointer token and $c_t\in[0,1]$ is visibility confidence, the update rule is
$$
\alpha_t=\alpha_0(1-c_t),\qquad
\mathbf m_t=\alpha_t\mathbf m_{t-1}+(1-\alpha_t)\mathbf p_t,\qquad
\|\mathbf m_t\|_2=1,
$$
with **$\alpha_0=0.9$** and amplification **$\gamma=1.5$** for cross-attention [2510.18213]. High confidence makes the prototype quickly track the current pointer; low confidence makes it freeze. The method adds only **one small MLP (∼0.01 M parameters)** plus a vector update and normalization, for **total additional FLOPs < 0.1 %** of SAM-2’s baseline **∼200 GFLOPs per frame**, while preserving **≈30 FPS on a single A100** [2510.18213]. On the **PTMC-RFA** dataset it improves **maxDice from 0.82 to 0.86**, **maxIoU from 0.72 to 0.76**, and reduces false positives from **≈0.14 to 0.10 (–29%)**. On external benchmarks it raises **CVC-300-TV maxDice from 0.906 to 0.924**, **CVC-612-V from 0.901 to 0.907**, and keeps **CVC-612-T maxDice at 0.880** while improving maxIoU from **0.820 to 0.833** [2510.18213].

**CamSAM2** targets video camouflaged object segmentation without modifying SAM2’s original weights [2503.19730]. It adds a learnable **decamouflaged token** $\mathbf T\in\mathbb R^{1\times256}$, **Implicit Object-Aware Fusion (IOF)** for combining early-layer features with memory-conditioned features, **Explicit Object-Aware Fusion (EOF)** for refinement using current mask logits and temporal prototypes, and **Object Prototype Generation (OPG)** to summarize masked high-resolution features into a small set of prototypes via farthest point sampling and one iteration of k-means [2503.19730]. The final logits are modulated by the decamouflaged token output:
$$
\mathbf R^c_t=\mathrm{MLP}(\mathbf T')\odot \mathbf F^{eof\prime}_t.
$$
The full model adds only **0.5 M parameters** and reports large gains on three VCOS benchmarks. On **MoCA-Mask** with **1-click prompt** and **Hiera-T**, mDice rises from **52.1** to **64.3** and mIoU from **44.8** to **54.6**. On **SUN-SEG-Hard** with **mask prompt**, mDice rises from **61.0** to **80.6**, a **+19.6** gain [2503.19730]. The ablation on MoCA-Mask shows progressive gains from the decamouflaged token, IOF, EOF, and OPG, with the full model reaching **64.3 mDice / 54.6 mIoU** [2503.19730].

**HiM2SAM** pursues long-term tracking through a trainless combination of hierarchical motion estimation and memory-bank optimization [2507.07603]. The first motion stage is a **Kalman filter** over box center, velocity, and scale. The second stage is a selective **point-tracker** refinement using **CoTracker3**, invoked only when the combined confidence falls below a threshold [2507.07603]. Memory is split into **short-term** and **long-term** pools; frames enter short-term memory when both motion and mask-overlap criteria are met, and enter long-term memory when proposal ambiguity measured by directed Hausdorff distance exceeds a threshold [2507.07603]. On **LaSOT**, the large model improves **AUC from 68.5 to 75.1**, a **+9.6% relative** gain; on **LaSOT\(_{\text{ext}}\)** it improves from **58.6 to 62.8**, a **+7.2% relative** gain [2507.07603]. The ablation on LaSOT shows **68.32 AUC** for baseline SAM2, **70.81** with Kalman filter only, **72.67** with short-term memory only, **73.71** with long-term memory only, and **75.09** for the full system, at **+3.68 ms** latency per frame [2507.07603].

These variants share a narrow technical premise: substantial gains can be obtained by reweighting or restructuring memory, rather than by replacing the SAM 2 backbone. This is particularly clear in EMA-SAM, CamSAM2, and HiM2SAM, which all preserve most of the original architecture and intervene through lightweight tokens, prototypes, or motion-aware memory rules.

## 6. Evaluation findings, misconceptions, and research trajectory

The most systematic precursor to the SAM 2++ agenda is a comprehensive evaluation of SAM and SAM 2 on **11 context-dependent concepts** across **2D and 3D images and videos** in natural, medical, and industrial scenes [2412.01240]. The study shows that SAMs remain strong on context-independent categories, with the original SAM paper reporting **mIoU in the mid-80 percent range** on COCO Panoptic validation for rigid classes such as people, cars, and roads. By contrast, performance on context-dependent concepts is far less stable. For **Salient Object Detection**, **SAM (box)** reaches **$F^\omega_\beta\approx0.82$**, **$S_m\approx0.85$**, and **MAE $\approx0.025$**, while **SAM 2 (box)** is slightly weaker at **$F^\omega_\beta\approx0.80$**, **$S_m\approx0.83$**, and **MAE $\approx0.030$**. Automatic “everything” prompts drop to **$F^\omega_\beta\approx0.55$** [2412.01240]. On **2D medical lesion segmentation**, **SAM box** achieves approximately **0.70 IoU / 0.82 Dice**, point prompting about **0.65 / 0.78**, and automatic prompting about **0.50 / 0.65** [2412.01240]. On **BraTS2020 3D brain tumors**, **SAM 2 (5 frames)** reaches **IoU ≈ 0.82** and **Dice ≈ 0.88**, surpassing a **3D U-Net baseline** [2412.01240].

The same evaluation isolates **prompt sensitivity** as a central weakness. Perturbations of **±10 px** in point coordinates, **±10%** box-edge jitter, or **5-step** mask erosion or dilation can produce **5–10 pp relative drops** in IoU or F-measure [2412.01240]. The reported relative change
$$
\Delta=\frac{\mathrm{Perf}_{\mathrm{perturbed}}-\mathrm{Perf}_{\mathrm{ideal}}}{\mathrm{Perf}_{\mathrm{ideal}}}
$$
has standard deviation across five trials of **$O(1e{-3})$**, but mean $\Delta$ is often **−0.05 to −0.10**, indicating systematic rather than merely noisy degradation [2412.01240]. A second misconception is therefore that SAM 2++ work is only about scaling or domain-specific data; much of it is an attempt to mitigate brittle prompt-response behavior.

The same source also reports that **in-context learning** in SAM 2 can already recover part of the gap. Supplying **$K=20$ exemplar image-mask pairs** before inference yields **5–8 pp IoU gain** on COD, TOS, SD, and four LOS tasks, with IoU increasing from **∼0.60 to ∼0.68** [2412.01240]. This finding anticipates later exemplar-driven or context-driven methods such as Proxy Prompt, and suggests that memory conditioning on support data is a general route to SAM 2++ behavior.

The future directions proposed across the literature are consistent. The evaluation paper recommends **hybrid multi-stage prompt embedding**, memory encoders for **arbitrary exemplar modalities**, **cross-prompt attention heads**, joint CI–CD pretraining on a unified **ConceptNet**, concept-balanced sampling with
$$
w_i=\frac{1}{\log(1+N_i)},
$$
prompt-noise augmentation, multi-modal prompts, and **dual-branch prompt decoders** [2412.01240]. Biomedical work proposes **semantic heads**, **text/image co-grounding**, **learnable cold-start modules**, **PETL/LoRA-style adapters**, and **3D-aware attention or 3D conv layers** [2408.03286]. EMA-SAM further suggests **multi-object or multi-class tracking by maintaining separate EMA pointers**, **adaptive gain schedules**, **weakly supervised or semi-supervised EMA update**, and **cross-modal integration** such as Doppler or elasticity [2510.18213].

A plausible implication is that future uses of “SAM 2++” will continue to span two levels simultaneously: a specific unified tracking model on one hand, and a broader methodological program for making SAM 2 less prompt-fragile, more memory-stable, and more adaptable to context-dependent or domain-shifted regimes on the other.

Source: https://www.emergentmind.com/topics/sam-2