SAMITE: Zero-Shot Visual Object Tracker

• SAMITE is a zero-shot visual object tracker that augments SAM2 with calibrated memory selection and positional prompting, addressing occlusion and distractors.
• The Prototypical Memory Bank filters out low-quality frames by scoring foreground prototypes against temporal anchors, thereby intercepting error propagation.
• The Positional Prompt Generator, with cycle-consistent checking, generates dense mask prompts to improve spatial accuracy while mitigating prompt noise.

Searching arXiv for the specified paper and closely related SAM2-based VOT work. SAMITE is a zero-shot visual object tracker built upon SAM2 that augments SAM2’s memory-based tracking loop with calibrated memory selection and positional prompting to address occlusion, distractors, and error propagation in visual object tracking (VOT) (Xu et al., 29 Jul 2025). In the formulation reported for "SAMITE: Position Prompted SAM2 with Calibrated Memory for Visual Object Tracking" (Xu et al., 29 Jul 2025), the method targets a longstanding tension between template matching trackers, which largely ignore temporal dependencies, and autoregressive trackers, which can become biased toward seen categories during training and generalize weakly to unseen classes or out-of-domain settings. SAMITE advances SAM2-based VOT by introducing a Prototypical Memory Bank (PMB) for per-frame calibrated memory selection and a Positional Prompt Generator (PPG) with cycle-consistent checking (CCC) for dense pseudo mask prompting, while retaining SAM2’s promptable and memory-conditioned inference pipeline.

1. Problem setting and design motivation

Visual Object Tracking requires continuous localization of an arbitrary target under appearance change, occlusion, and distractors (Xu et al., 29 Jul 2025). The work situates SAMITE against two mainstream paradigms. Template matching trackers typically match a single template, usually the first ground-truth-annotated frame, to each search frame, but one-to-one template-search matching does not capture long-range dynamics. Autoregressive trackers condition current predictions on previous outputs, but during training they tend to become biased toward seen categories, leading to weak generalization to unseen classes and out-of-domain settings.

The paper identifies adapting SAM2 to VOT as promising because SAM2 is a promptable, memory-based video segmentation foundation model trained on VOS and already performs memory-conditioned inference by encoding predicted objects into memory and conditioning future frames via Memory Attention (Xu et al., 29 Jul 2025). Its default selection policy, consisting of the first prompted frame and the 6 most recent frames, yields strong zero-shot behavior. However, existing SAM2-based VOT adaptations are described as struggling with occlusions and distractors and lacking mechanisms to intercept error propagation. The paper explicitly notes that SAMURAI uses occlusion likelihood and motion filtering, and that SAM2.1++ measures mask-confidence gaps, but that both are fragile in partial occlusions and near-field distractors and cannot prevent long-term drift (Xu et al., 29 Jul 2025).

Within this framing, SAMITE is positioned as a zero-shot tracker that modifies the SAM2 loop rather than retraining it. This suggests that the principal contribution is not a new learned objective but a calibration and prompting strategy that changes which memories are trusted and how spatial priors are injected into decoding.

2. Architectural composition within SAM2

SAMITE inherits four components from SAM2: an Image Encoder that extracts $F_t \in \mathbb{R}^{H \times W \times C}$, a Memory Encoder that encodes $(F_t, \text{mask})$ as $Mem_t \in \mathbb{R}^{H \times W \times C}$, a Prompt Encoder supporting sparse prompts such as points or boxes and dense prompts such as masks, and a Mask Decoder implemented as a two-way transformer (Xu et al., 29 Jul 2025).

On top of this substrate, the method adds three functions. First, prototype and memory extraction compresses frame features into foreground and background one-pixel prototypes using mask-weighted global average pooling while also producing memory. Second, the Prototypical Memory Bank maintains per-frame $Mem_t$, $P_t^{FG}$, and $P_t^{BG}$ and calibrates memory selection through correctness scoring. Third, the Positional Prompt Generator generates positional mask prompts from selected prototypes and a positional prior, while CCC decides whether that prompt should be applied (Xu et al., 29 Jul 2025).

The tracking loop is explicitly specified. At initialization, $t=1$, the tracker extracts $F_1 = ImageEncoder(I_1)$, decodes with the ground-truth box prompt to predict $\hat{M}_1$, computes memory and prototypes, and appends them to PMB. For each subsequent frame $t \ge 2$, it extracts $(F_t, \text{mask})$0, performs PMB calibration and memory selection, conditions features through Memory Attention, generates positional prompts via PPG, verifies them with CCC, predicts $(F_t, \text{mask})$1, updates PMB, and finally converts $(F_t, \text{mask})$2 to a bounding box for VOT output (Xu et al., 29 Jul 2025).

This organization preserves SAM2’s autoregressive memory-conditioning pattern but replaces the default recent-frame memory policy with a score-based selection mechanism and supplements decoding with optional dense spatial prompts. A plausible implication is that SAMITE treats memory quality, rather than mere recency, as the core control variable for long-horizon zero-shot tracking.

3. Prototypical Memory Bank

The Prototypical Memory Bank is the calibrated memory mechanism responsible for intercepting error propagation (Xu et al., 29 Jul 2025). Its stored state consists of the per-frame memory tensor and foreground/background prototypes:

$(F_t, \text{mask})$3

$(F_t, \text{mask})$4

$(F_t, \text{mask})$5

PMB evaluates candidate memories with two correctness scores. The feature-wise score compares a candidate frame’s foreground prototype to the first frame’s foreground prototype, while the position-wise score compares it to the previous frame’s foreground prototype:

$(F_t, \text{mask})$6

The paper interprets these terms mechanistically. Occluded frames tend to have incomplete or contaminated foreground features, lowering $(F_t, \text{mask})$7 and thus reducing $(F_t, \text{mask})$8. Distractor frames may correspond to the wrong instance, so $(F_t, \text{mask})$9 falls when the tracked instance deviates spatially over time, reducing $Mem_t \in \mathbb{R}^{H \times W \times C}$0 (Xu et al., 29 Jul 2025).

The final score fuses both terms:

$Mem_t \in \mathbb{R}^{H \times W \times C}$1

with $Mem_t \in \mathbb{R}^{H \times W \times C}$2 by default and identified as best in the parameter study. Candidates are drawn from frames $Mem_t \in \mathbb{R}^{H \times W \times C}$3 with $Mem_t \in \mathbb{R}^{H \times W \times C}$4, and the tracker selects the Top-5 candidates:

$Mem_t \in \mathbb{R}^{H \times W \times C}$5

$Mem_t \in \mathbb{R}^{H \times W \times C}$6

The memory set used for conditioning thus consists of 2 anchors, frame 1 and frame $Mem_t \in \mathbb{R}^{H \times W \times C}$7, plus the Top-5 calibrated candidates, for a total of 7 memories (Xu et al., 29 Jul 2025).

Per-frame PMB entries are offloaded to CPU and appended to the memory bank. Pruning is implicit rather than explicit: only high-$Mem_t \in \mathbb{R}^{H \times W \times C}$8 frames are used for the current step, while low-scoring memories are not selected. Calibration cost is reported as $Mem_t \in \mathbb{R}^{H \times W \times C}$9 cosine computations on $Mem_t$0 prototypes, with $Mem_t$1 yielding low overhead (Xu et al., 29 Jul 2025).

The central claim is that this scoring policy prevents low-quality memories from reinforcing mistakes. In the paper’s terminology, PMB prioritizes frames whose target descriptors are simultaneously feature-accurate and spatially close, thereby suppressing occluded or distracting memories and intercepting error propagation (Xu et al., 29 Jul 2025).

4. Positional Prompt Generator and cycle-consistent checking

The Positional Prompt Generator introduces dense pseudo mask prompts that supply explicit positional clues for the target and suppress distractors (Xu et al., 29 Jul 2025). For each selected frame $Mem_t$2, it first computes discriminative prior masks using foreground and background prototype similarity:

$Mem_t$3

$Mem_t$4

This produces a per-pixel probability favoring foreground over background and can locate same-class objects, though the paper notes that it may include multiple instances (Xu et al., 29 Jul 2025).

To disambiguate instances spatially, PPG computes a positional prior from positional encodings and the previous frame’s mask:

$Mem_t$5

$Mem_t$6

The paper describes this term as encoding spatial proximity to the previous target location and suppressing distant distractors (Xu et al., 29 Jul 2025).

The final per-frame positional mask prompt multiplies discriminative and positional priors and normalizes the result:

$Mem_t$7

The tracker averages the 7 resulting prompts to obtain the prompt used for the current frame:

$Mem_t$8

This dense mask prompt is then provided to the Mask Decoder (Xu et al., 29 Jul 2025).

Because prompt quality may degrade under challenging motion or blur, SAMITE adds cycle-consistent checking. CCC decodes $Mem_t$9 with and without $P_t^{FG}$0, reverse-generates prompts to selected memories, and computes mIoU against their masks. The prompt is applied only if the mean mIoU is at least $P_t^{FG}$1, with $P_t^{FG}$2 by default (Xu et al., 29 Jul 2025). The computational complexity for discriminative priors and the positional prior is reported as $P_t^{FG}$3 per frame total, linear in pixels, and applied both in PPG and CCC.

The paper’s stated interpretation is that PPG attenuates distractors even when they are feature-similar, while CCC prevents prompt-induced drift by enforcing consistency against selected memories via an mIoU threshold (Xu et al., 29 Jul 2025). This suggests that PPG is not intended as an unconditional refinement module; rather, it is a spatial prior generator whose effects are explicitly gated for reliability.

5. Zero-shot regime, implementation, and reproducibility

SAMITE is explicitly described as zero-shot and introduces no additional training, losses, or fine-tuning on VOT datasets (Xu et al., 29 Jul 2025). It leverages pretrained SAM2 weights, and the paper states that there are no segmentation loss terms, contrastive losses, or regularization losses defined for SAMITE. PPG is training-free and based on cosine similarities and priors.

The reported hyperparameters are $P_t^{FG}$4 for PMB score fusion, $P_t^{FG}$5 for the reduced PMB candidate window, and $P_t^{FG}$6 for the CCC threshold (Xu et al., 29 Jul 2025). Backbone variants are SAM2-T, SAM2-S, SAM2-B, and SAM2-L, with main experiments using SAM2-B with 81M parameters. Input resolution for efficiency measurements is 1024×1024, and hardware is a single NVIDIA V100 32GB (Xu et al., 29 Jul 2025).

Benchmark protocols are also specified. LaSOT, LaSOT_ext, TrackingNet, NFS, and OTB are evaluated at 30 FPS, while GOT-10k is evaluated at 10 FPS. Metrics and evaluation are said to follow prior SAM2-based VOT works, enabling direct comparison (Xu et al., 29 Jul 2025). The abstract reports code availability at the project repository, while the appendix notes that code will be made public after acceptance and that reproduction instructions are in the README. License information is not specified in the paper (Xu et al., 29 Jul 2025).

A common misconception for foundation-model-based trackers is that performance gains necessarily depend on task-specific finetuning. SAMITE is presented as a counterexample to that assumption: its improvements are attributed to memory calibration and prompt generation rather than additional optimization on tracking data. That claim, however, should be read narrowly, because it concerns the method as reported and does not imply that training-free adaptation is always sufficient for all VOT domains.

6. Empirical performance and ablation profile

The paper reports experiments on six benchmarks: LaSOT, LaSOT_ext, GOT-10k, TrackingNet, NFS, and OTB (Xu et al., 29 Jul 2025). The metric suite comprises AUC, Normalized Precision ($P_t^{FG}$7), and Precision ($P_t^{FG}$8) on LaSOT and LaSOT_ext; Average Overlap (AO) and Success Rates at 0.5 and 0.75 on GOT-10k; and AUC on TrackingNet, NFS, and OTB.

Benchmark	Reported SAMITE-B result	Comparative note
LaSOT	AUC 74.9, $P_t^{FG}$9 83.4, $P_t^{BG}$0 81.4	+1.7 AUC over ODTrack-B; +2.0 AUC over SAM2.1++-B
LaSOT_ext	AUC 60.7, $P_t^{BG}$1 73.1, $P_t^{BG}$2 71.2	$P_t^{BG}$3 +3.6 over SAM2.1++-B; +7.6 over EVPTrack
GOT-10k	AO 78.9, $P_t^{BG}$4 89.9, $P_t^{BG}$5 72.5	Second-best AO among SAM2-based methods in the table
TrackingNet	AUC 84.5	Ties HIPTrack-B 84.5; surpasses SAM2.1-B 80.7
NFS	AUC 69.2	Slightly above SAM2.1-B 69.0
OTB	AUC 69.9	Competitive

On LaSOT, SAMITE-B achieves AUC 74.9, $P_t^{BG}$6 83.4, and $P_t^{BG}$7 81.4. These numbers improve over the best supervised ODTrack-B, which has AUC 73.2, $P_t^{BG}$8 83.2, and $P_t^{BG}$9 80.6, by +1.7 AUC and +0.8 $t=1$0, and over SAM2.1++-B, which has AUC 72.9, $t=1$1 81.0, and $t=1$2 78.7, by +2.0 AUC, +2.4 $t=1$3, and +2.7 $t=1$4 (Xu et al., 29 Jul 2025). On LaSOT_ext, SAMITE-B reports AUC 60.7, $t=1$5 73.1, and $t=1$6 71.2, including a $t=1$7 gain of +3.6 over SAM2.1++-B and +7.6 over EVPTrack, alongside an AUC gain of +2.2 over SAM2.1++-B (Xu et al., 29 Jul 2025).

Attribute-wise gains on LaSOT are consistent with the proposed robustness mechanisms. Against SAM2.1++-B, SAMITE-B improves AUC by +3.7 on Background Clutter, +3.1 on Partial Occlusion, +3.0 on Full Occlusion, +3.5 on Camera Motion, +3.2 on Rotation, and +3.2 on Low Resolution, with other attributes also showing consistent improvements (Xu et al., 29 Jul 2025). These gains are aligned with the paper’s claim that PMB suppresses corrupted memories and that PPG supplies explicit spatial clues.

The ablation study begins from a tailored SAM2 baseline without PMB or PPG, which yields AUC 72.1, $t=1$8 80.1, and $t=1$9 77.8 on LaSOT (Xu et al., 29 Jul 2025). PMB with only the feature-wise anchor yields AUC 73.7, and PMB with only the position-wise anchor also yields AUC 73.7, while PMB with both anchors reaches AUC 74.6. PPG alone gives AUC 71.9, which is a decrease of 0.2, and the paper attributes this to prompt noise. Adding CCC raises this to AUC 73.2, indicating that CCC effectively filters prompt noise. The full SAMITE system, combining PMB, PPG, and CCC, reaches AUC 74.9, $F_1 = ImageEncoder(I_1)$0 83.4, and $F_1 = ImageEncoder(I_1)$1 81.4, corresponding to gains of +2.8, +3.3, and +3.6 over the tailored baseline (Xu et al., 29 Jul 2025).

Sensitivity analyses indicate that $F_1 = ImageEncoder(I_1)$2, $F_1 = ImageEncoder(I_1)$3, and $F_1 = ImageEncoder(I_1)$4 are the best settings within the reported study. Specifically, $F_1 = ImageEncoder(I_1)$5 or $F_1 = ImageEncoder(I_1)$6 yields AUC 73.7, while $F_1 = ImageEncoder(I_1)$7 yields AUC 74.9. Larger $F_1 = ImageEncoder(I_1)$8 slows inference and may degrade performance with fast motion. Too small a $F_1 = ImageEncoder(I_1)$9 admits noisy prompts, whereas too large a $\hat{M}_1$0 discards useful priors (Xu et al., 29 Jul 2025).

7. Robustness, efficiency, and limitations

The paper’s account of robustness centers on error interception. PMB’s correctness scoring is said to filter out frames with incomplete semantics, because occlusions reduce $\hat{M}_1$1, and spatially inconsistent targets, because distractors reduce $\hat{M}_1$2 (Xu et al., 29 Jul 2025). The fusion

$\hat{M}_1$3

therefore prioritizes memories that are both feature-accurate and spatially close. PPG adds explicit positional priors $\hat{M}_1$4, and CCC prevents prompt-induced drift through memory-consistency verification. Qualitative examples are described as showing recovery after drift and resilience across long sequences (Xu et al., 29 Jul 2025).

The efficiency cost is measurable. At 1024×1024 input on a single V100, SAM2.1-B runs at 14.5 FPS, SAMITE-B at 9.2 FPS, SAM2.1++-B at 9.2 FPS, and SAMURAI-B at 14.3 FPS. For the large variant, SAM2.1-L runs at 10.0 FPS, SAMITE-L at 7.8 FPS, and SAM2.1++-L at 6.1 FPS (Xu et al., 29 Jul 2025). Model sizes are reported as 39M for SAM2-T, 46M for SAM2-S, 81M for SAM2-B, and 224M for SAM2-L. SAMITE operates across these sizes, and the largest gains are reported with smaller models, exemplified by SAM2-B showing AUC +8.9 over SAM2.1 (Xu et al., 29 Jul 2025).

The limitations are stated directly. First, PMB and PPG add linear complexity, and FPS drops relative to SAM2.1; the paper cites a reduction of 5.3 FPS for the B-size model and suggests further optimization of prototype computation, calibration, and prior generation (Xu et al., 29 Jul 2025). Second, while SAMITE excels in out-of-domain settings such as LaSOT_ext and GOT-10k, gains on TrackingNet and OTB are narrower, suggesting that domain-specific nuances might not be captured by SAM2-based zero-shot models. Third, prompt noise remains a failure mode under severe motion blur or rapid scale changes; CCC mitigates but does not eliminate these cases (Xu et al., 29 Jul 2025).

These limitations matter for interpreting the method’s contribution. SAMITE is not presented as a universal replacement for all tracking pipelines, but as a specific zero-shot SAM2 adaptation that improves robustness through prototype-driven memory calibration and positional prompting. A plausible implication is that its strongest value lies in settings where class-agnostic generalization and long-horizon stability are more critical than maximum throughput or aggressive domain specialization.