Track-Anything: Interactive Video Segmentation

• Track-Anything is a promptable video segmentation and tracking paradigm that employs sparse user interactions, such as clicks or text, paired with temporal memory for consistent object propagation.
• It integrates SAM for high-quality spatial segmentation and XMem for temporal propagation, achieving one-pass inference while minimizing annotation effort.
• Extensions include multimodal interaction, automated annotation tools, and on-device memory compression, broadening its applications across research, robotics, and efficient deployment.

Track-Anything is a promptable video object segmentation and tracking paradigm in which a target is initialized with sparse interaction—typically clicks, boxes, strokes, text, or a single mask—and then propagated through a video by a temporal memory or association module. The term is most directly associated with the Track Anything Model (TAM), which combines Segment Anything Model (SAM) with XMem to deliver interactive tracking and segmentation in videos with one-pass inference and without additional training (Yang et al., 2023). In later work, the same design space expanded toward multimodal interaction, automatic annotation, domain-specific tracking systems, universal zero-shot association, and lightweight SAM 2-based models optimized for mobile and other resource-constrained deployments (Zhou et al., 13 Jan 2025).

1. Origins and task formulation

Track-Anything emerged from a specific limitation of SAM. The original SAM is a zero-shot image segmentation foundation model based on ViT, trained on 11 M images / 1.1 B masks, and it supports point, box, or mask prompts. However, it has no mechanism for temporal correspondence, so when it is applied frame-by-frame to video, masks drift, flicker, and lack inter-frame consistency (Yang et al., 2023). The resulting problem is not image segmentation per se, but temporally stable prompt-conditioned video object segmentation.

This limitation was especially salient relative to prior video object segmentation (VOS) systems such as STM, AOT, and XMem. Those models typically require a full object mask at frame 0 and are trained on large annotated video sets, which increases annotation cost and introduces sensitivity to initialization errors. Interactive VOS mitigates some of that burden through scribbles or boxes, but existing methods often require many rounds of correction. TAM was introduced against this background as a one-pass, click-based design intended to reduce human effort while retaining strong mask quality and temporal continuity (Yang et al., 2023).

The core conceptual shift was to treat “track anything” not as category-specific tracking, but as promptable propagation of arbitrary objects of interest. In this formulation, the object definition is externalized into a user prompt rather than a fixed label space. This suggests a closer alignment with open-world segmentation and tracking than with conventional closed-set MOT or detector-conditioned tracking.

2. Canonical TAM architecture

TAM is a four-stage pipeline that stitches together SAM and XMem. Given video frames $\{I_0, I_1, \dots, I_T\}$, the first stage uses SAM’s prompt encoder on a small set of user clicks $P_0$ on frame $I_0$. The second stage produces the initial mask

$M_0 = D_{\text{mask}}(F_0, P),$

where $F_0 = E_{\text{image}}(I_0)$ is the ViT image feature and $D_{\text{mask}}$ is the SAM mask decoder (Yang et al., 2023).

The third stage performs temporal propagation with XMem and conditional refinement with SAM. For each subsequent frame, XMem stores key/value memories and computes matching affinities across frames, yielding a propagated mask $M_t^x$. In the formulation given for TAM,

$$M_t^x = \sigma(\text{SoftMax}(A)\cdot V_{\text{mem}}),$$

where $V_{\text{mem}}$ are memory values and $\sigma$ is a sigmoid. TAM then applies a quality check $P_0$0. If the propagated mask is judged poor, the system converts $P_0$1 into point prompts $P_0$2 together with a mask prompt and re-invokes the SAM mask decoder to obtain a refined mask; otherwise, it accepts the propagated result directly (Yang et al., 2023).

The fourth stage is optional human correction. At any frame, a user may add a few clicks to correct $P_0$3; the corrected mask is then fed back into XMem as updated memory for future propagation. This closed interaction loop is essential to TAM’s design. It is not a purely automatic tracker, but an interactive tracker whose temporal state can be re-seeded during inference.

Architecturally, the important division of labor is clear. SAM provides prompt-conditioned, high-quality spatial segmentation on individual frames, while XMem provides temporal persistence through a memory bank. TAM’s novelty lies in the orchestration of these components rather than in additional end-to-end retraining.

3. Inference regime, supervision model, and benchmark behavior

TAM is explicitly training-free at the system level. It does not require any additional parameter training or fine-tuning; SAM and XMem are used with their off-the-shelf, published, pretrained weights, and there is no joint loss over the combined system. The selection between XMem output and SAM refinement is heuristic, based on a quality threshold $P_0$4 (Yang et al., 2023). This property distinguishes TAM from later integrated video foundation models.

Performance in the original report is presented on DAVIS using the standard region and contour metrics. Region similarity is

$P_0$5

and contour accuracy is

$P_0$6

On DAVIS, TAM with click initialization and one-pass evaluation reports $P_0$7, $P_0$8, and $P_0$9; on DAVIS-17 test-dev it reports $I_0$0, $I_0$1, and $I_0$2, respectively (Yang et al., 2023).

Method	Init / Eval	Reported result
STM	Mask / One-Pass	$I_0$3
AOT	Mask / One-Pass	$I_0$4
XMem	Mask / One-Pass	$I_0$5
SiamMask	Box / One-Pass	$I_0$6
MiVOS	Scribble (8 rounds)	$I_0$7
TAM	Click / One-Pass	$I_0$8
TAM	DAVIS-17 test-dev	$I_0$9

The reported ablations clarify the behavior of the interaction loop. Single-click versus multi-click initialization exhibits diminishing returns after approximately three points. Removing SAM refinement and using only XMem drops $M_0 = D_{\text{mask}}(F_0, P),$0 by approximately $M_0 = D_{\text{mask}}(F_0, P),$1–$M_0 = D_{\text{mask}}(F_0, P),$2 points on DAVIS-2016, while skipping human corrections reduces robustness on long sequences (Yang et al., 2023). These observations indicate that TAM’s performance is not attributable to either component alone; the interactive refinement path materially affects sequence-level stability.

4. Typical failure modes and recurrent misconceptions

A recurrent misconception in this area is that strong image segmentation can be directly repurposed as video tracking. The original TAM report states the opposite: SAM performs strongly on images but poorly on consistent video segmentation when applied independently on each frame, with target drift, flicker, and inconsistent boundaries (Yang et al., 2023). Track-Anything systems therefore require an explicit temporal mechanism rather than repeated image-only prompting.

The TAM analysis identifies several failure modes. Figure-level discussion in the report attributes some failures to low texture or cluttered backgrounds, which induce target drift, and to object deformation, which induces mask fragmentation. XMem’s temporal matching helps maintain object identity, and SAM’s mask-prompt-plus-click refinement improves boundary quality, but those mechanisms do not remove all error sources (Yang et al., 2023).

The same report also notes limitations in very long-term frames, where XMem’s memory can “shrink” the mask and SAM prompt refinement can only partially recover it. Complex structures such as bicycle spokes challenge coarse prompt projection. More generally, long-range occlusion and out-of-view tracking remain weak, highly intricate masks cannot always be captured with sparse clicks, and reliance on quality heuristics of the form $M_0 = D_{\text{mask}}(F_0, P),$3 can either miss opportunities to refine or trigger over-refinement (Yang et al., 2023).

These limitations delineate the methodological boundary of early Track-Anything systems. They are strong interactive propagators, not universal solutions to long-horizon re-identification, severe occlusion, or arbitrarily fine topology reconstruction.

5. Multimodal, annotation, and application-oriented extensions

The phrase “track anything” quickly broadened from TAM itself to a wider family of systems that preserve the same prompt-then-propagate logic while changing the interaction modality, temporal backbone, or application domain.

System	Main components	Reported focus
TAM (Yang et al., 2023)	SAM + XMem	Click-based one-pass video tracking and segmentation
SAM-Track (Cheng et al., 2023)	SAM + Grounding-DINO + DeAOT	Click, stroke, and text interaction; multi-object tracking
Track Anything Annotate (Ivanov et al., 23 May 2025)	SAM 2 + XMem++ + WebUI	Video annotation and YOLO-format dataset generation
Annolid (Yang et al., 2024)	Grounding-DINO + SAM + Cutie	Markerless tracking for animal behavior analysis
TAR (Puthanveettil et al., 2024)	SAM + DINO + CLIP + visual servoing	Multimodal UAV tracking with pose estimation
MASA (Li et al., 2024)	SAM proposals + universal adapter	Zero-shot association without tracking labels

SAM-Track extends the interaction space from click prompts to click, stroke, and text, combining SAM, Grounding-DINO, and an AOT-based tracking model, DeAOT. It reports 92.0% on DAVIS-2016 Val and 79.2% on DAVIS-2017 Test with click initialization, and its pipeline includes an automatic mode that periodically detects newly appearing objects and hands them to DeAOT for continued tracking (Cheng et al., 2023). This broadens “track anything” from single-target interactive VOS toward multimodal multi-object tracking.

Track Anything Annotate reformulates the idea as an annotation system rather than as a benchmark model. Its prototype uses SAM 2 for initial segmentation, XMem++ for propagation, and a GUI/Web interface for refinement and export, producing images and annotations in YOLO format. On an RTX 2060 Super, the paper compares FastSAM and SAM 2, reporting mask prediction times of 15 ms and 50 ms, respectively, with SAM 2 using 1476 MB VRAM and providing cleaner masks (Ivanov et al., 23 May 2025). Here, “track anything” functions as a dataset-generation accelerator.

Annolid integrates Grounding-DINO, SAM, and Cutie in a human-in-the-loop workflow aimed primarily at animal behavior analysis. The system supports text-command segmentation, markerless tracking of multiple animals from single annotated frames, and correction-driven re-seeding. Reported examples include 100 % auto-tracking for two mice over 2559 frames and 100 % accuracy for fourteen ants over 35 964 frames, while dense scenarios such as eighty Drosophila required 292 corrections over 20 494 frames (Yang et al., 2024). The same pattern appears in a domain-specific context: promptable segmentation, temporal propagation, and targeted human intervention.

Track Anything Rapter (TAR) couples SAM, DINO, and CLIP with a ROS2/PX4 visual-servoing control stack on a UAV. It accepts multimodal queries such as text, images, and clicks, segments candidate masks, computes region descriptors, estimates pose, and produces control signals. In the reported Vicon-based evaluation, DINO modality yields lower mean DTW distance than bounding-box modality, with 0.62 m on the turtle case and 0.67 m on the AprilTag case versus 0.98 m for bounding-box cases (Puthanveettil et al., 2024). This shows that the Track-Anything pattern can extend beyond segmentation output into closed-loop control.

MASA is adjacent rather than derivative: it learns universal instance association from unlabeled static images using SAM-generated regions and a universal adapter, enabling foundational detection or segmentation models to track any detected objects. On TAO, the reported TETA score is 37.5 versus 29.1 for TETer under zero-shot association, and on BDD100K MOT it reports IDF1 71.7 versus 70.3 for ByteTrack (Li et al., 2024). A plausible implication is that “track anything” has bifurcated into two strands: interactive promptable propagation and universal open-set association.

6. Efficiency, compression, and on-device Track-Anything

The next major shift occurred with SAM 2 and its memory-bank-based video formulation. Later work characterizes SAM 2 as a foundation model for video segmentation and tracking anything, but also identifies the multi-stage image encoder and memory module as major barriers to deployment (Xiong et al., 2024). Efficiency research in this area therefore targets two bottlenecks simultaneously: frame feature extraction and memory attention.

EfficientTAM revisits the plain, nonhierarchical ViT as an image encoder and introduces an efficient memory module that reduces the complexity of both frame feature extraction and memory computation. It is trained on SA-1B and SA-V for video object segmentation and track anything tasks. The reported EfficientTAM-S model performs comparably to SAM 2 with approximately $M_0 = D_{\text{mask}}(F_0, P),$4 speedup on A100 and approximately $M_0 = D_{\text{mask}}(F_0, P),$5 parameter reduction, while EfficientTAM-S/2 reports 74.0% $M_0 = D_{\text{mask}}(F_0, P),$6 on the SA-V test set, 109 FPS on A100, and 450 ms/frame latency on iPhone 15 Pro Max; the abstract further states that EfficientTAMs can run at approximately 10 FPS on iPhone 15 Pro Max (Xiong et al., 2024).

EdgeTAM focuses more directly on on-device memory compression. Its central component is a 2D Spatial Perceiver that compresses dense frame-level memories with a fixed set of learnable queries while preserving spatial structure through global-level and patch-level query groups. A two-stage distillation pipeline improves performance without inference overhead. The reported results are 87.7, 70.0, 72.3, and 71.7 $M_0 = D_{\text{mask}}(F_0, P),$7 on DAVIS 2017, MOSE, SA-V val, and SA-V test, respectively, at 16 FPS on iPhone 15 Pro Max (Zhou et al., 13 Jan 2025). In this line of work, “track anything” becomes explicitly synonymous with mobile-feasible promptable video segmentation.

TinySAM 2 pushes compression further. It introduces a memory quality management mechanism that selects and retains high-informative historical frames, joint-spatial-temporal token compression that first compresses redundancy tokens in the spatial domain through average pooling and then selects informative tokens across frames in the temporal domain based on token-level similarity measurement, and a RepViT lightweight image encoder. On DAVIS and SA-V, the abstract reports that TinySAM 2 achieves 90% of the performance of SAM 2.1 with only 7% memory tokens and 3% training data (Ding et al., 18 May 2026). This suggests that the principal trajectory of recent Track-Anything research is no longer only prompt design or tracking accuracy, but compression of memory representations under tight deployment constraints.

Taken together, these developments redefine Track-Anything from a training-free composition of SAM and XMem into a broader research program spanning promptable VOS, multimodal selection, zero-shot association, annotation tooling, robotic control, and aggressive memory-aware model compression.