Seg2Track-SAM2: Zero-Shot MOTS Framework

• Seg2Track-SAM2 is a multi-object tracking and segmentation system that integrates SAM2's video capabilities with a custom module for track initialization and management.
• It combines detector-agnostic tracking-by-detection with a sliding-window memory strategy, reducing memory usage by up to 75% while maintaining performance.
• The framework achieves competitive performance on KITTI MOTS by demonstrating strong association accuracy and improved identity preservation for cars and pedestrians.

Seg2Track-SAM2 is a multi-object tracking and segmentation framework built around the SAM2 video segmentation foundation model, a tracking-by-detection pipeline, and a custom Seg2Track module for track initialization, track management, and reinforcement. It is explicitly designed for zero-shot, detector-agnostic MOTS, and it combines pre-trained object detectors with SAM2 to improve identity preservation while bounding temporal memory. The reported system ranks fourth overall for both car and pedestrian classes on KITTI MOTS, establishes a new benchmark in association accuracy, and uses a sliding-window memory strategy that reduces memory usage by up to 75% with negligible performance degradation (Mendonça et al., 15 Sep 2025).

1. Problem formulation and motivation

Seg2Track-SAM2 addresses Multi-Object Tracking and Segmentation (MOTS), where a video sequence $\{I_1,\dots,I_T\}$ is mapped to a set of object trajectories

$\mathcal{T} = \{T_1,\dots,T_K\},$

with each trajectory represented as

$$T_k = \{(t, m_{t,k}, s_{t,k}, c_{t,k}, \text{id}_k)\}_{t \in F_k}.$$

Here, $m_{t,k} \in \{0,1\}^{H \times W}$ is a binary instance mask, $s_{t,k} \in [0,1]$ is a confidence score, $c_{t,k}$ is a semantic class label, $\text{id}_k$ is a persistent track identity, and $F_k$ is the set of frames in which the object is present (Mendonça et al., 15 Sep 2025).

The framework is motivated by a specific mismatch between SAM2’s native strengths and MOTS requirements. SAM2 already provides zero-shot, promptable video segmentation with temporal memory, but its direct use for MOTS remains limited by insufficient identity management and memory efficiency. In particular, SAM2 can track prompted instances, yet it does not by itself provide full track initialization from detector outputs, explicit track life-cycle logic, or class-aware multi-object management. At the same time, conventional MOT and MOTS pipelines typically rely on dataset-specific training, learned ReID modules, or task-specific segmentation heads, which weakens zero-shot transfer and detector interchangeability (Mendonça et al., 15 Sep 2025).

Within this setting, Seg2Track-SAM2 adopts a hybrid formulation. SAM2 supplies mask-level temporal propagation and confidence estimation, while the added Seg2Track logic handles new-object creation, uncertain-track reinforcement, low-quality-track deletion, and memory-bounded execution. A common misconception is that SAM2’s internal mask IDs alone are sufficient for MOTS identity management; Seg2Track-SAM2 is explicitly built on the opposite premise, namely that detector integration and track management must be externalized and controlled (Mendonça et al., 15 Sep 2025).

2. Core architecture and data flow

The architecture has three principal components: a detector, the SAM2 video segmentation model, and the Seg2Track module. The detector is detector-agnostic; the reported experiments use YOLOv11 or TrackR-CNN detections. For each frame $I_t$, the detector produces a set of bounding boxes

$$D_t = \phi_{\text{det}}(I_t) = \{ b_{t,1}, \dots, b_{t,N_t} \}, \quad b_{t,i} \in \mathbb{R}^4.$$

SAM2 then operates in video mode, receiving the frame and a prompt set $\mathcal{T} = \{T_1,\dots,T_K\},$0 selected by Seg2Track, and returns

$\mathcal{T} = \{T_1,\dots,T_K\},$1

where $\mathcal{T} = \{T_1,\dots,T_K\},$2 is a soft mask, $\mathcal{T} = \{T_1,\dots,T_K\},$3 is an IoU-based confidence score, $\mathcal{T} = \{T_1,\dots,T_K\},$4 is an encoded mask representation used for memory, and $\mathcal{T} = \{T_1,\dots,T_K\},$5 is SAM2’s instance identifier (Mendonça et al., 15 Sep 2025).

The Seg2Track module is divided into Track Quality Assessment, Binary Mask Generation, and Object Association and Filtering. The data flow is cyclic. The detector first produces proposals; Seg2Track then filters and associates them relative to the previous frame’s masks, producing the new prompt set $\mathcal{T} = \{T_1,\dots,T_K\},$6. SAM2 uses $\mathcal{T} = \{T_1,\dots,T_K\},$7 together with its temporal memory to segment the current frame. Seg2Track then classifies the resulting masks by quality, updates the binary union mask, decides whether memory should be updated, and maintains track metadata for the next frame (Mendonça et al., 15 Sep 2025).

This organization makes the framework simultaneously detector-conditioned and segmentation-centric. New tracks originate from detections, but temporal continuity is delegated to SAM2’s video memory. A plausible implication is that the design deliberately separates class-specific proposal generation from class-agnostic mask propagation: class labels come from the detector, whereas spatial delineation and temporal mask consistency come from SAM2 (Mendonça et al., 15 Sep 2025).

3. Seg2Track module: track quality, association, and prompt selection

Track Quality Assessment uses SAM2’s IoU-based confidence to assign each mask to one of three states:

$\mathcal{T} = \{T_1,\dots,T_K\},$8

with default thresholds $\mathcal{T} = \{T_1,\dots,T_K\},$9 and $$T_k = \{(t, m_{t,k}, s_{t,k}, c_{t,k}, \text{id}_k)\}_{t \in F_k}.$$0. High-quality tracks update SAM2’s memory. Uncertain tracks are not written to memory and are candidates for reinforcement through fresh detector prompts. Low-quality tracks accumulate a failure counter and are removed if that counter reaches $$T_k = \{(t, m_{t,k}, s_{t,k}, c_{t,k}, \text{id}_k)\}_{t \in F_k}.$$1 (Mendonça et al., 15 Sep 2025).

Binary Mask Generation collapses all current masks into a single spatial prior,

$$T_k = \{(t, m_{t,k}, s_{t,k}, c_{t,k}, \text{id}_k)\}_{t \in F_k}.$$2

where $$T_k = \{(t, m_{t,k}, s_{t,k}, c_{t,k}, \text{id}_k)\}_{t \in F_k}.$$3 denotes logical OR on binarized masks. This union mask is central to proposal filtering at the next frame, because detections are tested against it rather than against boxes alone (Mendonça et al., 15 Sep 2025).

Object Association and Filtering begins by computing, for each detection box $$T_k = \{(t, m_{t,k}, s_{t,k}, c_{t,k}, \text{id}_k)\}_{t \in F_k}.$$4, its overlap with the previous union mask:

$$T_k = \{(t, m_{t,k}, s_{t,k}, c_{t,k}, \text{id}_k)\}_{t \in F_k}.$$5

Detections with $$T_k = \{(t, m_{t,k}, s_{t,k}, c_{t,k}, \text{id}_k)\}_{t \in F_k}.$$6 are treated as new-object candidates, while detections with $$T_k = \{(t, m_{t,k}, s_{t,k}, c_{t,k}, \text{id}_k)\}_{t \in F_k}.$$7 form the set

$$T_k = \{(t, m_{t,k}, s_{t,k}, c_{t,k}, \text{id}_k)\}_{t \in F_k}.$$8

The threshold is class-dependent: $$T_k = \{(t, m_{t,k}, s_{t,k}, c_{t,k}, \text{id}_k)\}_{t \in F_k}.$$9 for cars and $m_{t,k} \in \{0,1\}^{H \times W}$0 for pedestrians. For $m_{t,k} \in \{0,1\}^{H \times W}$1, Seg2Track uses the Hungarian algorithm with Euclidean distance between detection centers and previous-mask centers as the assignment cost. Only matches whose previous tracks were in the Uncertain state are used for reinforcement prompting (Mendonça et al., 15 Sep 2025).

The resulting prompt set is

$m_{t,k} \in \{0,1\}^{H \times W}$2

This construction is one of the defining mechanisms of the framework. High-quality tracks are not re-prompted, because they are expected to propagate through SAM2’s own temporal memory. Uncertain tracks are reconditioned by detections, and detections that do not overlap the prior mask support initialize new SAM2 tracks (Mendonça et al., 15 Sep 2025).

4. Zero-shot operation and bounded temporal memory

Seg2Track-SAM2 is zero-shot in the strict sense used by the paper: neither SAM2 nor the detector is fine-tuned on KITTI MOT or KITTI MOTS. SAM2 runs with an official SAM2.1-large checkpoint, detectors are used with generic pre-trained weights, and the Seg2Track logic is rule-based rather than learned. The framework is also detector-agnostic: the same tracking logic is used with YOLOv11 and TrackR-CNN, and any detector producing boxes and classes can be substituted in principle. Class awareness enters only through the detector; SAM2 itself remains class-agnostic, and newly created tracks inherit their semantic class from the initializing detection (Mendonça et al., 15 Sep 2025).

A second major design decision is the replacement of SAM2’s effectively unbounded per-object memory with a sliding state window. For each track $m_{t,k} \in \{0,1\}^{H \times W}$3, only the most recent $m_{t,k} \in \{0,1\}^{H \times W}$4 encoded states are retained. If $m_{t,k} \in \{0,1\}^{H \times W}$5 denotes the memory of track $m_{t,k} \in \{0,1\}^{H \times W}$6 at time $m_{t,k} \in \{0,1\}^{H \times W}$7, then the update is

$m_{t,k} \in \{0,1\}^{H \times W}$8

The main experiments use $m_{t,k} \in \{0,1\}^{H \times W}$9 (Mendonça et al., 15 Sep 2025).

The reported ablation over $s_{t,k} \in [0,1]$0 shows that memory usage drops by roughly 75% for small windows, while HOTA, DetA, AssA, and LocA stabilize near the full-history baseline around $s_{t,k} \in [0,1]$1. This directly contradicts the common assumption that full-history memory is necessary for SAM2-based tracking. In the KITTI regime examined by the paper, most useful temporal information lies in a moderate recent window, making bounded-memory execution a practical deployment strategy rather than a large accuracy compromise (Mendonça et al., 15 Sep 2025).

5. Reported empirical performance

On KITTI MOTS, Seg2Track-SAM2 reports, for cars, HOTA $s_{t,k} \in [0,1]$2, DetA $s_{t,k} \in [0,1]$3, AssA $s_{t,k} \in [0,1]$4, and LocA $s_{t,k} \in [0,1]$5; for pedestrians, HOTA $s_{t,k} \in [0,1]$6, DetA $s_{t,k} \in [0,1]$7, AssA $s_{t,k} \in [0,1]$8, and LocA $s_{t,k} \in [0,1]$9. These results correspond to fourth overall ranking in both classes while establishing the best reported association accuracy. The SAM2-only baseline is markedly lower, with car HOTA/DetA/AssA of $c_{t,k}$0 and pedestrian HOTA/DetA/AssA of $c_{t,k}$1, indicating that the added track management logic materially improves both detection-support consistency and identity continuity (Mendonça et al., 15 Sep 2025).

On KITTI 2D MOT, where masks are converted to tight bounding boxes before evaluation, the framework reports for cars HOTA $c_{t,k}$2, DetA $c_{t,k}$3, AssA $c_{t,k}$4, MOTA $c_{t,k}$5, and $c_{t,k}$6 identity switches, and for pedestrians HOTA $c_{t,k}$7, DetA $c_{t,k}$8, AssA $c_{t,k}$9, MOTA $\text{id}_k$0, and $\text{id}_k$1 identity switches. The paper attributes part of the gap to stronger 2D+3D trackers to the visible-versus-amodal mismatch: boxes derived from visible masks can be smaller than ground-truth boxes under occlusion, which depresses localization-oriented measures even when identity preservation remains strong (Mendonça et al., 15 Sep 2025).

The ablation study isolates Track Quality Assessment as the dominant contributor. With TrackR-CNN detections, the baseline HOTA is $\text{id}_k$2 for cars and $\text{id}_k$3 for pedestrians; adding TQA raises these to $\text{id}_k$4 and $\text{id}_k$5, and adding the full TQA+OAF stack yields $\text{id}_k$6 and $\text{id}_k$7. With YOLOv11, the same progression is $\text{id}_k$8 for cars and $\text{id}_k$9 for pedestrians. The paper’s interpretation is that TQA prevents unreliable masks from contaminating memory and that OAF adds a further gain by selectively re-prompting uncertain tracks and filtering new-object proposals (Mendonça et al., 15 Sep 2025).

6. Position in the SAM2 tracking literature, misconceptions, and limitations

Seg2Track-SAM2 belongs to a broader family of SAM2-based tracking systems, but it occupies a distinct niche. Its central problem is zero-shot, class-aware multi-object tracking and segmentation with detector integration, whereas several contemporaneous SAM2 studies concentrated on single-object tracking or memory-policy refinement. DAM4SAM introduced a distractor-aware memory split into Recent Appearance Memory and Distractor Resolving Memory for visual object tracking (Videnovic et al., 2024). SAMITE added a prototypical memory bank and a positional prompt generator to intercept error propagation and suppress distractors (Xu et al., 29 Jul 2025). SENTRY reframed SAM2 memory writing as a refine-before-write validation problem based on short-horizon temporal consistency and neighbor-aware cycle consistency (Alansari et al., 23 Jun 2026). A plausible implication is that Seg2Track-SAM2 addresses the MOTS-specific layer of the design space—detector coupling, class inheritance, prompt selection, and bounded memory—while these other systems primarily optimize how SAM2 stores or validates temporal evidence.

The framework also has a direct successor. Seg2Track++ extends Seg2Track-SAM2 with Mask Centroid Distance, Confidence-Aware Cost Modulation, and Probabilistic Track Validation, explicitly targeting unreliable association and ghost-track suppression in zero-shot MOTS (Mendonça et al., 2 Jun 2026). This follow-up supports an objective reading of Seg2Track-SAM2’s main limitation: although its association accuracy is strong, false positives from the detector can still be propagated by SAM2 until the quality logic removes them.

Three recurring misconceptions can therefore be corrected. First, zero-shot does not mean detector-free; Seg2Track-SAM2 depends on a pre-trained detector for proposals and class labels (Mendonça et al., 15 Sep 2025). Second, SAM2’s internal identity handling is not equivalent to full MOTS track management; explicit track initialization, reinforcement, and deletion remain necessary (Mendonça et al., 15 Sep 2025). Third, unlimited temporal memory is not automatically beneficial; the reported sliding-window analysis shows that a moderate window can preserve HOTA, DetA, AssA, and LocA while sharply reducing memory (Mendonça et al., 15 Sep 2025).

The limitations reported by the paper are correspondingly concrete. Performance still depends on detector quality, the system is 2D-only, and conversion from visible masks to boxes can underperform against amodal box annotations in KITTI MOT. Extremely crowded interactions can also stress the center-distance assignment used inside OAF. Even so, the framework establishes that SAM2 can serve as the segmentation-and-memory core of a competitive zero-shot MOTS system when surrounded by explicit proposal filtering, confidence-gated memory writes, and bounded temporal state management (Mendonça et al., 15 Sep 2025).