---
title: 'Seg2Track-SAM2: Zero-Shot MOTS Framework'
url: https://www.emergentmind.com/topics/seg2track-sam2
type: topic
---

# Seg2Track-SAM2: Zero-Shot MOTS Framework

Seg2Track-SAM2 is a multi-object tracking and segmentation framework built around the SAM2 video segmentation foundation model, a tracking-by-detection pipeline, and a custom Seg2Track module for track initialization, track management, and reinforcement. It is explicitly designed for zero-shot, detector-agnostic MOTS, and it combines pre-trained object detectors with SAM2 to improve identity preservation while bounding temporal memory. The reported system ranks fourth overall for both car and pedestrian classes on KITTI MOTS, establishes a new benchmark in association accuracy, and uses a sliding-window memory strategy that reduces memory usage by up to 75% with negligible performance degradation [2509.11772].

## 1. Problem formulation and motivation

Seg2Track-SAM2 addresses Multi-Object Tracking and Segmentation (MOTS), where a video sequence $\{I_1,\dots,I_T\}$ is mapped to a set of object trajectories
$$
\mathcal{T} = \{T_1,\dots,T_K\},
$$
with each trajectory represented as
$$
T_k = \{(t, m_{t,k}, s_{t,k}, c_{t,k}, \text{id}_k)\}_{t \in F_k}.
$$
Here, $m_{t,k} \in \{0,1\}^{H \times W}$ is a binary instance mask, $s_{t,k} \in [0,1]$ is a confidence score, $c_{t,k}$ is a semantic class label, $\text{id}_k$ is a persistent track identity, and $F_k$ is the set of frames in which the object is present [2509.11772].

The framework is motivated by a specific mismatch between SAM2’s native strengths and MOTS requirements. SAM2 already provides zero-shot, promptable video segmentation with temporal memory, but its direct use for MOTS remains limited by insufficient identity management and memory efficiency. In particular, SAM2 can track prompted instances, yet it does not by itself provide full track initialization from detector outputs, explicit track life-cycle logic, or class-aware multi-object management. At the same time, conventional MOT and MOTS pipelines typically rely on dataset-specific training, learned ReID modules, or task-specific segmentation heads, which weakens zero-shot transfer and detector interchangeability [2509.11772].

Within this setting, Seg2Track-SAM2 adopts a hybrid formulation. SAM2 supplies mask-level temporal propagation and confidence estimation, while the added Seg2Track logic handles new-object creation, uncertain-track reinforcement, low-quality-track deletion, and memory-bounded execution. A common misconception is that SAM2’s internal mask IDs alone are sufficient for MOTS identity management; Seg2Track-SAM2 is explicitly built on the opposite premise, namely that detector integration and track management must be externalized and controlled [2509.11772].

## 2. Core architecture and data flow

The architecture has three principal components: a detector, the SAM2 video segmentation model, and the Seg2Track module. The detector is detector-agnostic; the reported experiments use YOLOv11 or TrackR-CNN detections. For each frame $I_t$, the detector produces a set of bounding boxes
$$
D_t = \phi_{\text{det}}(I_t) = \{ b_{t,1}, \dots, b_{t,N_t} \}, \quad b_{t,i} \in \mathbb{R}^4.
$$
SAM2 then operates in video mode, receiving the frame and a prompt set $P_t$ selected by Seg2Track, and returns
$$
M_t = \phi_{\text{sam}}(I_t, P_t) =
\{(m_{t,i}, s_{t,i}, e_{t,i}, \text{id}_{t,i})\}_{i=1}^{N_t},
$$
where $m_{t,i}$ is a soft mask, $s_{t,i}$ is an IoU-based confidence score, $e_{t,i}$ is an encoded mask representation used for memory, and $\text{id}_{t,i}$ is SAM2’s instance identifier [2509.11772].

The Seg2Track module is divided into Track Quality Assessment, Binary Mask Generation, and Object Association and Filtering. The data flow is cyclic. The detector first produces proposals; Seg2Track then filters and associates them relative to the previous frame’s masks, producing the new prompt set $P_t$. SAM2 uses $P_t$ together with its temporal memory to segment the current frame. Seg2Track then classifies the resulting masks by quality, updates the binary union mask, decides whether memory should be updated, and maintains track metadata for the next frame [2509.11772].

This organization makes the framework simultaneously detector-conditioned and segmentation-centric. New tracks originate from detections, but temporal continuity is delegated to SAM2’s video memory. A plausible implication is that the design deliberately separates class-specific proposal generation from class-agnostic mask propagation: class labels come from the detector, whereas spatial delineation and temporal mask consistency come from SAM2 [2509.11772].

## 3. Seg2Track module: track quality, association, and prompt selection

Track Quality Assessment uses SAM2’s IoU-based confidence to assign each mask to one of three states:
$$
S_{t,i} =
\begin{cases}
\text{High}, & s_{t,i} > \tau_h \\
\text{Uncertain}, & \tau_l < s_{t,i} \le \tau_h \\
\text{Low}, & s_{t,i} \le \tau_l
\end{cases}
$$
with default thresholds $\tau_h = 0.70$ and $\tau_l = 0.10$. High-quality tracks update SAM2’s memory. Uncertain tracks are not written to memory and are candidates for reinforcement through fresh detector prompts. Low-quality tracks accumulate a failure counter and are removed if that counter reaches $n_{\text{tries}} = 5$ [2509.11772].

Binary Mask Generation collapses all current masks into a single spatial prior,
$$
N_t(x, y) = \bigvee_{i=1}^{N_t} m_{t,i}(x,y),
$$
where $\vee$ denotes logical OR on binarized masks. This union mask is central to proposal filtering at the next frame, because detections are tested against it rather than against boxes alone [2509.11772].

Object Association and Filtering begins by computing, for each detection box $b_{t,j}$, its overlap with the previous union mask:
$$
v_j = \text{IoU}(b_{t,j}, N_{t-1})
= \frac{|b_{t,j} \cap N_{t-1}|}{|b_{t,j} \cup N_{t-1}|}.
$$
Detections with $v_j < \tau_v$ are treated as new-object candidates, while detections with $v_j \ge \tau_v$ form the set
$$
D'_t = \{ b_{t,j} \mid v_j \ge \tau_v \} \subseteq D_t.
$$
The threshold is class-dependent: $\tau_v = 0.6$ for cars and $\tau_v = 0.85$ for pedestrians. For $D'_t$, Seg2Track uses the Hungarian algorithm with Euclidean distance between detection centers and previous-mask centers as the assignment cost. Only matches whose previous tracks were in the Uncertain state are used for reinforcement prompting [2509.11772].

The resulting prompt set is
$$
P_t =
\{ b'_{t,j} \mid (m_{t-1,i}, b'_{t,j}) \in A_t,\ \tau_l < s_{t-1,i} \le \tau_h \}
\cup
\{ b_{t,j} \mid v_j < \tau_v,\ \forall b_{t,j} \in D_t \}.
$$
This construction is one of the defining mechanisms of the framework. High-quality tracks are not re-prompted, because they are expected to propagate through SAM2’s own temporal memory. Uncertain tracks are reconditioned by detections, and detections that do not overlap the prior mask support initialize new SAM2 tracks [2509.11772].

## 4. Zero-shot operation and bounded temporal memory

Seg2Track-SAM2 is zero-shot in the strict sense used by the paper: neither SAM2 nor the detector is fine-tuned on KITTI MOT or KITTI MOTS. SAM2 runs with an official SAM2.1-large checkpoint, detectors are used with generic pre-trained weights, and the Seg2Track logic is rule-based rather than learned. The framework is also detector-agnostic: the same tracking logic is used with YOLOv11 and TrackR-CNN, and any detector producing boxes and classes can be substituted in principle. Class awareness enters only through the detector; SAM2 itself remains class-agnostic, and newly created tracks inherit their semantic class from the initializing detection [2509.11772].

A second major design decision is the replacement of SAM2’s effectively unbounded per-object memory with a sliding state window. For each track $k$, only the most recent $T_w$ encoded states are retained. If $E_k^{(t)}$ denotes the memory of track $k$ at time $t$, then the update is
$$
E_k^{(t)} =
\begin{cases}
E_k^{(t-1)} \cup \{e_{t,k}\}, & |E_k^{(t-1)}| < T_w \\
\{e_{t-T_w+1,k}, \dots, e_{t-1,k}, e_{t,k}\}, & |E_k^{(t-1)}| = T_w.
\end{cases}
$$
The main experiments use $T_w = 16$ [2509.11772].

The reported ablation over $T_w \in \{3,\dots,30\}$ shows that memory usage drops by roughly 75% for small windows, while HOTA, DetA, AssA, and LocA stabilize near the full-history baseline around $T_w \approx 15$. This directly contradicts the common assumption that full-history memory is necessary for SAM2-based tracking. In the KITTI regime examined by the paper, most useful temporal information lies in a moderate recent window, making bounded-memory execution a practical deployment strategy rather than a large accuracy compromise [2509.11772].

## 5. Reported empirical performance

On KITTI MOTS, Seg2Track-SAM2 reports, for cars, HOTA $74.13\%$, DetA $71.03\%$, AssA $78.15\%$, and LocA $89.69\%$; for pedestrians, HOTA $60.00\%$, DetA $56.61\%$, AssA $65.86\%$, and LocA $80.40\%$. These results correspond to fourth overall ranking in both classes while establishing the best reported association accuracy. The SAM2-only baseline is markedly lower, with car HOTA/DetA/AssA of $67.53/64.31/71.60$ and pedestrian HOTA/DetA/AssA of $51.12/50.10/57.56$, indicating that the added track management logic materially improves both detection-support consistency and identity continuity [2509.11772].

On KITTI 2D MOT, where masks are converted to tight bounding boxes before evaluation, the framework reports for cars HOTA $60.42$, DetA $54.92$, AssA $67.95$, MOTA $61.60$, and $193$ identity switches, and for pedestrians HOTA $44.41$, DetA $39.48$, AssA $50.51$, MOTA $37.81$, and $296$ identity switches. The paper attributes part of the gap to stronger 2D+3D trackers to the visible-versus-amodal mismatch: boxes derived from visible masks can be smaller than ground-truth boxes under occlusion, which depresses localization-oriented measures even when identity preservation remains strong [2509.11772].

The ablation study isolates Track Quality Assessment as the dominant contributor. With TrackR-CNN detections, the baseline HOTA is $64.21$ for cars and $46.16$ for pedestrians; adding TQA raises these to $75.98$ and $55.18$, and adding the full TQA+OAF stack yields $77.50$ and $55.73$. With YOLOv11, the same progression is $63.86 \rightarrow 75.36 \rightarrow 76.95$ for cars and $50.79 \rightarrow 60.69 \rightarrow 60.86$ for pedestrians. The paper’s interpretation is that TQA prevents unreliable masks from contaminating memory and that OAF adds a further gain by selectively re-prompting uncertain tracks and filtering new-object proposals [2509.11772].

## 6. Position in the SAM2 tracking literature, misconceptions, and limitations

Seg2Track-SAM2 belongs to a broader family of SAM2-based tracking systems, but it occupies a distinct niche. Its central problem is zero-shot, class-aware multi-object tracking and segmentation with detector integration, whereas several contemporaneous SAM2 studies concentrated on single-object tracking or memory-policy refinement. DAM4SAM introduced a distractor-aware memory split into Recent Appearance Memory and Distractor Resolving Memory for visual object tracking [2411.17576]. SAMITE added a prototypical memory bank and a positional prompt generator to intercept error propagation and suppress distractors [2507.21732]. SENTRY reframed SAM2 memory writing as a refine-before-write validation problem based on short-horizon temporal consistency and neighbor-aware cycle consistency [2606.24449]. A plausible implication is that Seg2Track-SAM2 addresses the MOTS-specific layer of the design space—detector coupling, class inheritance, prompt selection, and bounded memory—while these other systems primarily optimize how SAM2 stores or validates temporal evidence.

The framework also has a direct successor. Seg2Track++ extends Seg2Track-SAM2 with Mask Centroid Distance, Confidence-Aware Cost Modulation, and Probabilistic Track Validation, explicitly targeting unreliable association and ghost-track suppression in zero-shot MOTS [2606.03875]. This follow-up supports an objective reading of Seg2Track-SAM2’s main limitation: although its association accuracy is strong, false positives from the detector can still be propagated by SAM2 until the quality logic removes them.

Three recurring misconceptions can therefore be corrected. First, zero-shot does not mean detector-free; Seg2Track-SAM2 depends on a pre-trained detector for proposals and class labels [2509.11772]. Second, SAM2’s internal identity handling is not equivalent to full MOTS track management; explicit track initialization, reinforcement, and deletion remain necessary [2509.11772]. Third, unlimited temporal memory is not automatically beneficial; the reported sliding-window analysis shows that a moderate window can preserve HOTA, DetA, AssA, and LocA while sharply reducing memory [2509.11772].

The limitations reported by the paper are correspondingly concrete. Performance still depends on detector quality, the system is 2D-only, and conversion from visible masks to boxes can underperform against amodal box annotations in KITTI MOT. Extremely crowded interactions can also stress the center-distance assignment used inside OAF. Even so, the framework establishes that SAM2 can serve as the segmentation-and-memory core of a competitive zero-shot MOTS system when surrounded by explicit proposal filtering, confidence-gated memory writes, and bounded temporal state management [2509.11772].

Source: https://www.emergentmind.com/topics/seg2track-sam2