---
title: Video Object Segmentation and Tracking
url: https://www.emergentmind.com/topics/video-object-segmentation-and-tracking-vost
type: topic
---

# Video Object Segmentation and Tracking

Video Object Segmentation and Tracking (VOST) addresses the intertwined problems of delineating object masks at the pixel level over time and maintaining their identities and locations as the video evolves. In the literature summarized here, VOST appears in semi-supervised, unsupervised, online, offline, multi-object, egocentric, and promptable forms: it may start from first-frame mask(s), a first-frame bounding box, a prompt such as points or boxes, or no target specification at all, yet it always couples pixel-accurate segmentation with temporal association and identity preservation [1904.09172] [2507.22792].

## 1. Scope, task settings, and conceptual boundaries

The field spans several task regimes. Semi-supervised formulations provide first-frame mask(s) and ask for mask propagation over the remaining frames; self-supervised work explicitly treats semi-supervised VOS as dense tracking; box-initialized methods seek class-agnostic segmentation from only a first-frame box; promptable formulations condition segmentation and tracking on user-provided points, boxes, or masks; and unsupervised approaches discover salient objects and track them without any target-specific input [2006.12480] [1812.05050] [2508.05829] [2001.05425].

Older survey literature organizes the space into unsupervised VOS, semi-supervised VOS, interactive VOS, weakly supervised VOS, and segmentation-based tracking [1904.09172]. More recent review work recasts the same territory through “past–present–future”: how memory from previous frames is retained and updated, how current-frame features are extracted and refined, and how motion is predicted to anticipate upcoming dynamics [2507.22792]. This suggests that VOST is not a single algorithmic template but a family of formulations distinguished by supervision, memory, temporal causality, and the representation used for identity.

A recurrent source of confusion is the relationship between VOS, VOT, MOTS, and VOST. The reviewed work does not collapse them into a single benchmark tradition. Instead, VOST sits at the interface: it inherits pixel-level evaluation from VOS, explicit temporal association from tracking, and, in multi-object settings, identity reasoning that resembles MOTS [1904.09172] [2507.22792]. Another common misconception is that VOST is necessarily mask propagation from a first-frame mask. The literature includes box-conditioned tracking and segmentation, tracklet- and proposal-based association, spectral clustering over pixels, and prompt-driven SAM/SAM2 pipelines [1707.06545] [1904.04552] [2212.08058] [2507.22792].

## 2. Mathematical formulations of segmentation and temporal association

Several papers formulate VOST as an optimization problem over space and time. One line treats segmentation as variational partitioning. In “Object Segmentation Tracking from Generic Video Cues,” frame-to-frame propagation is driven by optical-flow consistency, color, and image or motion boundaries, and the next-frame partition is obtained by minimizing a boundary-weighted energy of the form
$$
\min_{\Omega_1,\dots,\Omega_n \subset \Omega} \frac{\lambda}{2}\sum_{i=1}^{n}\mathrm{Per}(\Omega_i;\Omega) \;+\; \sum_{i=1}^{n}\int_{\Omega_i} h_i(x)\,dx,
$$
subject to a disjoint partition of the image domain [1910.02258]. A related earlier formulation combines detector-derived object tubes, motion cues, and a spatio-temporal multi-label MRF over superpixels, again coupling unary location/appearance terms with spatial and temporal smoothness [1608.03066].

A second line formulates VOST as spectral grouping. “Learning a Fast 3D Spectral Approach to Object Segmentation and Tracking over Space and Time” models the video as a dense space-time graph with one node per pixel and local space-time neighborhoods, then seeks the principal eigenvector of the adjacency matrix:
$$
\mathbf{x}_s \;=\; \underset{\mathbf{x}}{\arg\max}\;\frac{\mathbf{x}^\top \mathbf{M}\,\mathbf{x}}{\mathbf{x}^\top \mathbf{x}},
\qquad
\mathbf{M}\mathbf{v}=\lambda_{\max}\mathbf{v}.
$$
The paper argues that the leading mode corresponds to the strongest spatiotemporal cluster and approximates the matrix–vector product with separable 3D filtering, avoiding explicit construction of the adjacency matrix [2212.08058].

A third line centers on learned correspondence and memory attention. In self-supervised dense tracking, a query frame is reconstructed from reference frames by local affinity weights,
$$
A_{t}^{ij}=\frac{\exp\left\langle f_{t}^{i}, f_{t-1}^{j}\right\rangle}{\sum_{n\in \mathcal{N}} \exp\left\langle f_{t}^{i}, f_{t-1}^{n}\right\rangle},
$$
and the same affinities propagate masks through time [2006.12480]. Transformer-based long-video methods generalize this to multi-scale memory matching. TAM-VT computes dense matching between query and memory tokens at multiple scales and modulates the attention logits with learned multiplicative relative-time encoding, while SAM2-style review literature expresses the same design pattern as memory read–write over key–value banks [2312.08514] [2507.22792].

These formulations differ in machinery, but they converge on the same technical objective: encode objectness, appearance, motion, and temporal continuity in a way that stabilizes per-pixel identity across frames. This suggests that the conceptual boundary between “segmentation” and “tracking” is increasingly encoded in the choice of temporal operator rather than in the output format alone.

## 3. Major architectural paradigms

One durable paradigm is tracking-by-detection. “Video Object Segmentation using Tracked Object Proposals” combines a category-independent appearance segmentation network with Faster R-CNN proposals and a temporal IoU continuity constraint on selected boxes [1707.06545]. “Proposal, Tracking and Segmentation (PTS)” decomposes the pipeline into an Object Proposal Network, an Object Tracking Network, and a Dynamic Reference Segmentation Network, using RPN-based proposals to reduce the search space, a tracker to identify the target among proposals, and dynamic references to adapt segmentation without heavy online fine-tuning [1907.01203]. “Fast Video Object Segmentation With Temporal Aggregation Network and Dynamic Template Matching” reframes semi-supervised VOS as tracking-by-detection with an online detector, short-clip temporal aggregation, Hungarian matching between detections and template banks, and an FCN mask head [2007.05687].

A closely related box-first family pursues segmentation from tracked localization. SiamMask augments a fully convolutional Siamese tracker with a segmentation branch so that a single forward pass yields objectness scores, bounding boxes, and a per-frame binary mask from only a first-frame bounding box [1812.05050]. BoLTVOS explicitly splits the problem into bounding-box-level tracking and box-to-mask segmentation, using a conditioned detector, temporal consistency rescoring over tracklets, and a Box2Seg network [1904.04552]. MITS extends this unification to the multi-object setting with a unified identification module that accepts both mask and box references and a pinpoint box predictor that learns object-centric box localization while jointly decoding masks and boxes [2308.13266].

A second paradigm is propagation with explicit memory. “Dual Temporal Memory Network for Efficient Video Object Segmentation” couples a short-term graph-based memory over local spatial-temporal neighborhoods with a long-term Simplified-GRU over object-masked pooled features [2003.06125]. Self-supervised VOS introduces a momentum memory for long-range correspondence and a sequence-specific online appearance model trained on pseudo-labels [2006.12480]. “Learning Spatial-Semantic Features for Robust Video Object Segmentation” combines pixel memory with discriminative object queries and masked query updates to focus propagation on representative object parts [2407.07760]. TAM-VT pushes this design toward clip-based online inference with multi-scale matching and decoding, specifically targeting long videos, small objects, and deformation-heavy egocentric sequences [2312.08514].

A third paradigm is explicit global association. UnOVOST builds short tracklets by optical-flow-warped IoU, merges them into long-term tracks with a tracklet-based Forest Path Cutting algorithm, and selects final tracks by video saliency [2001.05425]. MHP-VOS defers per-frame decisions by maintaining multiple box hypotheses, scoring them with motion and mask-propagation consistency, and selecting globally consistent trajectories with a Maximum Weighted Independent Set formulation plus N-scan pruning [1904.08141].

A fourth paradigm is promptable foundation-model VOST. The review literature around SAM and SAM2 frames video segmentation and tracking in terms of prompt encoding, memory attention, streaming memory, motion-aware memory selection, and trajectory-guided prompting [2507.22792]. Domain-specialized derivatives include TSMS-SAM2 for surgical video, which adds multi-temporal-scale sampling and memory splitting with pruning [2508.05829], and HQ-SMem, which combines SAM-HQ refinement, DINO-based candidate validation, smart memory, and teacher-forced feedback into DeAOT-style trackers [2507.18921].

## 4. Temporal consistency, memory management, and identity preservation

Temporal consistency is a central design target rather than a by-product. SFSeg introduces TCONT, a temporal coherence measure based on direct and reverse optical-flow alignment of soft masks, and reports that SFSeg++ improves temporal consistency IoU with respect to ground truth by +4.5% over a strong ensemble baseline [2212.08058]. In tracking-by-detection systems, temporal consistency is enforced through explicit association: PTS uses success-gated short- and long-term tracker updates [1907.01203], Fast VOS employs Hungarian matching with combined location IoU and cosine appearance similarity plus dynamic template bank updates [2007.05687], and BoLTVOS rescales track hypotheses by detector score, aspect-ratio consistency, IoU continuity, and center displacement [1904.04552].

Identity preservation in multi-object settings often requires stronger combinatorial structure. UnOVOST imposes a unique-predecessor forest over tracklets and cuts root-to-leaf paths greedily according to visual consistency and temporal density [2001.05425]. MHP-VOS maintains multiple hypotheses over time and selects a globally consistent subset through MWIS, thereby postponing ambiguous local decisions until more evidence accumulates [1904.08141]. MITS avoids separate association by propagating a unified identification embedding for all targets, allowing all objects to be processed simultaneously from encoding through decoding [2308.13266].

Recent work increasingly treats memory itself as the locus of temporal robustness. TAM-VT keeps the initial reference frame fixed in memory, appends only the last frame of each processed clip, and uses multiplicative relative-time encoding to preserve recency while retaining non-zero importance for the first reference frame [2312.08514]. TSMS-SAM2 splits memory into short-term and long-term groups and prunes the most redundant feature in each group when the bank is full [2508.05829]. HQ-SMem scores stored embeddings by a relevance–freshness criterion and deletes the frame with highest removal score, aiming to keep memory compact but salient [2507.18921].

A common misconception is that more memory is always better. The surveyed literature repeatedly disputes this. Memory redundancy, stale templates, and prompt ambiguity are treated as major causes of drift, and several methods improve performance precisely by restricting, splitting, pruning, or reweighting memory rather than by expanding it without control [2507.22792] [2508.05829] [2507.18921].

## 5. Benchmarks, metrics, and representative empirical results

The evaluation landscape is heterogeneous because VOST spans segmentation-heavy, tracking-heavy, and domain-specific settings.

| Benchmark family | Examples | Reported metrics |
|---|---|---|
| VOS and UVOS | DAVIS-2016, DAVIS-2017, YouTube-VOS, SegTrackv2 | $J$, $F$, $J\&F$ |
| Tracking and unified VOST | OTB-100, UAV123, NFS, GOT-10k, TrackingNet, LaSOT, VOTS, VOTSt | AUC, AO, SR\_50, SR\_75, Precision, Normalized Precision, $Q$, Acc, Rob, NRE, DRE, ADQ |
| Long-video, egocentric, and surgical | VOST, VISOR, LVD, LVOS, EndoVis2017, EndoVis2018 | $J$, $J_{tr}$, $J\&F$, Dice, CIoU |

Across these benchmarks, the canonical segmentation measures are the Jaccard index and contour F-measure:
$$
J=\frac{|M\cap G|}{|M\cup G|},\qquad
F=\frac{2PR}{P+R}.
$$
Tracking-oriented evaluations add metrics such as AUC, AO, success rates, or the VOTS challenge measures $Q$, Accuracy, Robustness, Not-Reported Error, Drift-Rate Error, and Absence-Detection Quality [2001.05425] [1812.05050] [2507.18921].

Representative reported results illustrate the breadth of the field rather than a single frontier. UnOVOST obtains a mean $J\&F$ score of 67.9% on the DAVIS 2017 Unsupervised validation split and first place in the DAVIS 2019 Unsupervised Video Object Segmentation Challenge [2001.05425]. The self-supervised dense tracking approach reports 70.7 $J\&F$ on DAVIS-2017 and 67.3 overall on YouTube-VOS, surpassing most supervised approaches except STM in the paper’s comparison [2006.12480]. The tracking-by-detection TAN+DTTM system reports 75.9% $J\&F$ at 0.14 second per frame on DAVIS [2007.05687]. MITS reports 80.4 AO on GOT-10k, 83.4 AUC on TrackingNet, and 84.9 $J\!+\!F$ on DAVIS 2017 with mask initialization [2308.13266]. In the egocentric setting, TAM-VT reaches $J_{tr}=37.7$ and $J=49.3$ on VOST validation under identical pre-training and backbone settings [2312.08514]. In surgical promptable VOST, TSMS-SAM2 reports mean Dice scores of 95.24 on EndoVis2017 and 86.73 on EndoVis2018 [2508.05829]. For long-video tracking and segmentation, HQ-SMem reports $Q=0.660$, Acc $=0.793$, and Rob $=0.792$ on VOTS2024 [2507.18921].

These numbers should not be read as directly interchangeable. The datasets differ in supervision, object scale, video length, prompt regime, and domain. A plausible implication is that VOST performance is increasingly benchmark-specific: long-video, egocentric, and surgical settings expose failure modes that are less visible on conventional DAVIS-style evaluation.

## 6. Limitations, misconceptions, and emerging directions

Failure modes recur across otherwise different formulations. Spectral space-time clustering assumes that one dominant salient object aligns with the principal mode, so multi-object scenes can cause the leading eigenvector to focus on only one object or merge nearby objects [2212.08058]. Tracklet-merging and multiple-hypothesis systems remain vulnerable to crowded scenes, similar-looking instances, prolonged occlusions, and fragmented proposals [2001.05425] [1904.08141]. Memory-based propagation can drift under severe spatio-temporal discontinuities, prompt ambiguity, or overfitting of online adaptation [2006.12480] [2507.22792]. Spatial-semantic query methods can expand part-only targets into full-instance masks because semantic priors favor whole objects [2407.07760]. Surgical and long-video studies emphasize rapid motion, disappearance and reappearance, clutter, smoke, reflections, and memory pollution as persistent problems [2508.05829] [2507.18921].

Another misconception is that VOST is primarily a boundary-refinement problem. Boundary quality matters, and several methods use denseCRF, SAM-HQ, High-Quality masks, or contextualized feature pyramids, but many reported gains come from temporal association, memory scheduling, and object identity mechanisms rather than from refinement alone [2212.08058] [2507.18921] [2312.08514].

Recent directions are explicit in the reviewed literature. Foundation-model pipelines center on streaming memory, prompt efficiency, motion-aware memory selection, and trajectory-guided prompting [2507.22792]. Egocentric long-video methods highlight short clips, longer memory, learned time-coding, and multi-scale matching for small objects undergoing transformation [2312.08514]. Domain-specific systems for surgery refine promptable VOST with multi-rate temporal augmentation and split-and-prune memory policies [2508.05829]. Smart-memory methods combine high-quality mask refinement with selective update and feedback to mitigate drift on long sequences [2507.18921].

This suggests that current VOST research is converging on a layered view of the problem: segmentation quality, temporal correspondence, memory selection, and identity representation are being optimized jointly rather than treated as separate stages. The open questions recorded across the papers—memory redundancy, error accumulation, part–whole ambiguity, recovery after absence, and computational efficiency on long videos—indicate that the field has moved from first-frame propagation toward general video reasoning over objects, prompts, and temporal state [2507.22792].

Source: https://www.emergentmind.com/topics/video-object-segmentation-and-tracking-vost