Video Object Segmentation and Tracking

• Video Object Segmentation and Tracking is a field that combines pixel-accurate segmentation with temporal association to maintain object identities over video frames.
• It encompasses a range of methods including semi-supervised, unsupervised, box-initialized, and promptable approaches to address diverse application needs.
• Recent developments leverage memory-based propagation, transformer models, and prompt-driven strategies to enhance robustness and temporal consistency.

Video Object Segmentation and Tracking (VOST) addresses the intertwined problems of delineating object masks at the pixel level over time and maintaining their identities and locations as the video evolves. In the literature summarized here, VOST appears in semi-supervised, unsupervised, online, offline, multi-object, egocentric, and promptable forms: it may start from first-frame mask(s), a first-frame bounding box, a prompt such as points or boxes, or no target specification at all, yet it always couples pixel-accurate segmentation with temporal association and identity preservation (Yao et al., 2019, Xu et al., 30 Jul 2025).

1. Scope, task settings, and conceptual boundaries

The field spans several task regimes. Semi-supervised formulations provide first-frame mask(s) and ask for mask propagation over the remaining frames; self-supervised work explicitly treats semi-supervised VOS as dense tracking; box-initialized methods seek class-agnostic segmentation from only a first-frame box; promptable formulations condition segmentation and tracking on user-provided points, boxes, or masks; and unsupervised approaches discover salient objects and track them without any target-specific input (Zhu et al., 2020, Wang et al., 2018, Xu et al., 7 Aug 2025, Luiten et al., 2020).

Older survey literature organizes the space into unsupervised VOS, semi-supervised VOS, interactive VOS, weakly supervised VOS, and segmentation-based tracking (Yao et al., 2019). More recent review work recasts the same territory through “past–present–future”: how memory from previous frames is retained and updated, how current-frame features are extracted and refined, and how motion is predicted to anticipate upcoming dynamics (Xu et al., 30 Jul 2025). This suggests that VOST is not a single algorithmic template but a family of formulations distinguished by supervision, memory, temporal causality, and the representation used for identity.

A recurrent source of confusion is the relationship between VOS, VOT, MOTS, and VOST. The reviewed work does not collapse them into a single benchmark tradition. Instead, VOST sits at the interface: it inherits pixel-level evaluation from VOS, explicit temporal association from tracking, and, in multi-object settings, identity reasoning that resembles MOTS (Yao et al., 2019, Xu et al., 30 Jul 2025). Another common misconception is that VOST is necessarily mask propagation from a first-frame mask. The literature includes box-conditioned tracking and segmentation, tracklet- and proposal-based association, spectral clustering over pixels, and prompt-driven SAM/SAM2 pipelines (Sharir et al., 2017, Voigtlaender et al., 2019, Burceanu et al., 2022, Xu et al., 30 Jul 2025).

2. Mathematical formulations of segmentation and temporal association

Several papers formulate VOST as an optimization problem over space and time. One line treats segmentation as variational partitioning. In “Object Segmentation Tracking from Generic Video Cues,” frame-to-frame propagation is driven by optical-flow consistency, color, and image or motion boundaries, and the next-frame partition is obtained by minimizing a boundary-weighted energy of the form

$$\min_{\Omega_1,\dots,\Omega_n \subset \Omega} \frac{\lambda}{2}\sum_{i=1}^{n}\mathrm{Per}(\Omega_i;\Omega) \;+\; \sum_{i=1}^{n}\int_{\Omega_i} h_i(x)\,dx,$$

subject to a disjoint partition of the image domain (Kardoost et al., 2019). A related earlier formulation combines detector-derived object tubes, motion cues, and a spatio-temporal multi-label MRF over superpixels, again coupling unary location/appearance terms with spatial and temporal smoothness (Drayer et al., 2016).

A second line formulates VOST as spectral grouping. “Learning a Fast 3D Spectral Approach to Object Segmentation and Tracking over Space and Time” models the video as a dense space-time graph with one node per pixel and local space-time neighborhoods, then seeks the principal eigenvector of the adjacency matrix:

$$\mathbf{x}_s \;=\; \underset{\mathbf{x}}{\arg\max}\;\frac{\mathbf{x}^\top \mathbf{M}\,\mathbf{x}}{\mathbf{x}^\top \mathbf{x}}, \qquad \mathbf{M}\mathbf{v}=\lambda_{\max}\mathbf{v}.$$

The paper argues that the leading mode corresponds to the strongest spatiotemporal cluster and approximates the matrix–vector product with separable 3D filtering, avoiding explicit construction of the adjacency matrix (Burceanu et al., 2022).

A third line centers on learned correspondence and memory attention. In self-supervised dense tracking, a query frame is reconstructed from reference frames by local affinity weights,

$$A_{t}^{ij}=\frac{\exp\left\langle f_{t}^{i}, f_{t-1}^{j}\right\rangle}{\sum_{n\in \mathcal{N}} \exp\left\langle f_{t}^{i}, f_{t-1}^{n}\right\rangle},$$

and the same affinities propagate masks through time (Zhu et al., 2020). Transformer-based long-video methods generalize this to multi-scale memory matching. TAM-VT computes dense matching between query and memory tokens at multiple scales and modulates the attention logits with learned multiplicative relative-time encoding, while SAM2-style review literature expresses the same design pattern as memory read–write over key–value banks (Goyal et al., 2023, Xu et al., 30 Jul 2025).

These formulations differ in machinery, but they converge on the same technical objective: encode objectness, appearance, motion, and temporal continuity in a way that stabilizes per-pixel identity across frames. This suggests that the conceptual boundary between “segmentation” and “tracking” is increasingly encoded in the choice of temporal operator rather than in the output format alone.

3. Major architectural paradigms

One durable paradigm is tracking-by-detection. “Video Object Segmentation using Tracked Object Proposals” combines a category-independent appearance segmentation network with Faster R-CNN proposals and a temporal IoU continuity constraint on selected boxes (Sharir et al., 2017). “Proposal, Tracking and Segmentation (PTS)” decomposes the pipeline into an Object Proposal Network, an Object Tracking Network, and a Dynamic Reference Segmentation Network, using RPN-based proposals to reduce the search space, a tracker to identify the target among proposals, and dynamic references to adapt segmentation without heavy online fine-tuning (Zhou et al., 2019). “Fast Video Object Segmentation With Temporal Aggregation Network and Dynamic Template Matching” reframes semi-supervised VOS as tracking-by-detection with an online detector, short-clip temporal aggregation, Hungarian matching between detections and template banks, and an FCN mask head (Huang et al., 2020).

A closely related box-first family pursues segmentation from tracked localization. SiamMask augments a fully convolutional Siamese tracker with a segmentation branch so that a single forward pass yields objectness scores, bounding boxes, and a per-frame binary mask from only a first-frame bounding box (Wang et al., 2018). BoLTVOS explicitly splits the problem into bounding-box-level tracking and box-to-mask segmentation, using a conditioned detector, temporal consistency rescoring over tracklets, and a Box2Seg network (Voigtlaender et al., 2019). MITS extends this unification to the multi-object setting with a unified identification module that accepts both mask and box references and a pinpoint box predictor that learns object-centric box localization while jointly decoding masks and boxes (Xu et al., 2023).

A second paradigm is propagation with explicit memory. “Dual Temporal Memory Network for Efficient Video Object Segmentation” couples a short-term graph-based memory over local spatial-temporal neighborhoods with a long-term Simplified-GRU over object-masked pooled features (Zhang et al., 2020). Self-supervised VOS introduces a momentum memory for long-range correspondence and a sequence-specific online appearance model trained on pseudo-labels (Zhu et al., 2020). “Learning Spatial-Semantic Features for Robust Video Object Segmentation” combines pixel memory with discriminative object queries and masked query updates to focus propagation on representative object parts (Li et al., 2024). TAM-VT pushes this design toward clip-based online inference with multi-scale matching and decoding, specifically targeting long videos, small objects, and deformation-heavy egocentric sequences (Goyal et al., 2023).

A third paradigm is explicit global association. UnOVOST builds short tracklets by optical-flow-warped IoU, merges them into long-term tracks with a tracklet-based Forest Path Cutting algorithm, and selects final tracks by video saliency (Luiten et al., 2020). MHP-VOS defers per-frame decisions by maintaining multiple box hypotheses, scoring them with motion and mask-propagation consistency, and selecting globally consistent trajectories with a Maximum Weighted Independent Set formulation plus N-scan pruning (Xu et al., 2019).

A fourth paradigm is promptable foundation-model VOST. The review literature around SAM and SAM2 frames video segmentation and tracking in terms of prompt encoding, memory attention, streaming memory, motion-aware memory selection, and trajectory-guided prompting (Xu et al., 30 Jul 2025). Domain-specialized derivatives include TSMS-SAM2 for surgical video, which adds multi-temporal-scale sampling and memory splitting with pruning (Xu et al., 7 Aug 2025), and HQ-SMem, which combines SAM-HQ refinement, DINO-based candidate validation, smart memory, and teacher-forced feedback into DeAOT-style trackers (Kazemi et al., 25 Jul 2025).

4. Temporal consistency, memory management, and identity preservation

Temporal consistency is a central design target rather than a by-product. SFSeg introduces TCONT, a temporal coherence measure based on direct and reverse optical-flow alignment of soft masks, and reports that SFSeg++ improves temporal consistency IoU with respect to ground truth by +4.5% over a strong ensemble baseline (Burceanu et al., 2022). In tracking-by-detection systems, temporal consistency is enforced through explicit association: PTS uses success-gated short- and long-term tracker updates (Zhou et al., 2019), Fast VOS employs Hungarian matching with combined location IoU and cosine appearance similarity plus dynamic template bank updates (Huang et al., 2020), and BoLTVOS rescales track hypotheses by detector score, aspect-ratio consistency, IoU continuity, and center displacement (Voigtlaender et al., 2019).

Identity preservation in multi-object settings often requires stronger combinatorial structure. UnOVOST imposes a unique-predecessor forest over tracklets and cuts root-to-leaf paths greedily according to visual consistency and temporal density (Luiten et al., 2020). MHP-VOS maintains multiple hypotheses over time and selects a globally consistent subset through MWIS, thereby postponing ambiguous local decisions until more evidence accumulates (Xu et al., 2019). MITS avoids separate association by propagating a unified identification embedding for all targets, allowing all objects to be processed simultaneously from encoding through decoding (Xu et al., 2023).

Recent work increasingly treats memory itself as the locus of temporal robustness. TAM-VT keeps the initial reference frame fixed in memory, appends only the last frame of each processed clip, and uses multiplicative relative-time encoding to preserve recency while retaining non-zero importance for the first reference frame (Goyal et al., 2023). TSMS-SAM2 splits memory into short-term and long-term groups and prunes the most redundant feature in each group when the bank is full (Xu et al., 7 Aug 2025). HQ-SMem scores stored embeddings by a relevance–freshness criterion and deletes the frame with highest removal score, aiming to keep memory compact but salient (Kazemi et al., 25 Jul 2025).

A common misconception is that more memory is always better. The surveyed literature repeatedly disputes this. Memory redundancy, stale templates, and prompt ambiguity are treated as major causes of drift, and several methods improve performance precisely by restricting, splitting, pruning, or reweighting memory rather than by expanding it without control (Xu et al., 30 Jul 2025, Xu et al., 7 Aug 2025, Kazemi et al., 25 Jul 2025).

5. Benchmarks, metrics, and representative empirical results

The evaluation landscape is heterogeneous because VOST spans segmentation-heavy, tracking-heavy, and domain-specific settings.

Benchmark family	Examples	Reported metrics
VOS and UVOS	DAVIS-2016, DAVIS-2017, YouTube-VOS, SegTrackv2	$J$, $F$, $J$
Tracking and unified VOST	OTB-100, UAV123, NFS, GOT-10k, TrackingNet, LaSOT, VOTS, VOTSt	AUC, AO, SR_50, SR_75, Precision, Normalized Precision, $Q$, Acc, Rob, NRE, DRE, ADQ
Long-video, egocentric, and surgical	VOST, VISOR, LVD, LVOS, EndoVis2017, EndoVis2018	$J$, $J_{tr}$, $J$, Dice, CIoU

Across these benchmarks, the canonical segmentation measures are the Jaccard index and contour F-measure:

$$\mathbf{x}_s \;=\; \underset{\mathbf{x}}{\arg\max}\;\frac{\mathbf{x}^\top \mathbf{M}\,\mathbf{x}}{\mathbf{x}^\top \mathbf{x}}, \qquad \mathbf{M}\mathbf{v}=\lambda_{\max}\mathbf{v}.$$0

Tracking-oriented evaluations add metrics such as AUC, AO, success rates, or the VOTS challenge measures $$\mathbf{x}_s \;=\; \underset{\mathbf{x}}{\arg\max}\;\frac{\mathbf{x}^\top \mathbf{M}\,\mathbf{x}}{\mathbf{x}^\top \mathbf{x}}, \qquad \mathbf{M}\mathbf{v}=\lambda_{\max}\mathbf{v}.$$1, Accuracy, Robustness, Not-Reported Error, Drift-Rate Error, and Absence-Detection Quality (Luiten et al., 2020, Wang et al., 2018, Kazemi et al., 25 Jul 2025).

Representative reported results illustrate the breadth of the field rather than a single frontier. UnOVOST obtains a mean $$\mathbf{x}_s \;=\; \underset{\mathbf{x}}{\arg\max}\;\frac{\mathbf{x}^\top \mathbf{M}\,\mathbf{x}}{\mathbf{x}^\top \mathbf{x}}, \qquad \mathbf{M}\mathbf{v}=\lambda_{\max}\mathbf{v}.$$2 score of 67.9% on the DAVIS 2017 Unsupervised validation split and first place in the DAVIS 2019 Unsupervised Video Object Segmentation Challenge (Luiten et al., 2020). The self-supervised dense tracking approach reports 70.7 $$\mathbf{x}_s \;=\; \underset{\mathbf{x}}{\arg\max}\;\frac{\mathbf{x}^\top \mathbf{M}\,\mathbf{x}}{\mathbf{x}^\top \mathbf{x}}, \qquad \mathbf{M}\mathbf{v}=\lambda_{\max}\mathbf{v}.$$3 on DAVIS-2017 and 67.3 overall on YouTube-VOS, surpassing most supervised approaches except STM in the paper’s comparison (Zhu et al., 2020). The tracking-by-detection TAN+DTTM system reports 75.9% $$\mathbf{x}_s \;=\; \underset{\mathbf{x}}{\arg\max}\;\frac{\mathbf{x}^\top \mathbf{M}\,\mathbf{x}}{\mathbf{x}^\top \mathbf{x}}, \qquad \mathbf{M}\mathbf{v}=\lambda_{\max}\mathbf{v}.$$4 at 0.14 second per frame on DAVIS (Huang et al., 2020). MITS reports 80.4 AO on GOT-10k, 83.4 AUC on TrackingNet, and 84.9 $$\mathbf{x}_s \;=\; \underset{\mathbf{x}}{\arg\max}\;\frac{\mathbf{x}^\top \mathbf{M}\,\mathbf{x}}{\mathbf{x}^\top \mathbf{x}}, \qquad \mathbf{M}\mathbf{v}=\lambda_{\max}\mathbf{v}.$$5 on DAVIS 2017 with mask initialization (Xu et al., 2023). In the egocentric setting, TAM-VT reaches $$\mathbf{x}_s \;=\; \underset{\mathbf{x}}{\arg\max}\;\frac{\mathbf{x}^\top \mathbf{M}\,\mathbf{x}}{\mathbf{x}^\top \mathbf{x}}, \qquad \mathbf{M}\mathbf{v}=\lambda_{\max}\mathbf{v}.$$6 and $$\mathbf{x}_s \;=\; \underset{\mathbf{x}}{\arg\max}\;\frac{\mathbf{x}^\top \mathbf{M}\,\mathbf{x}}{\mathbf{x}^\top \mathbf{x}}, \qquad \mathbf{M}\mathbf{v}=\lambda_{\max}\mathbf{v}.$$7 on VOST validation under identical pre-training and backbone settings (Goyal et al., 2023). In surgical promptable VOST, TSMS-SAM2 reports mean Dice scores of 95.24 on EndoVis2017 and 86.73 on EndoVis2018 (Xu et al., 7 Aug 2025). For long-video tracking and segmentation, HQ-SMem reports $$\mathbf{x}_s \;=\; \underset{\mathbf{x}}{\arg\max}\;\frac{\mathbf{x}^\top \mathbf{M}\,\mathbf{x}}{\mathbf{x}^\top \mathbf{x}}, \qquad \mathbf{M}\mathbf{v}=\lambda_{\max}\mathbf{v}.$$8, Acc $$\mathbf{x}_s \;=\; \underset{\mathbf{x}}{\arg\max}\;\frac{\mathbf{x}^\top \mathbf{M}\,\mathbf{x}}{\mathbf{x}^\top \mathbf{x}}, \qquad \mathbf{M}\mathbf{v}=\lambda_{\max}\mathbf{v}.$$9, and Rob $$A_{t}^{ij}=\frac{\exp\left\langle f_{t}^{i}, f_{t-1}^{j}\right\rangle}{\sum_{n\in \mathcal{N}} \exp\left\langle f_{t}^{i}, f_{t-1}^{n}\right\rangle},$$0 on VOTS2024 (Kazemi et al., 25 Jul 2025).

These numbers should not be read as directly interchangeable. The datasets differ in supervision, object scale, video length, prompt regime, and domain. A plausible implication is that VOST performance is increasingly benchmark-specific: long-video, egocentric, and surgical settings expose failure modes that are less visible on conventional DAVIS-style evaluation.

6. Limitations, misconceptions, and emerging directions

Failure modes recur across otherwise different formulations. Spectral space-time clustering assumes that one dominant salient object aligns with the principal mode, so multi-object scenes can cause the leading eigenvector to focus on only one object or merge nearby objects (Burceanu et al., 2022). Tracklet-merging and multiple-hypothesis systems remain vulnerable to crowded scenes, similar-looking instances, prolonged occlusions, and fragmented proposals (Luiten et al., 2020, Xu et al., 2019). Memory-based propagation can drift under severe spatio-temporal discontinuities, prompt ambiguity, or overfitting of online adaptation (Zhu et al., 2020, Xu et al., 30 Jul 2025). Spatial-semantic query methods can expand part-only targets into full-instance masks because semantic priors favor whole objects (Li et al., 2024). Surgical and long-video studies emphasize rapid motion, disappearance and reappearance, clutter, smoke, reflections, and memory pollution as persistent problems (Xu et al., 7 Aug 2025, Kazemi et al., 25 Jul 2025).

Another misconception is that VOST is primarily a boundary-refinement problem. Boundary quality matters, and several methods use denseCRF, SAM-HQ, High-Quality masks, or contextualized feature pyramids, but many reported gains come from temporal association, memory scheduling, and object identity mechanisms rather than from refinement alone (Burceanu et al., 2022, Kazemi et al., 25 Jul 2025, Goyal et al., 2023).

Recent directions are explicit in the reviewed literature. Foundation-model pipelines center on streaming memory, prompt efficiency, motion-aware memory selection, and trajectory-guided prompting (Xu et al., 30 Jul 2025). Egocentric long-video methods highlight short clips, longer memory, learned time-coding, and multi-scale matching for small objects undergoing transformation (Goyal et al., 2023). Domain-specific systems for surgery refine promptable VOST with multi-rate temporal augmentation and split-and-prune memory policies (Xu et al., 7 Aug 2025). Smart-memory methods combine high-quality mask refinement with selective update and feedback to mitigate drift on long sequences (Kazemi et al., 25 Jul 2025).

This suggests that current VOST research is converging on a layered view of the problem: segmentation quality, temporal correspondence, memory selection, and identity representation are being optimized jointly rather than treated as separate stages. The open questions recorded across the papers—memory redundancy, error accumulation, part–whole ambiguity, recovery after absence, and computational efficiency on long videos—indicate that the field has moved from first-frame propagation toward general video reasoning over objects, prompts, and temporal state (Xu et al., 30 Jul 2025).