Open-World Video Segmentation

Abstract: While video segmentation has advanced rapidly on short clips and closed-set benchmarks, open-world video segmentation remains largely unexplored. The challenge is twofold: (1) existing methods are not designed to support object discovery and identity maintenance in long videos of dynamic ego-motion, and (2) existing evaluation protocols rely on a rigid 1:1 matching that unfairly penalizes semantically valid predictions with mismatched granularity. To address both gaps, we introduce Savvy, a practical and strong system for zero-shot open-world long-horizon video segmentation. Savvy combines hierarchical mask discovery, deferred admission, and track consolidation to support persistent object discovery, safe track promotion, and stable long-range identity maintenance. We further propose OGA, a granularity-aware evaluation suite for open-world video segmentation. Built on a Granularity-Agnostic (GA) matching protocol, OGA relaxes conventional 1:1 matching to an n:1 mapping, but still enforces temporal rigor by detecting support discontinuities through sever points and scoring each reference object through its dominant coherent fragment. This prevents fragmented or flickering support from being over-rewarded while enabling GA-adapted metrics and structural diagnostics: identity persistence (IP), and identity concentration (IC). On VIPSeg, we show that standard 1:1 evaluation substantially underestimates open-world methods, whereas GA evaluation recovers much of their suppressed performance. On the more realistic long-horizon benchmarks: ScanNet and HM3D, Savvy consistently outperforms strong baselines across both classical and proposed metrics, including STQ, VPQ$_\infty$, IP and IC. Together, these results establish a practical benchmark and a strong baseline for open-world long-horizon video segmentation.

• The paper introduces Savvy, a semi-online system that combines hierarchical mask discovery, deferred identity admission, and track consolidation to maintain objects across long videos, achieving a 2.6× ScanNet VPQ∞ improvement over EntitySAM.
• The paper presents OGA, a reference-anchored n:1 evaluation protocol that supports valid granularity differences while using dominant-fragment selection, sever penalties, and identity diagnostics to preserve temporal and association rigor.
• The paper shows that rigid 1:1 benchmarks can reward annotation-aligned granularity, while Savvy delivers stronger long-horizon performance on ScanNet and HM3D and remains computationally practical at 7.4 FPS with 6–7 GB of GPU memory.

The problem: a circular dependency in open-world video segmentation

This paper addresses open-world video segmentation (OVS), where the object set is not predefined, new entities can appear at any time, previously observed objects may disappear and reappear over long temporal gaps, and semantic boundaries need not follow a fixed category ontology. The authors argue that progress on this setting is blocked by a circular dependency: existing methods are not designed for persistent object discovery and identity maintenance in long videos with dynamic ego-motion, while existing evaluation protocols—built on rigid 1:1 matching between predicted and ground-truth tubes, as in STQ (Weber et al., 2021) and VPQ [9859-9868]—systematically penalize semantically valid predictions whose granularity differs from the annotation. The paper breaks this deadlock on both fronts simultaneously, introducing Savvy as a modeling framework and OGA as an evaluation suite, together with a new long-horizon benchmark.

A motivating empirical observation frames the difficulty: on ScanNet, the accumulated number of distinct seen objects and the rate of reappearance events grow substantially faster than on VIPSeg, and existing baselines (DEVA+SAM [1316-1326], EntitySAM [24234-24243]) track the ground-truth discovery and re-association curves poorly. This establishes that the long-horizon setting is qualitatively different from short-clip panoptic video segmentation, not merely an extension of it.

Savvy: object-set maintenance as the core abstraction

Savvy (Segment Any Video and eVerYthing) treats OVS as the problem of maintaining an explicit object set over time rather than deriving each frame from frame-wise mask-logit arbitration. The system is semi-online, uses SAM2.1 Hiera-Large as the propagation backbone and SAM1 ViT-H for image-level discovery, and is organized around three lifecycle stages.

Hierarchical mask discovery (HMD) proposes candidates only in regions unexplained by the current track set. It samples a prompt grid over unexplained regions, groups SAM multimask outputs into small-, medium-, and large-scale proposal sets, and merges them recursively through a survival-based winner-take-all arbitration with a log-scaled area prior in the quality score. Fragments that retain insufficient support after arbitration are suppressed. This structured discovery contrasts with raw SAM automatic mask generation, and the ablation attributes a substantial share of the system's gains to it.

Deferred admission places every new discovery in a transient buffer of 30 frames before assigning a persistent identity. During this window, candidates accumulate visibility evidence and duplicate-agreement statistics against established tracks; repeated agreement with the same established object triggers merge or discard, while sustained distinct visibility triggers promotion, with backfilling of buffered masks so that temporal support is not lost. This converts object registration from a single-frame decision into an evidence-accumulation process, directly targeting the ID fragmentation that greedy admission produces in long videos.

Track consolidation regularizes the established set through three mechanisms: appearance self-consistency gating, which compares a reappearance descriptor (a multi-level pooled SAM2 feature with per-period EMA committed to a historical bank) against past appearances and suppresses inconsistent revivals; competition arbitration, which suppresses the weaker of two heavily overlapping established tracks using appearance consistency, continuous support length, and seniority as a tie-breaker; and part-whole absorption, which removes redundant part-level identities when a stabilized whole spatially subsumes them (and vice versa), gated by containment (intersection-over-self ≥ 0.7) and area-ratio checks. Lightweight memory pruning, a 300-object active-track budget prioritized by maximum historical area, recent-eviction suppression regions, and adaptive discovery density keep the state bounded on long videos.

The design philosophy is explicit: discovery, admission, and consolidation are first-class lifecycle decisions rather than post-processing heuristics. This is what distinguishes Savvy from DEVA, whose admission is a greedy overlap heuristic, and from EntitySAM, whose discovery of genuinely new objects is limited and whose COCO pretraining aligns its granularity with VIPSeg-style annotations.

OGA: granularity-agnostic evaluation with temporal rigor

OGA's central move is to replace rigid 1:1 matching with a Granularity-Agnostic (GA) protocol: an $n{:}1$, ground-truth-referenced mapping in which each prediction may support at most one reference instance, while multiple predictions may jointly support one reference. Validity is determined by an asymmetric prediction support ratio $S(p,g)=|p\cap g|/|p|$ (or tube IoU), and each prediction is assigned to its strongest supporting reference. The asymmetry is deliberate: relaxing the 1:n direction would credit identity bleeding and over-grouping, so OGA is granularity-agnostic but not many-to-many agnostic.

The key technical device is dominant-fragment selection. Under $n{:}1$ support, a reference instance may be carried by different prediction subsets at different times. OGA builds a support chain of (frame, active support set, support mass) triples, computes a persistent-support ratio $\rho_t(g)$ between adjacent states measuring how much support is carried by the same identities, declares a sever when $\rho_t$ falls below a threshold (default 0.5), and scores each reference only through its dominant coherent fragment—the fragment with the largest accumulated support mass. The authors formalize the ideal alternative as a maximum-weight temporally coherent subgraph selection problem, argue it is combinatorially intractable as an evaluation primitive (related to maximum-weight connected subgraph and prize-collecting Steiner problems), and present dominant-fragment selection as a deterministic, conservative surrogate. Conservatism is enforced structurally: the union term in the virtual IoU uses the full reference area, not just the fragment span, so a short pure fragment cannot score highly unless it explains most of the reference; a soft stability term penalizes identity churn within the fragment; and an area-efficiency penalty discourages support that uses excessive prediction area.

GA-adapted STQ and VPQ preserve the outer metric structure. GA-VPQ additionally decomposes false positives into ordinary FPs and fragment-level FPs, where each excess sever-induced fragment of a matched reference contributes one FP unit, preserving the penalty on identity fragmentation. Because $n{:}1$ support can hide how a score is achieved, OGA adds two structural diagnostics: identity persistence (IP), a mass-based measure of whether overlap mass is dominated by one identity on each axis, and identity concentration (IC), a graph-based measure of how many distinct identities ever participate in valid support. The paper is careful to note that prediction-side scores (IP$_P$, IC$_P$) can be misleading in isolation—many locally pure short-lived fragments yield high prediction-axis scores while shattering reference objects—and must be read jointly with the reference-side diagnostics.

Results: granularity alignment inflates standard scores; Savvy leads on long horizons

VIPSeg as a controlled testbed. Under standard 1:1 evaluation, EntitySAM appears dominant (VPQ$_\infty$ 54.68, STQ 43.67), while Savvy scores 26.15 and 46.29. Under GA evaluation, the picture changes sharply: Savvy rises to 55.35 VPQ$_\infty$ (+29.18) and 72.09 STQ (+25.80), SAM2 gains +26.84 VPQ$S(p,g)=|p\cap g|/|p|$0, and DEVA+SAM gains +24.53, while EntitySAM gains only +4.55. This is the paper's strongest claim about evaluation: benchmark-aligned granularity confers a dual advantage under 1:1 matching—direct support plus fewer, coarser masks with better spatiotemporal coherence—and rigid matching conflates this alignment advantage with genuine segmentation quality. Notably, class-agnostic evaluation alone does not fix this; the bias persists even with semantic labels removed. The full VIPSeg diagnostics also temper the conclusion: EntitySAM retains the highest VPQ$S(p,g)=|p\cap g|/|p|$1 and IC$S(p,g)=|p\cap g|/|p|$2 under GA evaluation, so OGA reduces but does not erase annotation-alignment bias.

Long-horizon benchmarks. On ScanNet and HM3D, Savvy achieves the best VPQ$S(p,g)=|p\cap g|/|p|$3, VPQ$S(p,g)=|p\cap g|/|p|$4, STQ, and AQ on both datasets, exceeding DEVA+SAM by over 50% on ScanNet VPQ$S(p,g)=|p\cap g|/|p|$5 (28.63 vs. 18.53) and more than doubling EntitySAM's score (10.82). On HM3D, Savvy reaches 19.59 VPQ$S(p,g)=|p\cap g|/|p|$6 versus 8.47 (DEVA+SAM) and 1.46 (EntitySAM). The structural diagnostics show Savvy strongest on both persistence axes (ScanNet IP$S(p,g)=|p\cap g|/|p|$7/IP$S(p,g)=|p\cap g|/|p|$8 75.44/57.92; HM3D 53.15/37.07) and on reference-side concentration, with the largest margins in the temporal-pattern diagnostics (ScanNet Cluster 65.71, Patt$S(p,g)=|p\cap g|/|p|$9 58.67). EntitySAM's collapse—despite its VIPSeg strength—supports the paper's argument that benchmark alignment does not transfer to long-horizon open-world behavior. DEVA+SAM's consistently highest GQ (95.87/94.81) alongside weaker association scores demonstrates that strong instantaneous foreground coverage is insufficient for OVS. Size-stratified results show Savvy's advantage holds across small, medium, and large objects, with the caveat that EntitySAM's high IC$n{:}1$0 on large HM3D objects reflects coarse blobbing rather than better maintenance.

Runtime. The full pipeline sustains a mean cumulative throughput of 7.4 FPS on an A6000 with GPU memory bounded at roughly 6–7 GB over the ScanNet validation split, indicating that explicit object-set maintenance is tractable, though not real-time-optimized.

Ablations, stress tests, and robustness

Module-level ablations attribute the design's value to each component: replacing HMD with raw AMG drops VPQ$n{:}1$1 from 31.31 to 25.42 and IC$n{:}1$2 from 31.51 to 24.89; replacing transient buffering with immediate admission drops VPQ$n{:}1$3 to 28.07; disabling track consolidation drops VPQ$n{:}1$4 to 30.13. The raw-AMG result is instructive: raw AMG yields slightly higher IC$n{:}1$5, illustrating prediction-side purity without GT-side compactness.

Synthetic stress tests with controlled perturbations (clutter, dropout, flicker, sever, void-only dilation) show that OGA responds selectively rather than collapsing all failures into one overlap signal: dropout sharply degrades VPQ$n{:}1$6 and STQ; flicker and sever leave VPQ$n{:}1$7 and GQ unchanged while collapsing long-window and GT-side identity metrics (VPQ$n{:}1$8 falls to 0.05 under 10% flicker); void-only expansion is nearly harmless under the void-tolerant policy, while GQ paradoxically increases under clutter—confirming that GQ alone is insufficient. Parameter sweeps over discovery stride, transient window, competition margin, and appearance threshold show stable behavior, with degradation concentrated in degenerate settings (disabled appearance gating, zero competition margin). OGA threshold sensitivity is modest: the support threshold mainly affects diagnostics, while the sever threshold trades long-window scores against temporal strictness; EntitySAM's near-invariance to the sever threshold is correctly read as a structural signature of its coarse, low-discovery output rather than robustness.

Limitations and open questions

The paper is explicit about its constraints. Savvy is class-agnostic rather than open-vocabulary. Granularity remains unstable under changing scale and viewpoint; long-gap re-association fails under large viewpoint change or among repeated similar objects (e.g., identical chairs), where local appearance cues are insufficient; spatially adjacent similar objects can still blob into a shared identity; and highly cluttered scenes expose an unresolved need for adaptive granularity selection rather than better propagation. On the evaluation side, OGA anchors on the reference annotation and relaxes only the $n{:}1$9 direction—a deliberate choice, but one that means 1:n over-grouping is not accommodated—and evaluation remains sensitive to annotation completeness and reference granularity, as the semantic-versus-instance HM3D comparison confirms (EntitySAM's VPQ$\rho_t(g)$0 falls from 14.65 to 2.90 when reference granularity is refined, while all methods' IC$\rho_t(g)$1 paradoxically rises). The benchmark suite omits outdoor driving regimes, which the authors identify as complementary. The dominant-fragment rule is a surrogate for an intractable coherent-core search; whether better approximations change method rankings is left unexamined.

Conclusion

This paper makes a coordinated contribution on three fronts: a practical semi-online system (Savvy) that treats discovery, deferred admission, and consolidation as explicit lifecycle decisions; an evaluation suite (OGA) that relaxes granularity in a controlled, reference-anchored $\rho_t(g)$2 direction while preserving temporal rigor through sever points and dominant-fragment scoring; and a new HM3D-106 long-horizon benchmark with repeated re-observation. Empirically, it demonstrates that standard 1:1 evaluation substantially understates methods without benchmark-aligned granularity, that class-agnostic evaluation alone does not repair this, and that explicit object-set maintenance yields decisive long-horizon gains—most prominently a 2.6× improvement over EntitySAM on ScanNet VPQ$\rho_t(g)$3—while remaining computationally tractable. The remaining open problems are well-posed: stable granularity selection under scale change, long-gap re-identification beyond local appearance, and relational reasoning over repeated similar objects.