Referring Video Segmentation (RVOS)

• RVOS is the process of segmenting video frames based on natural language expressions to generate binary masks for a referred object.
• It employs cross-modal fusion methods, using techniques like early element-wise multiplication and cross-attention, to align linguistic details with evolving visual features.
• Efficient temporal modeling in RVOS leverages transformer-based architectures, prompting strategies, and compression techniques to ensure spatio-temporal consistency and scalability.

Referring Video Segmentation (RVOS) is the task of temporally segmenting the object(s) in a video sequence described by a natural language expression, producing binary masks per frame for the referred entity. Unlike conventional video object segmentation (VOS), which is typically instance- or class-driven, RVOS is language-guided and requires aligning complex linguistic cues with visual content under dynamic temporal and spatial changes. Accurate RVOS models must jointly handle cross-modal reasoning, spatio-temporal consistency, and fine-grained mask prediction for real-world scenarios where, for instance, multiple object instances, complex motions, and ambiguous queries often coexist (Bellver et al., 2020).

1. Problem Definition and Key Challenges

Formally, RVOS takes as input a video sequence $V = \{I_1, I_2, \dots, I_T\}$ and a referring expression $S$. The goal is to output a sequence of binary masks $M = \{M_1, \dots, M_T\}$, where each $M_t$ localizes the referred object at time $t$. The technically critical aspects are:

• Cross-Modal Semantics: High-quality segmentation relies on deep alignment between linguistic cues (which may describe attributes, relationships, or actions) and visual features that vary over time. Expressions may be underspecified or ambiguous when multiple object instances of the same class are present.
• Temporal Consistency: Maintaining spatio-temporal coherence in mask predictions is challenging, especially under rapid motion, occlusion, or shot changes (Miao et al., 28 Mar 2024, Liang et al., 19 May 2025).
• Benchmark Limitations: Datasets such as DAVIS-2017 and A2D contain a high proportion of “trivial” referring expressions where identification is straightforward. Consequently, strong benchmark results may overstate general cross-modal and temporal reasoning capabilities (Bellver et al., 2020, Liang et al., 19 May 2025).
• Efficiency and Scalability: Frame-by-frame processing (common in early architectures) ignores context and incurs redundant computation. Scaling input frames and mask tokens burdens memory and compute (Zhang et al., 28 Sep 2025, Yan et al., 9 Oct 2025).

2. Model Architectures and Cross-Modal Fusion Strategies

Major RVOS models share a multi-stage modular approach:

Visual Backbone	Language Encoder	Fusion	Temporal Modeling	Segmentation Head
CNN (DeepLabv3), ViT	BERT, RoBERTa	Multiplicative, Cross-attention	Frame-indep. (Bellver et al., 2020), Transformer (Luo et al., 1 Jan 2024)	Conv, Transformer-based
VLP Models (CLIP, VLMo)	CLIP, VLMo	Joint token space, Cross-modal attention	Prompt-tuning, Temporal tokens (Zhou et al., 17 May 2024)	Mask2Former, dynamic kernel
Foundation Models (SAM2)	CLIP, RoBERTa	Aligned object tokens, Lightweight selector	Sequence modeling (Kim et al., 2 Dec 2024)	Foundation decoder (SAM2)

• Early Fusion: Element-wise multiplication of sentence embedding (from BERT or RoBERTa) with deep visual features, as in RefVOS (Bellver et al., 2020), is effective for integrating global semantics.
• Attention-Based Fusion: More recent models (SOC (Luo et al., 2023), FTEA (Li et al., 2023), and ReferDINO (Liang et al., 24 Jan 2025)) rely on multi-head cross-attention between image and language tokens, enabling flexible, fine-grained correspondence.
• Vision-Language Pretrained (VLP) Models: Approaches such as VLP-RVOS (Zhou et al., 17 May 2024), and solutions using frozen CLIP backbones (Pan et al., 7 Jun 2024), directly exploit multimodal alignment from massive pretraining, often combined with parameter-efficient adaptation (e.g., prompt-tuning or temporal tokens). This promotes strong generalization but requires bridging gaps between training (e.g., static region-level) and RVOS deployment (dynamic, pixel-level).

3. Temporal Modeling and Aggregation

Capturing object evolution and context over extended sequences is a central RVOS challenge.

• Frame-Independent and Early Models: Initial approaches (e.g., RefVOS (Bellver et al., 2020)) treat each frame independently; they excel when identification is driven by static appearance but fail under dense actor interactions or motion-based queries.
• Explicit Temporal Modules: Recent models employ transformers for temporal aggregation (SOC (Luo et al., 2023): clusters object queries across time; BIFIT (Lan et al., 2023): inter-frame attention in decoder; ReferMo (Liang et al., 19 May 2025): combines motion, local, and global attention). Hybrid memory (Miao et al., 28 Mar 2024) propagates features across reference and target frames for temporally stable prediction.
• Long-Sequence Optimization: To manage long videos, LTCA (Yan et al., 9 Oct 2025) introduces dilated window and random attention, aggregating local and random global contexts with linear complexity. SVAC (Zhang et al., 28 Sep 2025) compresses tokens via anchor-based spatio-temporal strategies to handle thousands of frames efficiently.
• Temporal Prompting: Recent modular approaches decouple temporal reasoning (object track proposal or prompting) from pixel-level segmentation, using tracking or visual-language candidate selection (Tenet (Lin et al., 8 Oct 2025), GroPrompt (Lin et al., 18 Jun 2024)).

4. Segmentation Heads and Training Paradigms

• Mask Decoding: State-of-the-art RVOS models adopt dynamic kernel-based mask heads (as in Mask2Former or FTEA) or hybrid heads (integrating dot-product and dynamic convolution, as in HCD (Zhang et al., 19 Aug 2025)). Deformable attention mechanisms are often applied for efficient spatial aggregation at object centers (Liang et al., 24 Jan 2025).
• Prompt-Based Segmentation: Integration with foundation models (SAM2) allows segmentation from bounding box or point prompts. GroPrompt (Lin et al., 18 Jun 2024) and Tenet (Lin et al., 8 Oct 2025) demonstrate that competitive RVOS is possible with only bounding box supervision and masked prompt selection, drastically reducing annotation and training requirements.
• Weak and Semi-supervised Learning: SimRVOS (Zhao et al., 2023) and GroPrompt (Lin et al., 18 Jun 2024) show that cross-frame dynamic filtering and text-aware (contrastive) alignment yield near-parity with fully-supervised approaches at much lower annotation cost.
• Dataset-Driven Baselines: On new long-form video benchmarks (Long-RVOS (Liang et al., 19 May 2025)), motion-aware architectures such as ReferMo that fuse static, motion, and language features in local-to-global manner achieve the most robust long-term tracking and segmentation.

5. Evaluation Datasets, Metrics, and Benchmark Limitations

The last five years have witnessed the construction of challenging RVOS datasets:

Dataset	Avg. Length	Challenge	Expressions	Core Metrics
DAVIS-2017	~2–5s	Short, single/multi-obj	Simple, mostly trivial	𝒥 (IoU), ℱ (contour), 𝒥&ℱ
Ref-YouTube-VOS	~10s	More diverse	Varying, mid-long	𝒥, ℱ, 𝒥&ℱ
MeViS	up to ~30s	Motion-expr., higher ambiguity	Verb-focused	𝒥, 𝒥-𝒻
Long-RVOS	>60s	Occlusion, absences, shot changes	Static/dynamic/hybrid	𝒥, ℱ, tIoU, vIoU

• Spatial metrics (IoU/𝒥, contour/ℱ) are standard, but inadequate for temporal robustness and rare events.
• Temporal and Spatiotemporal metrics such as Temporal IoU (tIoU), spatiotemporal volume IoU (vIoU) (Liang et al., 19 May 2025), and Mask Consistency Score (MCS) (Miao et al., 28 Mar 2024) are increasingly used to quantify temporal stability, correct silent periods, and global matching.
• This suggests that evaluation protocols must be carefully matched to target use cases; high spatial metrics alone may not indicate robust tracking or correct handling of occlusion or disappearance.

6. Trends, Innovations, and Future Directions

• Scaling and Compression: Recent advances (e.g., SVAC (Zhang et al., 28 Sep 2025), LTCA (Yan et al., 9 Oct 2025)) aggressively scale both the temporal extent (frames, tokens) and the semantic resolution (object queries), while introducing novel compression (ASTC) and attention (dilated/random, global queries) mechanisms for tractable computation.
• Prompting and Modular Decoupling: The paradigm is shifting from monolithic, end-to-end models toward modular constructions—using foundation segmentation models with plug-and-play temporal prompting and language-guided candidate selection (GroPrompt (Lin et al., 18 Jun 2024), Tenet (Lin et al., 8 Oct 2025), SOLA (Kim et al., 2 Dec 2024)). This facilitates adaptation to diverse domains and lowers data barriers.
• Hierarchical/LLM-based Reasoning: Emerging frameworks now parse queries into structured commands (motion, spatial, attribute) with hierarchical coarse-to-fine reasoning (e.g., PARSE-VOS (Zhao et al., 6 Sep 2025)), sometimes powered by LLMs. A plausible implication is that future RVOS will interleave multi-agent or multi-stage LLM reasoning with visual processing, supporting fine-grained compositional instructions.
• Robustness and Negative Pairs: Models and benchmarks are being designed to handle negative text-video pairs, where the described object might not exist in the video (Robust R-VOS (Li et al., 2022)), reflecting real-world retrieval-like deployments.
• Weak and Efficient Supervision: Designs such as SimRVOS, GroPrompt, and Tenet, underpinned by prompt-based adaptation of large pre-trained models, demonstrate that annotation-efficient and even training-free schemes can achieve competitive results, especially as foundation models continue to improve.

7. Open Problems and Outlook

Major open problems highlighted by the literature include:

• Generalization to complex motion and occlusion: Despite improvements, long-term tracking and segmentation under occlusion, intermittent visibility, and re-appearance remain extremely challenging (Liang et al., 19 May 2025).
• Fine-grained language-vision alignment: Understanding and executing under composite expressions, ambiguous referents, or event-based queries still tests the limits of cross-modal reasoning (Bellver et al., 2020, Liang et al., 24 Jan 2025).
• Temporal consistency and efficiency tradeoffs: Achieving temporally stable but spatially detailed masks at scale often conflicts with computational tractability, especially for minute-long videos (Zhang et al., 28 Sep 2025, Yan et al., 9 Oct 2025).
• Foundational evaluation metrics: The introduction of temporal and spatiotemporal metrics is advancing the field, but more unified, application-aligned metrics are necessary to compare future architectures fairly on real-world criteria.

This suggests that the RVOS field will continue to see rapid growth along two interrelated axes: foundation model adaptation with weak or no training, and principled scaling in time, space, and modality to match real-world deployment—requiring ongoing innovation in cross-modal aggregation, lightweight temporal modeling, and robust evaluation.