FlowRVS: Continuous Flow for Referring Video Segmentation

• FlowRVS is a novel framework that recasts referring video segmentation as a continuous, deterministic deformation from video latent to mask representation under global language conditioning.
• It leverages pretrained text-to-video models and an ODE-based flow with techniques like Boundary-Biased Sampling, Start-Point Augmentation, and Direct Video Injection to enhance accuracy.
• Empirical evaluations on benchmarks such as MeViS, Ref-DAVIS17, and Ref-YouTube-VOS demonstrate state-of-the-art performance with improved temporal consistency and pixel-level precision.

Searching arXiv for the specified FlowRVS paper and closely related context papers. FlowRVS is a framework for referring video object segmentation (RVOS) that reformulates the task as a conditional continuous flow problem rather than a cascaded “locate-then-segment” pipeline. In this formulation, a holistic spatio-temporal video representation is deformed directly into the target mask sequence under persistent natural-language conditioning, with the aim of preserving fine-grained pixel control, text-video semantic alignment, and temporal coherence. The method was introduced in “Deforming Videos to Masks: Flow Matching for Referring Video Segmentation” (Wang et al., 7 Oct 2025).

1. Task setting and motivation

Referring video object segmentation requires segmenting specific objects in a video guided by a natural-language description. The central difficulties are grounding abstract linguistic concepts into a specific set of pixels and maintaining temporal consistency through complex video dynamics. Examples noted for this task include expressions such as “the smaller monkey,” as well as temporally structured queries such as “the first tiger to run across” (Wang et al., 7 Oct 2025).

Prior work is described as predominantly relying on a two-stage “locate-then-segment” design. In that pattern, a visual-language grounding model first maps the query to a coarse geometric prompt, such as a point, region, or bounding box, and a subsequent segmentation stage produces the pixel mask. The limitations emphasized for this family of methods are threefold. First, the locate stage introduces an information bottleneck by compressing rich linguistic semantics into a coarse geometric intermediary. Second, segmentation is often decoupled from language grounding, so the final mask may only weakly reflect the original query. Third, temporal handling is fragile when videos contain occlusions, appearance changes, or temporally specific language (Wang et al., 7 Oct 2025).

FlowRVS is positioned as a direct response to these limitations. Rather than treating language grounding and segmentation as separate subproblems, it performs a one-stage generative transformation from video representation to mask representation while retaining language as a global condition throughout the trajectory. This suggests that the framework is intended not merely as a segmentation head attached to a grounding module, but as a unified spatio-temporal-linguistic inference process.

2. Continuous deformation formulation

The defining idea of FlowRVS is to model RVOS as learning a continuous, deterministic flow from a video latent to a mask latent. Let $V$ denote the video input, $c$ the language query, $\mathbf{z}_0 \sim \mathcal{P}_{video}$ the latent encoding of the video, and $\mathbf{z}_1 \sim \mathcal{P}_{mask}$ the latent of the target mask sequence. The core evolution equation is

$$\frac{d\mathbf{z}_t}{dt} = \mathbf{v}(\mathbf{z}_t, c, t),$$

where $\mathbf{v}(\cdot)$ is a learned velocity field and $t \in [0,1]$ indexes the trajectory from video latent to mask latent (Wang et al., 7 Oct 2025).

An ODE solver is used to integrate this vector field and deterministically obtain the target mask latent, which is then decoded into pixel masks. The text query $c$ is not treated as an initialization artifact or auxiliary prompt; it is a global conditional available along the trajectory. This is a central distinction from schemes that first derive a coarse cue from text and then discard most of the original semantic structure.

The paper contrasts this formulation with standard text-to-video generation. Text-to-video is characterized there as a divergent, noise-to-video, one-to-many process, whereas RVOS is described as a convergent, video-to-mask, many-to-one process. A plausible implication is that FlowRVS interprets segmentation as conditional information contraction: the model starts from a dense spatio-temporal representation and progressively contracts it into a sparse, query-consistent mask.

3. Backbone reuse and end-to-end architecture

FlowRVS is built on top of pretrained text-to-video models such as Wan 2.1 (Wan et al., 26 Mar 2025). These models are highlighted as being trained for pixel-level video synthesis conditioned on text and as natively modeling fine-grained spatial and temporal dynamics. The framework reuses this pretrained multimodal structure, but changes the operational direction from noise-to-video generation to video-to-mask deformation (Wang et al., 7 Oct 2025).

The input video is encoded by the frozen video or visual encoder of the text-to-video model to produce $\mathbf{z}_0$. The language query is embedded by a frozen text encoder, described in the source as being from “CLIP or similar.” These representations condition a flow model in which the velocity field $\mathbf{v}(\mathbf{z}_t, c, t)$ is realized by a Diffusion Transformer (DiT) block fine-tuned for the video-to-mask deformation task. Over multiple ODE steps, the entire spatio-temporal latent is deformed into the latent of the target mask sequence (Wang et al., 7 Oct 2025).

The framework explicitly rejects two naive transfer strategies. One is “feature-extraction decoherence,” in which frozen text-to-video features are used with a separate decoder, thereby decoupling temporal and linguistic reasoning. The other is “text-decoupled flows,” referring to flow-matching adaptations that ignore text conditioning and therefore cannot resolve object selection under referring expressions. FlowRVS instead preserves the generative backbone and keeps language conditioning active at every step.

The paper also states that the visual encoder is frozen while the VAE decoder is fine-tuned specifically on mask outputs using mask-appropriate losses, exemplified by a combined focal and dice loss. This decoder adaptation is intended to improve high-fidelity mask reconstruction from the latent space (Wang et al., 7 Oct 2025).

4. Task-specific training and inference mechanisms

Three named mechanisms are presented as key adaptations for converting a pretrained text-to-video generator into a discriminative RVOS model: Boundary-Biased Sampling (BBS), Start-Point Augmentation (SPA), and Direct Video Injection (DVI) (Wang et al., 7 Oct 2025).

Boundary-Biased Sampling (BBS) oversamples ODE time steps near $c$0 during training. The stated motivation is that the initial motion away from the video manifold is critical in the video-to-mask setting, because the earliest steps must push the latent toward the correct object under the semantics of the query. The paper reports that proper BBS boosts performance by +10 points and improves training stability, accuracy, and correspondence.

Start-Point Augmentation (SPA) perturbs the initial latent $c$1 stochastically during training. This creates a denser region of start states and regularizes the learned velocity field, with the stated effect of improving robustness to variation in real videos.

Direct Video Injection (DVI) makes the initial video latent available throughout the trajectory by concatenating $c$2 with the current state $c$3 at every ODE step:

$c$4

The purpose is to prevent drift away from the original video content and to support precise, temporally consistent mask generation. The paper states that DVI further increases accuracy and stability by +2 points (Wang et al., 7 Oct 2025).

These mechanisms jointly define the operational character of FlowRVS. BBS shapes supervision over trajectory time, SPA smooths the start-state geometry, and DVI anchors the evolving latent to the original video content. This suggests that the method is not only a reformulation of RVOS in continuous-flow terms, but also a targeted redesign of optimization dynamics for a convergent generative task.

5. Temporal consistency and pixel-level selection

A major claim of FlowRVS is that temporal consistency is handled intrinsically rather than by post hoc linkage across independently segmented frames. Because the model evolves a holistic video latent through a deterministic ODE flow, temporal relationships are preserved within the state itself rather than reconstructed from frame-wise predictions (Wang et al., 7 Oct 2025).

The method is also described as providing fine-grained pixel control. Since the entire deformation is conditioned by both the full video latent and the text embedding, the model can directly select relevant pixels in each frame without relying on coarse prompts such as points or boxes. In the paper’s framing, this avoids the semantic compression characteristic of earlier pipelines and supports segmentation under abstract, dynamic, or relational descriptions.

The qualitative behavior singled out in the source includes stronger handling of nuanced queries such as “the jumping rabbit” and temporally structured expressions such as “the first tiger...”. The paper attributes this to the fact that visual, temporal, and linguistic reasoning remain unified throughout the deformation process, rather than being broken into loosely coupled modules (Wang et al., 7 Oct 2025).

A common misconception would be to treat FlowRVS as either a standard noise-to-mask generator or a direct discriminative mask predictor. The paper explicitly positions it against both alternatives: it is neither “generating from noise to mask” nor “directly predicting mask,” but instead learns a direct, language-guided deformation from a video’s holistic representation to its target mask.

6. Empirical evaluation and reported significance

FlowRVS is evaluated on MeViS, Ref-YouTube-VOS, and Ref-DAVIS17 using $c$5 for region similarity, $c$6 for contour accuracy, and their average $c$7. The reported benchmark results are as follows (Wang et al., 7 Oct 2025):

Benchmark	FlowRVS	Prior SOTA / Δ
MeViS	51.1	49.5 (SAMWISE), +1.6
Ref-DAVIS17 (zero-shot)	73.3	70.6 (SAMWISE), +2.7
Ref-YouTube-VOS	69.6	69.3 (ReferDINO), +0.3

The paper describes these as new state-of-the-art results across the major RVOS benchmarks. MeViS is characterized as a setting with long videos, complex motion, and nuanced temporal and linguistic tests. Ref-DAVIS17 is described as emphasizing high-quality, high-precision annotations and temporal consistency, while Ref-YouTube-VOS is presented as a large-scale benchmark with diverse scenes.

The ablation results emphasized in the source identify BBS as “absolutely critical,” with proper $c$8 bias giving +10 points, and DVI as contributing an additional +2 points. The paper also states that removing the pretrained text-to-video weights causes performance collapse, which is presented as evidence that the underlying multimodal pretraining is essential (Wang et al., 7 Oct 2025).

The broader significance claimed for FlowRVS is fourfold: it bypasses information bottlenecks introduced by intermediate prompts, enables global and temporally consistent pixel-accurate segmentation, unifies visual, temporal, and linguistic reasoning in one stage, and demonstrates strong zero-shot generalization. A plausible implication is that the framework may be read as a concrete example of repurposing video generation priors for discriminative video understanding through continuous deformation rather than autoregressive or frame-wise prediction.