GeoLaV: Geometry-enhanced RVOS

• The paper introduces a two-stage framework that leverages monocular geometry pretraining and geometry-aware distillation to enhance cross-frame segmentation consistency.
• It distills geometry-aware features by aligning the student’s memory representation with both 3D structural and semantic cues, adding no inference overhead.
• Empirical results demonstrate significant performance gains on benchmarks like Ref-YouTube-VOS and MeViS, validating the approach over traditional 2D methods.

GeoLaV, short for Geometry-enhanced Language-guided Video segmentation, is a two-stage framework for text-driven referring video object segmentation (RVOS). It is motivated by the claim that standard RVOS pipelines are largely optimized with 2D supervision and therefore underutilize the 3D geometric consistency that links objects across frames. GeoLaV addresses this by first learning geometry-consistent visual representations from synthetic multi-view sequences generated from single images, and then distilling 3D structural priors from frozen teachers during video fine-tuning. In the formulation used by the method, the input is a video $\mathbf{V}=\{\mathbf{I}_t\}_{t=1}^{T}$ and a referring expression $\mathbf{W}=\{w_i\}_{i=1}^{L}$, and the output is a binary mask sequence $\mathbf{M}=\{\mathbf{M}_t\}_{t=1}^{T}$ with $\mathbf{M}=\mathcal{G}(\mathbf{V},\mathbf{W})$ (Zhu et al., 23 Jun 2026).

1. Problem setting and conceptual position

GeoLaV is situated within the RVOS problem family, where a model must localize and segment the object referred to by language across a video. Earlier work established the practical importance of replacing first-frame masks with language referring expressions, arguing that language can be a cheaper and more natural supervision modality while also helping to reduce drift and improve robustness to appearance change (Khoreva et al., 2018). Subsequent RVOS and video reasoning segmentation systems concentrated on temporal memory, multimodal fusion, or MLLM-based reasoning, but generally did not inject explicit geometry. Locater introduced a finite memory for local-global temporal context (Liang et al., 2022), ViLLa emphasized reasoning-aware video segmentation with context aggregation and multi-level segmentation tokens (Zheng et al., 2024), GLUS unified sparse global context frames and continuous query frames inside a single MLLM (Lin et al., 10 Apr 2025), and SteerSeg improved grounding by steering LVLM attention rather than by modeling 3D structure (Cheraghian et al., 14 May 2026).

Against that backdrop, GeoLaV’s central claim is that geometric consistency and 3D-aware priors should be explicitly injected into language-guided video segmentation. The method differs from image-only referring segmentation because it targets temporal and geometric consistency rather than only text-image alignment, from video RVOS methods that rely on memory or temporal attention because it adds explicit geometry-aware supervision rather than only implicit temporal modeling, and from large-VLM systems because it improves a non-large-VLM architecture with geometry rather than depending on giant multimodal LLMs (Zhu et al., 23 Jun 2026).

This positioning is especially consequential on motion-centric datasets. MeViS, for example, was designed so that the target often cannot be identified from a single frame, includes 2,006 videos, 8,171 objects, 28,570 expressions, and 443k masks, and emphasizes motion expressions and multi-object ambiguity rather than static attributes (Ding et al., 2023). That benchmark logic aligns closely with GeoLaV’s premise that cross-frame structure, rather than framewise 2D supervision alone, is central to robust language grounding.

2. Two-stage architecture and backbone design

GeoLaV is “basically built upon the SAMWISE framework,” with geometry-aware modifications. The core architecture uses a frozen SAM2 image encoder as visual backbone, a frozen text encoder, trainable cross-modal adapters, frozen memory attention, a prompt encoder that keeps mostly frozen SAM2 weights with a small trainable subset, a mask decoder, and a memory encoder / bank for temporal state. Geometry enters not by changing the inference-time decoding paradigm, but by changing how the representation is pretrained and regularized (Zhu et al., 23 Jun 2026).

The framework is explicitly divided into two stages. Stage I: Monocular Geometry Pretraining (MGP) converts single images into pseudo-videos through monocular depth-based novel-view synthesis, allowing the RVOS backbone to learn geometry-consistent representations from image referring segmentation data. Stage II: Geometry-Aware Distillation (GAD) fine-tunes the Stage-I-initialized model on real RVOS videos while regularizing its intermediate memory features with two frozen teachers: a 3D-aware teacher and a VFM semantic teacher (Zhu et al., 23 Jun 2026).

This organization implies a specific view of geometry. GeoLaV does not use explicit geometry as an inference-time reconstruction module. Instead, geometry is treated as a training prior that shapes the intermediate representation underlying language-conditioned mask prediction. The projection heads used for distillation are training-only, so the geometry-aware branches add no inference overhead (Zhu et al., 23 Jun 2026).

3. Monocular Geometry Pretraining

Stage I begins from a monocular image $\mathbf{I}_{\text{mon}}$. A geometry model $\mathcal{M}_{\text{geo}}$ predicts dense depth, using models such as $\pi^3$ or VGGT. To handle scale ambiguity, the predicted depth is perturbed as

$$\mathbf{D} = a\,\mathcal{M}_{\text{geo}}(\mathbf{I}_{\text{mon}}) + b.$$

The image is then lifted into 3D, transformed along a continuous short virtual camera trajectory, reprojected, and inpainted in occluded regions to form a synthetic sequence. The resulting pseudo-video is denoted $$\mathbf{V}_{\text{syn}}=\{\mathbf{I}^{t}\}_{t=1}^{T}$$, and in experiments five novel-view images are synthesized from each COCO single-view input (Zhu et al., 23 Jun 2026).

The synthetic sequence is processed by the frozen SAM2 image encoder $\mathcal{E}_{\text{img}}$, while the referring sentence $\mathbf{W}=\{w_i\}_{i=1}^{L}$0 is processed by the frozen text encoder $\mathbf{W}=\{w_i\}_{i=1}^{L}$1:

$\mathbf{W}=\{w_i\}_{i=1}^{L}$2

A trainable cross-modal adapter aligns the two modalities:

$\mathbf{W}=\{w_i\}_{i=1}^{L}$3

The adapted features are passed through the frozen memory attention module to produce memory-enhanced features $\mathbf{W}=\{w_i\}_{i=1}^{L}$4 (Zhu et al., 23 Jun 2026).

In parallel, a visual foundation model encoder such as DINOv3 processes the same synthetic video and provides teacher features. A projection head

$\mathbf{W}=\{w_i\}_{i=1}^{L}$5

maps the student memory feature into the VFM embedding space, and alignment is enforced with cosine similarity:

$\mathbf{W}=\{w_i\}_{i=1}^{L}$6

The Stage-I loss is

$\mathbf{W}=\{w_i\}_{i=1}^{L}$7

The paper explicitly notes that it does not include photometric reconstruction, depth smoothness, epipolar, or neural rendering losses; geometry is imposed through the synthesis process and feature alignment rather than through explicit 3D reconstruction objectives (Zhu et al., 23 Jun 2026).

4. Geometry-aware distillation and representation learning

Stage II fine-tunes the student on real RVOS videos while injecting geometry priors from frozen teachers. The two teachers are a 3D-aware encoder, exemplified by $\mathbf{W}=\{w_i\}_{i=1}^{L}$8, and a VFM encoder, exemplified by DINOv3-ViT-L. Their intended roles are complementary: the 3D teacher captures structural and depth relationships, while the VFM teacher captures semantic boundaries and category-level object cues (Zhu et al., 23 Jun 2026).

A defining feature of GeoLaV is that it distills features, not explicit depth maps, occupancy grids, camera poses, or masks. The distillation target is the student’s memory representation, before prompt-conditioned mask decoding. The teacher alignment is therefore feature-level and acts on general frame-level representations rather than object-specific features. Two projection heads are used:

$\mathbf{W}=\{w_i\}_{i=1}^{L}$9

The dual-teacher distillation loss is

$\mathbf{M}=\{\mathbf{M}_t\}_{t=1}^{T}$0

The text states that Stage II combines this distillation with the same segmentation objective used in Stage I (Zhu et al., 23 Jun 2026).

This separation between geometry prior and referential supervision is central. Language grounding is still learned from referred-object mask supervision through the SAMWISE/SAM2-style decoding path; geometry-aware distillation regularizes the backbone so that the representation is more stable under viewpoint change and cross-frame structural variation. This suggests that GeoLaV’s contribution is not a new explicit 3D inference engine, but a method for making the video-language memory representation itself more geometry-consistent.

5. Training data, benchmarks, and empirical behavior

GeoLaV uses COCO as the image source for synthetic pseudo-video generation and trains Stage I on RefCOCO, RefCOCO+, and RefCOCOg. For video fine-tuning and evaluation it uses Ref-YouTube-VOS, Ref-DAVIS17, and MeViS. The paper describes these datasets as follows: Ref-YouTube-VOS has 3,978 high-resolution videos and about 15K expressions; Ref-DAVIS17 has 90 videos and over 1.5K linguistic annotations; MeViS has 2,006 videos and about 28K motion-centric expressions (Zhu et al., 23 Jun 2026).

Evaluation uses $\mathbf{M}=\{\mathbf{M}_t\}_{t=1}^{T}$1, $\mathbf{M}=\{\mathbf{M}_t\}_{t=1}^{T}$2, and their mean $\mathbf{M}=\{\mathbf{M}_t\}_{t=1}^{T}$3. In the zero-shot setting, GeoLaV is trained only on RefCOCO/+/g, synthetic multi-view pseudo-videos are generated from images, there is no video fine-tuning, and the model is directly tested on Ref-YouTube-VOS and MeViS. Under this protocol, GeoLaV obtains 47.0 on Ref-YouTube-VOS and 31.6 on MeViS, compared with image-only SAMWISE results of 31.9 and 26.4, respectively. The reported improvement is therefore +15.1 on Ref-YouTube-VOS and +5.2 on MeViS (Zhu et al., 23 Jun 2026).

After video fine-tuning with geometry-aware distillation, GeoLaV reports 70.5 $\mathbf{M}=\{\mathbf{M}_t\}_{t=1}^{T}$4 on Ref-YouTube-VOS, 72.5 on Ref-DAVIS17, and 50.0 on MeViS. The corresponding $\mathbf{M}=\{\mathbf{M}_t\}_{t=1}^{T}$5 and $\mathbf{M}=\{\mathbf{M}_t\}_{t=1}^{T}$6 values are 69.1 / 71.8 for Ref-YouTube-VOS, 69.9 / 75.2 for Ref-DAVIS17, and 47.4 / 52.9 for MeViS. The paper characterizes these results as state of the art among non-large-VLM methods, with gains of +1.2 over the previous best on Ref-YouTube-VOS, +1.9 on Ref-DAVIS17, and +0.5 on MeViS, while using 202M parameters (Zhu et al., 23 Jun 2026).

The ablations isolate the effect of geometry. The reported $\mathbf{M}=\{\mathbf{M}_t\}_{t=1}^{T}$7 values are 65.1 for the vanilla model, 66.5 with planar augmentation, 67.2 with MGP, 67.5 with GAD, and 70.5 for the full model. The paper interprets this as evidence that planar augmentation helps a little, MGP helps more than planar augmentation, GAD alone also helps strongly, and the combination is substantially stronger than either component alone (Zhu et al., 23 Jun 2026). Qualitatively, the method is reported to avoid including the boat wake when segmenting a boat and to more consistently track the rightmost giraffe across frames, which the authors attribute to improved boundaries, reduced distractor leakage, and stronger cross-frame identity stability (Zhu et al., 23 Jun 2026).

6. Relation to the broader literature and methodological significance

GeoLaV belongs to a line of work that treats language-guided video segmentation as a progression from 2D semantic grounding toward stronger spatiotemporal reasoning. The 2018 language-referring-expression baseline replaced first-frame masks with language and added a temporal consistency re-ranking heuristic, but it contained no explicit geometry modeling and relied on 2D box overlap and temporal distance (Khoreva et al., 2018). Later systems such as the deeply interleaved two-stream encoder used progressive vision-language fusion and language-guided temporal filtering, yet still lacked explicit depth, flow, or 3D correspondence (Feng et al., 2022). Locater introduced fixed-capacity global and local memory for efficient long-range temporal reasoning without geometric correspondence (Liang et al., 2022). ViLLa and GLUS extended the field toward reasoning-heavy and MLLM-centered formulations, but they likewise remained non-geometric in the explicit sense (Zheng et al., 2024, Lin et al., 10 Apr 2025).

In that landscape, GeoLaV is distinguished by treating geometry as a representation prior rather than as an external auxiliary signal or an implicit by-product of temporal attention. This makes it different from methods such as SteerSeg, which improve grounding by producing more spatially concentrated attention maps and better temporal tracklet selection, yet are best understood as spatially enhanced rather than explicitly geometry-enhanced (Cheraghian et al., 14 May 2026). A plausible implication is that GeoLaV and attention-centric or MLLM-centric methods are not mutually exclusive; the former regularizes the intermediate representation with 3D-aware structure, while the latter improve reasoning, ambiguity resolution, or global-local coordination.

The limitations stated or implied by the GeoLaV paper are correspondingly specific. The method depends on external geometry priors such as $\mathbf{M}=\{\mathbf{M}_t\}_{t=1}^{T}$8 or VGGT, incurs additional training-time overhead from novel-view synthesis and dual-teacher distillation, may face domain mismatch between synthetic geometry and real video, does not explicitly model dynamic 3D motion, and omits some implementation details such as exact loss weights and feature resolutions (Zhu et al., 23 Jun 2026). At the same time, because the projection heads are removed at test time, the extra cost is primarily training-only, not inference-time (Zhu et al., 23 Jun 2026).

GeoLaV’s broader significance lies in its explicit answer to a long-running deficiency in RVOS training. Rather than assuming that better language grounding follows from stronger text-image supervision alone, it argues that video-language segmentation benefits when the backbone is made geometry-aware before and during video learning. For RVOS research, this establishes a distinct methodological axis: beyond memory, propagation, attention steering, and MLLM reasoning, 3D structural consistency can be transferred into language-guided video segmentation without turning the inference pipeline into a full 3D reconstruction system (Zhu et al., 23 Jun 2026).