---
title: 'GeoLaV: Geometry-enhanced RVOS'
url: https://www.emergentmind.com/topics/geometry-enhanced-language-guided-video-segmentation-geolav
type: topic
---

# GeoLaV: Geometry-enhanced RVOS

GeoLaV, short for **Geometry-enhanced Language-guided Video segmentation**, is a two-stage framework for **text-driven referring video object segmentation (RVOS)**. It is motivated by the claim that standard RVOS pipelines are largely optimized with 2D supervision and therefore underutilize the **3D geometric consistency** that links objects across frames. GeoLaV addresses this by first learning geometry-consistent visual representations from synthetic multi-view sequences generated from single images, and then distilling 3D structural priors from frozen teachers during video fine-tuning. In the formulation used by the method, the input is a video $\mathbf{V}=\{\mathbf{I}_t\}_{t=1}^{T}$ and a referring expression $\mathbf{W}=\{w_i\}_{i=1}^{L}$, and the output is a binary mask sequence $\mathbf{M}=\{\mathbf{M}_t\}_{t=1}^{T}$ with $\mathbf{M}=\mathcal{G}(\mathbf{V},\mathbf{W})$ [2606.24464].

## 1. Problem setting and conceptual position

GeoLaV is situated within the RVOS problem family, where a model must localize and segment the object referred to by language across a video. Earlier work established the practical importance of replacing first-frame masks with language referring expressions, arguing that language can be a cheaper and more natural supervision modality while also helping to reduce drift and improve robustness to appearance change [1803.08006]. Subsequent RVOS and video reasoning segmentation systems concentrated on temporal memory, multimodal fusion, or MLLM-based reasoning, but generally did not inject explicit geometry. Locater introduced a finite memory for local-global temporal context [2203.09773], ViLLa emphasized reasoning-aware video segmentation with context aggregation and multi-level segmentation tokens [2407.14500], GLUS unified sparse global context frames and continuous query frames inside a single MLLM [2504.07962], and SteerSeg improved grounding by steering LVLM attention rather than by modeling 3D structure [2605.14908].

Against that backdrop, GeoLaV’s central claim is that **geometric consistency and 3D-aware priors should be explicitly injected into language-guided video segmentation**. The method differs from image-only referring segmentation because it targets temporal and geometric consistency rather than only text-image alignment, from video RVOS methods that rely on memory or temporal attention because it adds explicit geometry-aware supervision rather than only implicit temporal modeling, and from large-VLM systems because it improves a non-large-VLM architecture with geometry rather than depending on giant multimodal language models [2606.24464].

This positioning is especially consequential on motion-centric datasets. MeViS, for example, was designed so that the target often cannot be identified from a single frame, includes **2,006 videos**, **8,171 objects**, **28,570 expressions**, and **443k masks**, and emphasizes motion expressions and multi-object ambiguity rather than static attributes [2308.08544]. That benchmark logic aligns closely with GeoLaV’s premise that cross-frame structure, rather than framewise 2D supervision alone, is central to robust language grounding.

## 2. Two-stage architecture and backbone design

GeoLaV is “basically built upon the **SAMWISE** framework,” with geometry-aware modifications. The core architecture uses a **frozen SAM2 image encoder** as visual backbone, a **frozen text encoder**, trainable **cross-modal adapters**, frozen **memory attention**, a **prompt encoder** that keeps mostly frozen SAM2 weights with a small trainable subset, a **mask decoder**, and a **memory encoder / bank** for temporal state. Geometry enters not by changing the inference-time decoding paradigm, but by changing how the representation is pretrained and regularized [2606.24464].

The framework is explicitly divided into two stages. **Stage I: Monocular Geometry Pretraining (MGP)** converts single images into pseudo-videos through monocular depth-based novel-view synthesis, allowing the RVOS backbone to learn geometry-consistent representations from image referring segmentation data. **Stage II: Geometry-Aware Distillation (GAD)** fine-tunes the Stage-I-initialized model on real RVOS videos while regularizing its intermediate memory features with two frozen teachers: a **3D-aware teacher** and a **VFM semantic teacher** [2606.24464].

This organization implies a specific view of geometry. GeoLaV does not use explicit geometry as an inference-time reconstruction module. Instead, geometry is treated as a **training prior** that shapes the intermediate representation underlying language-conditioned mask prediction. The projection heads used for distillation are **training-only**, so the geometry-aware branches add **no inference overhead** [2606.24464].

## 3. Monocular Geometry Pretraining

Stage I begins from a monocular image $\mathbf{I}_{\text{mon}}$. A geometry model $\mathcal{M}_{\text{geo}}$ predicts dense depth, using models such as **$\pi^3$** or **VGGT**. To handle scale ambiguity, the predicted depth is perturbed as
$$
\mathbf{D} = a\,\mathcal{M}_{\text{geo}}(\mathbf{I}_{\text{mon}}) + b.
$$
The image is then lifted into 3D, transformed along a continuous short virtual camera trajectory, reprojected, and inpainted in occluded regions to form a synthetic sequence. The resulting pseudo-video is denoted $\mathbf{V}_{\text{syn}}=\{\mathbf{I}^{t}\}_{t=1}^{T}$, and in experiments **five novel-view images are synthesized from each COCO single-view input** [2606.24464].

The synthetic sequence is processed by the **frozen SAM2 image encoder** $\mathcal{E}_{\text{img}}$, while the referring sentence $\mathbf{W}$ is processed by the **frozen text encoder** $\mathcal{E}_{\text{txt}}$:
$$
\{\mathbf{F}^{t,l}\}_{l=1}^{L} = \mathcal{E}_{\text{img}}(\mathbf{I}^{t}), \qquad
\{\mathbf{P}^{l}\}_{l=1}^{L} = \mathcal{E}_{\text{txt}}(\mathbf{W}).
$$
A trainable cross-modal adapter aligns the two modalities:
$$
\mathbf{F}^{\prime\,t,l} = \mathcal{A}^{l}_{\theta}\big(\mathbf{F}^{t,l}, \mathbf{P}^{l}\big).
$$
The adapted features are passed through the frozen memory attention module to produce memory-enhanced features $\mathbf{F}_{\text{mem}}$ [2606.24464].

In parallel, a **visual foundation model encoder** such as **DINOv3** processes the same synthetic video and provides teacher features. A projection head
$$
\mathbf{F}_{\text{proj}} = \mathcal{P}_{\theta}(\mathbf{F}_{\text{mem}}) = \mathbf{W}_{2}\,\sigma(\mathbf{W}_{1}\mathbf{F}_{\text{mem}})
$$
maps the student memory feature into the VFM embedding space, and alignment is enforced with cosine similarity:
$$
\mathcal{L}_{\text{sim}} = 1 - \mathrm{Sim}(\mathbf{F}_{\text{proj}}, \mathbf{F}_{\text{VFM}}).
$$
The Stage-I loss is
$$
\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{seg}} + \mathcal{L}_{\text{sim}}, \qquad
\mathcal{L}_{\text{seg}} = \mathcal{L}_{\text{dice}} + \mathcal{L}_{\text{mask}}.
$$
The paper explicitly notes that it does **not** include photometric reconstruction, depth smoothness, epipolar, or neural rendering losses; geometry is imposed through the synthesis process and feature alignment rather than through explicit 3D reconstruction objectives [2606.24464].

## 4. Geometry-aware distillation and representation learning

Stage II fine-tunes the student on real RVOS videos while injecting geometry priors from frozen teachers. The two teachers are a **3D-aware encoder**, exemplified by **$\pi^3$**, and a **VFM encoder**, exemplified by **DINOv3-ViT-L**. Their intended roles are complementary: the 3D teacher captures **structural and depth relationships**, while the VFM teacher captures **semantic boundaries and category-level object cues** [2606.24464].

A defining feature of GeoLaV is that it distills **features**, not explicit depth maps, occupancy grids, camera poses, or masks. The distillation target is the student’s **memory representation**, before prompt-conditioned mask decoding. The teacher alignment is therefore feature-level and acts on **general frame-level representations rather than object-specific features**. Two projection heads are used:
$$
\mathbf{F}^{(k)}_{\text{proj}} = \mathcal{P}^{(k)}_{\theta}\big(\mathbf{F}_{\text{mem}}\big)
= \mathbf{W}^{(k)}_{2}\,\sigma\big(\mathbf{W}^{(k)}_{1}\mathbf{F}_{\text{mem}}\big), \quad
k \in \{\text{3D}, \text{VFM}\}.
$$
The dual-teacher distillation loss is
$$
\mathcal{L}_{\text{distill}} = \mathcal{L}_{\text{3D}} + \mathcal{L}_{\text{VFM}}
= \sum_{k \in \{\text{3D},\, \text{VFM}\}}
\left[1 - \mathrm{Sim}\big(\mathbf{F}^{(k)}_{\text{proj}}, \mathbf{F}_{k}\big)\right].
$$
The text states that Stage II combines this distillation with the same segmentation objective used in Stage I [2606.24464].

This separation between geometry prior and referential supervision is central. Language grounding is still learned from referred-object mask supervision through the SAMWISE/SAM2-style decoding path; geometry-aware distillation regularizes the backbone so that the representation is more stable under viewpoint change and cross-frame structural variation. This suggests that GeoLaV’s contribution is not a new explicit 3D inference engine, but a method for making the **video-language memory representation** itself more geometry-consistent.

## 5. Training data, benchmarks, and empirical behavior

GeoLaV uses **COCO** as the image source for synthetic pseudo-video generation and trains Stage I on **RefCOCO**, **RefCOCO+**, and **RefCOCOg**. For video fine-tuning and evaluation it uses **Ref-YouTube-VOS**, **Ref-DAVIS17**, and **MeViS**. The paper describes these datasets as follows: **Ref-YouTube-VOS** has **3,978 high-resolution videos** and about **15K expressions**; **Ref-DAVIS17** has **90 videos** and over **1.5K linguistic annotations**; **MeViS** has **2,006 videos** and about **28K motion-centric expressions** [2606.24464].

Evaluation uses $\mathcal{J}$, $\mathcal{F}$, and their mean $\mathcal{J}{+}\mathcal{F}$. In the zero-shot setting, GeoLaV is trained **only on RefCOCO/+/g**, synthetic multi-view pseudo-videos are generated from images, there is **no video fine-tuning**, and the model is directly tested on **Ref-YouTube-VOS** and **MeViS**. Under this protocol, GeoLaV obtains **47.0** on **Ref-YouTube-VOS** and **31.6** on **MeViS**, compared with image-only SAMWISE results of **31.9** and **26.4**, respectively. The reported improvement is therefore **+15.1** on Ref-YouTube-VOS and **+5.2** on MeViS [2606.24464].

After video fine-tuning with geometry-aware distillation, GeoLaV reports **70.5** $\mathcal{J}{+}\mathcal{F}$ on **Ref-YouTube-VOS**, **72.5** on **Ref-DAVIS17**, and **50.0** on **MeViS**. The corresponding $\mathcal{J}$ and $\mathcal{F}$ values are **69.1 / 71.8** for Ref-YouTube-VOS, **69.9 / 75.2** for Ref-DAVIS17, and **47.4 / 52.9** for MeViS. The paper characterizes these results as state of the art among non-large-VLM methods, with gains of **+1.2** over the previous best on Ref-YouTube-VOS, **+1.9** on Ref-DAVIS17, and **+0.5** on MeViS, while using **202M parameters** [2606.24464].

The ablations isolate the effect of geometry. The reported $\mathcal{J}{+}\mathcal{F}$ values are **65.1** for the vanilla model, **66.5** with planar augmentation, **67.2** with **MGP**, **67.5** with **GAD**, and **70.5** for the full model. The paper interprets this as evidence that planar augmentation helps a little, MGP helps more than planar augmentation, GAD alone also helps strongly, and the combination is substantially stronger than either component alone [2606.24464]. Qualitatively, the method is reported to avoid including the **boat wake** when segmenting a boat and to more consistently track the **rightmost giraffe** across frames, which the authors attribute to improved boundaries, reduced distractor leakage, and stronger cross-frame identity stability [2606.24464].

## 6. Relation to the broader literature and methodological significance

GeoLaV belongs to a line of work that treats language-guided video segmentation as a progression from 2D semantic grounding toward stronger spatiotemporal reasoning. The 2018 language-referring-expression baseline replaced first-frame masks with language and added a temporal consistency re-ranking heuristic, but it contained **no explicit geometry modeling** and relied on 2D box overlap and temporal distance [1803.08006]. Later systems such as the deeply interleaved two-stream encoder used progressive vision-language fusion and language-guided temporal filtering, yet still lacked explicit depth, flow, or 3D correspondence [2203.15969]. Locater introduced fixed-capacity global and local memory for efficient long-range temporal reasoning without geometric correspondence [2203.09773]. ViLLa and GLUS extended the field toward reasoning-heavy and MLLM-centered formulations, but they likewise remained non-geometric in the explicit sense [2407.14500; 2504.07962].

In that landscape, GeoLaV is distinguished by treating geometry as a **representation prior** rather than as an external auxiliary signal or an implicit by-product of temporal attention. This makes it different from methods such as SteerSeg, which improve grounding by producing more spatially concentrated attention maps and better temporal tracklet selection, yet are best understood as **spatially enhanced rather than explicitly geometry-enhanced** [2605.14908]. A plausible implication is that GeoLaV and attention-centric or MLLM-centric methods are not mutually exclusive; the former regularizes the intermediate representation with 3D-aware structure, while the latter improve reasoning, ambiguity resolution, or global-local coordination.

The limitations stated or implied by the GeoLaV paper are correspondingly specific. The method depends on external geometry priors such as **$\pi^3$** or **VGGT**, incurs additional training-time overhead from novel-view synthesis and dual-teacher distillation, may face domain mismatch between synthetic geometry and real video, does **not** explicitly model dynamic 3D motion, and omits some implementation details such as exact loss weights and feature resolutions [2606.24464]. At the same time, because the projection heads are removed at test time, the extra cost is primarily **training-only**, not inference-time [2606.24464].

GeoLaV’s broader significance lies in its explicit answer to a long-running deficiency in RVOS training. Rather than assuming that better language grounding follows from stronger text-image supervision alone, it argues that video-language segmentation benefits when the backbone is made **geometry-aware** before and during video learning. For RVOS research, this establishes a distinct methodological axis: beyond memory, propagation, attention steering, and MLLM reasoning, **3D structural consistency** can be transferred into language-guided video segmentation without turning the inference pipeline into a full 3D reconstruction system [2606.24464].

Source: https://www.emergentmind.com/topics/geometry-enhanced-language-guided-video-segmentation-geolav