---
title: PVUW MeViS-Text Challenge
url: https://www.emergentmind.com/topics/pvuw-mevis-text-challenge
type: topic
---

# PVUW MeViS-Text Challenge

PVUW MeViS-Text Challenge is the PVUW track for motion-expression-guided, text-based video object segmentation, built on the MeViS benchmark. In this setting, a system receives a video and a natural-language query that emphasizes motion, temporal dynamics, action, interaction, relative position, exclusion, or temporal behavior, and must localize and segment the referred object or objects throughout the video. The track is a specialized form of referring video object segmentation (RVOS), but it differs from appearance-dominant RVOS benchmarks in that the referent often cannot be identified from a single frame alone; correct grounding may depend on evidence distributed across time, in cluttered scenes with occlusion, viewpoint change, multiple similar instances, and, in later versions, no-target cases [2308.08544, 2406.17005, 2604.26031].

## 1. Benchmark foundations and problem setting

The challenge is rooted in MeViS, introduced as a large-scale benchmark for video segmentation with motion expressions. MeViS was designed to shift language-guided video segmentation away from static referring cues such as color or category names and toward motion as the primary grounding signal. Its construction explicitly filtered videos to favor multiple objects and substantial motion, and its annotation protocol required motion-focused expressions while removing sentences for which the target could be identified from a single frame. In the original release, MeViS contains **2,006 videos**, **8,171 objects**, **28,570 motion expressions**, and **443k annotation masks**; it was built from public video segmentation datasets including **OVIS**, **UVO**, **TAOVOS**, and **MOSE** [2308.08544].

A central property of MeViS is that one expression need not map to exactly one object. The benchmark supports multi-object expressions, and the original paper reports **4.28 objects per video on average** and **1.59 objects per expression** on average. This makes the task structurally different from top-1 RVOS formulations that assume a single target trajectory. The benchmark paper also showed a large transfer gap from conventional RVOS datasets: **VLT** achieves **60.4 \(\mathcal{J}\mathcal{F}\)** on DAVIS\(_{17}\)-RVOS and **63.1** on Refer-Youtube-VOS but only **27.8** on MeViS, while **ReferFormer** drops to **31.0** on MeViS, indicating that performance on appearance-oriented RVOS does not carry over to motion-expression grounding [2308.08544].

The later multimodal extension, **MeViSv2**, retains the same **2,006 videos** and **8,171 objects** but expands the language side to **33,072 human-annotated motion expressions**, adding **4,502** more challenging expressions, including **motion reasoning expressions** and **no-target expressions**. Its reported breakdown is **21,541 single-target expressions**, **8,028 multi-target expressions**, **3,503 no-target expressions**, and **999 motion reasoning expressions**, along with audio expressions for all **33,072** sentences. This extension aligns closely with the 2026 MeViS-Text protocol, where no-target behavior becomes part of the official evaluation [2512.10945].

## 2. PVUW challenge evolution

Within PVUW, the MeViS track first appeared in 2024 as the **Motion Expression guided Video Segmentation track**. The 2024 challenge report describes it as a language-guided video segmentation benchmark in which the model is given a long video and a natural-language expression that emphasizes motion, and must segment the referred object across the video. The challenge was hosted on **CodaLab**, with private ground truth, public validation, and a short final phase. For MeViS in 2024, the report lists **225 registered teams**, **50** participants in validation, and **5** valid final submissions, underscoring both the difficulty of the task and the selectivity of the competition phase [2406.17005].

By 2025, the challenge report characterizes MeViS as a benchmark for language-guided segmentation in crowded and dynamic environments, with motion-centric expressions rather than static object attributes. The 2025 edition used the MeViS testing set with newly added videos and confidential ground truth, and the report records **77 teams registered** for MeViS and **31 teams submitted** in the testing phase. The same report highlights a methodological shift: top-performing systems increasingly combined multimodal large language models with segmentation foundation models such as **SAM2** [2504.11326].

In 2026, the track was explicitly presented as **MeViS-Text**, alongside **MOSE** and the new **MeViS-Audio** track. The 2026 report describes MeViS-Text as “text-based Motion Expression Video Segmentation,” where referring expressions explicitly describe motion patterns and temporal dynamics, and where the evaluation set includes **No-target samples**. The evaluation platform is reported as **CodaBench**, and the report emphasizes that the test set uses fresh video material and undisclosed annotations, preserving the benchmark’s role as a hidden-test evaluation of motion-aware grounding [2604.26031].

## 3. Evaluation protocol and official rankings

Across PVUW editions, MeViS evaluation centers on **\(\mathcal{J}\)** for region similarity and **\(\mathcal{F}\)** for contour accuracy, with **\(\mathcal{J}\mathcal{F}\)** as the standard segmentation summary metric. The 2026 MeViS-Text protocol extends this by adding **N-acc.** for no-target accuracy and **T-acc.** for target accuracy, and defines the official ranking score as
\[
\text{Final} = \frac{1}{3}\left(\text{mean } \mathcal{J}\mathcal{F} + \text{N-acc.} + \text{T-acc.}\right).
\]
This change is consequential because it prevents systems from optimizing only positive-case mask quality while ignoring null queries or target-presence errors [2604.00404, 2604.26031].

The leaderboard progression across the first three PVUW editions shows both continuity in benchmark focus and discontinuity in methodology. Scores are not directly comparable across all years, because the 2026 ranking adds no-target-sensitive metrics and uses a different final score definition.

| Edition | First place | Reported result |
|---|---|---|
| PVUW 2024 | Tapall.ai | **J&F = 0.5447** |
| PVUW 2025 | MVP-Lab | **61.98 \(\mathcal{J}\mathcal{F}\)** |
| PVUW 2026 | HITsz_Dragon | **Final score = 0.909064**, **\(\mathcal{J}\mathcal{F} = 0.7897\)** |

In 2024, the final MeViS leaderboard placed **Tapall.ai** first with **54.5 \(\mathcal{J}\mathcal{F}\)** in the challenge report, while the dedicated solution report gives **J&F = 0.5447**, **J = 0.5048**, and **F = 0.5846** for the winning submission. In 2025, the challenge report lists **MVP-Lab** first at **61.98**, **ReferDINO-Plus** second at **60.43**, and **Sa2VA** third at **56.26**. In 2026, the winning method reported **Final score: 0.909064** and **\(\mathcal{J}\mathcal{F}\): 0.7897**, ranking first overall on the **PVUW 2026 MeViS-Text test set** [2406.17005, 2406.07043, 2504.11326, 2604.00404].

## 4. Methodological trajectory from 2024 to 2025

The 2024 competition was dominated by adapted RVOS architectures rather than fully foundation-model-based pipelines. The first-place Tapall.ai solution built on **MUTR**, retained earlier “static-dominant” RVOS data such as **Ref-COCO**, **Ref-COCO+**, **Ref-COCOg**, and **Ref-YouTube-VOS**, and addressed long videos by splitting them into sub-videos and using frame sampling. Its ablations reported a clear gain from adding previous datasets, with **MUTR** improving from **0.4343** to **0.4857**, and showed that moderate sub-video lengths and sampling improved over no sampling. The second-place 2024 solution also used **MUTR**, but augmented it with **DVIS** instance masks for query initialization, **global sampling** rather than local sampling, and **HQ-SAM** for spatial refinement, reaching **54.20 \(\mathcal{J}\mathcal{F}\)** on the test phase. The third-place solution moved toward stronger cross-modal feature alignment by using a frozen **ConvNeXt-Large CLIP** backbone from **OpenCLIP**, adding repeated cross-modal attention and a Hungarian-matching-based video-query initializer, and achieved **51.5 \(\mathcal{J}\mathcal{F}\)** on the test set [2406.07043, 2406.13939, 2406.04842].

The 2025 competition marks a clear transition toward multimodal large models and segmentation foundation models. The first-place solution, “Unleashing the Potential of Large Multimodal Models for Referring Video Segmentation,” used **Sa2VA** as a baseline, replaced first-five-frame inference with **uniformly sampled** key frames across the entire video, and fused multiple expert models by **pixel-level binary mask voting**. On validation, uniform sampling with a **26B** model improved from **52.11** to **55.69 \(\mathcal{J}\mathcal{F}\)** with five key frames, and the best validation score was **58.06** at **20 sampled frames**; the final test result was **61.98 \(\mathcal{J}\mathcal{F}\)** [2504.05178].

Other 2025 methods refined this foundation-model template rather than departing from it. **ReferDINO-Plus** used a two-stage pipeline in which **ReferDINO** produced candidate masks and scores, the highest-scoring mask prompted **SAM2**, and a frame-wise **Conditional Mask Fusion** rule decided whether to trust SAM2 alone or combine it with ReferDINO. The reported test result was **60.43 \(\mathcal{J}\mathcal{F}\)** with **\(\mathcal{J} = 56.79\)** and **\(\mathcal{F} = 64.07\)**. The third-place **Sa2VA** report introduced **Long-Interleaved Inference (LII)**, replacing the default first five frames with key frames **1, 4, 7, 10, 13**, which improved **Sa2VA-26B** from **54.1** to **56.3 \(\mathcal{J}\mathcal{F}\)** without additional training; notably, an attempted **SAM-2 ensembling** strategy reduced performance rather than improving it [2503.23509, 2504.00476, 2504.11326].

## 5. The 2026 winning solution: strong MLLMs meet SAM3

The 2026 first-place method, “Strong MLLMs Meet SAM3 for Referring Video Object Segmentation,” formalized the shift from prompt-enhanced segmentation to an explicitly reasoning-driven, fully training-free pipeline. The paper identifies three practical weaknesses in prior RVOS systems: weaker open-source MLLMs can struggle with difficult motion-language reasoning, many prior approaches rely on **SAM2** rather than **SAM3**, and intermediate **boxes or special tokens** can discard fine-grained referential information before segmentation. Its response is a three-stage system that preserves language semantics, converts the event into more tractable grounding tasks, and delegates precise mask generation and tracking to SAM3 [2604.00404].

In **Stage 1**, **Gemini-3.1 Pro** decomposes the original motion-centric query into **instance-level grounding targets**, isolates the central subject from auxiliary referents, selects the frame where the target is **most clearly visible**, and generates a **discriminative description** for that specific instance. The paper frames this as converting a difficult **video motion-expression problem** into a set of **image grounding problems**. This is particularly important when a query depends on behavior over time, such as identifying which visually similar instance actually satisfies the event [2604.00404].

In **Stage 2**, the selected frame and refined text prompt are passed to **SAM3-agent** to produce a **pixel-accurate seed mask**, rather than a box or token proxy. The agent is described as a reasoning loop: a multimodal planner, again **Gemini-3.1 Pro**, repeatedly chooses among SAM3 tools based on the image, text prompt, and intermediate outputs until a satisfactory mask is obtained or the object is judged absent. The official **SAM3 tracker** then propagates the seed through the entire video **in both temporal directions**. For multiple valid instances, each instance is segmented and tracked independently, and masks are merged only after propagation [2604.00404].

In **Stage 3**, the system performs refinement with **Qwen3.5-Plus** and **behavior-level verification**. The method flags structurally unreliable cases such as **empty predictions** or **high overlap** between different targets, regenerates more precise descriptions, and reruns grounding. For **directional** or **negative** expressions, it samples frames, overlays or highlights mask boundaries, and asks an MLLM to judge whether the tracked object actually conforms to the original event; inconsistent predictions are sent back for another grounding pass. This creates a self-correcting loop oriented toward semantic consistency rather than geometry alone [2604.00404].

The reported outcome is first place on the **PVUW 2026 MeViS-Text test set** without task-specific fine-tuning, with a **Final score of 0.909064** and **\(\mathcal{J}\mathcal{F} = 0.7897\)**. The paper also states that the method did not achieve the absolute best **N-acc.** or **T-acc.** individually, but achieved the best overall balance and the best region/boundary quality, outperforming the second-place method by a substantial margin in **\(\mathcal{J}\mathcal{F}\)** [2604.00404].

## 6. Recurrent technical themes and unresolved issues

Across PVUW editions, several technical motifs recur. First, **temporal coverage** is repeatedly treated as a bottleneck. The 2024 winner used sub-video partitioning and sampling because temporal modules were trained on short pseudo-videos; the 2024 second-place method replaced local with global sampling; the 2025 Sa2VA winner uniformly sampled frames across the whole video; and the 2025 third-place Sa2VA report showed gains from distributing key frames across a longer temporal window. This convergence suggests that motion-expression grounding is highly sensitive to what part of the video is visible to the grounding module at inference time [2406.07043, 2406.13939, 2504.05178, 2504.00476].

Second, the benchmark increasingly rewards methods that **preserve language semantics until mask generation**. Earlier systems often used text to initialize queries or select a target trajectory, whereas later methods either projected special segmentation tokens into mask prompts, as in **Sa2VA**, or avoided proxy representations entirely by generating a seed mask directly from a discriminative description, as in the 2026 **SAM3-agent** pipeline. This suggests a shift from indirect language-to-mask transfer toward tighter semantic conditioning of the segmentation stage [2504.05178, 2504.00476, 2604.00404].

Third, **multi-object** and **no-target** reasoning have become central rather than peripheral. MeViS was designed from the outset to allow one expression to refer to several targets, and MeViSv2 adds no-target cases explicitly. The 2025 **ReferDINO-Plus** paper identifies SAM2’s tendency to collapse multi-object prompts into a single-object mask, motivating **Conditional Mask Fusion**. The 2026 challenge report further emphasizes **existence verification** and false-positive suppression in top systems, and the official 2026 metric integrates **N-acc.** and **T-acc.** into the ranking criterion [2308.08544, 2512.10945, 2503.23509, 2604.26031].

The challenge literature also makes clear that progress is accompanied by a growing reliance on **heuristics, agentic loops, and test-time orchestration**. Examples include area-ratio rules in **ReferDINO-Plus**, majority voting across expert models in the 2025 winner, key-frame schedules such as **1, 4, 7, 10, 13** in **LII**, and self-correcting verification loops in the 2026 winner. A plausible implication is that MeViS-Text remains insufficiently addressed by a single monolithic end-to-end architecture; current high-performing systems often decompose the task into grounding, propagation, and verification subproblems rather than solving it in one pass [2504.05178, 2503.23509, 2504.00476, 2604.00404].

The broader significance of PVUW MeViS-Text is therefore twofold. Empirically, it serves as a stress test for whether video-language systems can ground motion-focused referring expressions in long, cluttered, realistic videos rather than relying on static appearance priors. Methodologically, it has traced an arc from adapted RVOS transformers, through LMM-plus-SAM2 hybrids, to reasoning-centric MLLM-plus-SAM3 systems with explicit existence checking and semantic verification. The challenge reports state that future progress is likely to involve stronger multimodal LLM integration, better temporal grounding, more robust handling of null-target and multi-target queries, and continued expansion of MeViS-style evaluation with refreshed hidden test data and more diverse modalities [2406.17005, 2504.11326, 2604.26031].

Source: https://www.emergentmind.com/topics/pvuw-mevis-text-challenge