PVUW MeViS-Text Challenge

Updated 5 July 2026

PVUW MeViS-Text Challenge is a benchmark that evaluates text-based video segmentation using natural language queries emphasizing motion and temporal dynamics.
It is built on the MeViS benchmark, which filters for multiple objects and substantial motion, challenging systems to aggregate evidence across time in realistic scenes.
The challenge has evolved from static RVOS methods to advanced MLLM-SAM pipelines that incorporate semantic verification, multi-object reasoning, and no-target handling.

PVUW MeViS-Text Challenge is the PVUW track for motion-expression-guided, text-based video object segmentation, built on the MeViS benchmark. In this setting, a system receives a video and a natural-language query that emphasizes motion, temporal dynamics, action, interaction, relative position, exclusion, or temporal behavior, and must localize and segment the referred object or objects throughout the video. The track is a specialized form of referring video object segmentation (RVOS), but it differs from appearance-dominant RVOS benchmarks in that the referent often cannot be identified from a single frame alone; correct grounding may depend on evidence distributed across time, in cluttered scenes with occlusion, viewpoint change, multiple similar instances, and, in later versions, no-target cases (Ding et al., 2023, Ding et al., 2024, Liu et al., 28 Apr 2026).

1. Benchmark foundations and problem setting

The challenge is rooted in MeViS, introduced as a large-scale benchmark for video segmentation with motion expressions. MeViS was designed to shift language-guided video segmentation away from static referring cues such as color or category names and toward motion as the primary grounding signal. Its construction explicitly filtered videos to favor multiple objects and substantial motion, and its annotation protocol required motion-focused expressions while removing sentences for which the target could be identified from a single frame. In the original release, MeViS contains 2,006 videos, 8,171 objects, 28,570 motion expressions, and 443k annotation masks; it was built from public video segmentation datasets including OVIS, UVO, TAOVOS, and MOSE (Ding et al., 2023).

A central property of MeViS is that one expression need not map to exactly one object. The benchmark supports multi-object expressions, and the original paper reports 4.28 objects per video on average and 1.59 objects per expression on average. This makes the task structurally different from top-1 RVOS formulations that assume a single target trajectory. The benchmark paper also showed a large transfer gap from conventional RVOS datasets: VLT achieves 60.4 $\mathcal{J}\mathcal{F}$ on DAVIS $_{17}$ -RVOS and 63.1 on Refer-Youtube-VOS but only 27.8 on MeViS, while ReferFormer drops to 31.0 on MeViS, indicating that performance on appearance-oriented RVOS does not carry over to motion-expression grounding (Ding et al., 2023).

The later multimodal extension, MeViSv2, retains the same 2,006 videos and 8,171 objects but expands the language side to 33,072 human-annotated motion expressions, adding 4,502 more challenging expressions, including motion reasoning expressions and no-target expressions. Its reported breakdown is 21,541 single-target expressions, 8,028 multi-target expressions, 3,503 no-target expressions, and 999 motion reasoning expressions, along with audio expressions for all 33,072 sentences. This extension aligns closely with the 2026 MeViS-Text protocol, where no-target behavior becomes part of the official evaluation (Ding et al., 11 Dec 2025).

2. PVUW challenge evolution

Within PVUW, the MeViS track first appeared in 2024 as the Motion Expression guided Video Segmentation track. The 2024 challenge report describes it as a language-guided video segmentation benchmark in which the model is given a long video and a natural-language expression that emphasizes motion, and must segment the referred object across the video. The challenge was hosted on CodaLab, with private ground truth, public validation, and a short final phase. For MeViS in 2024, the report lists 225 registered teams, 50 participants in validation, and 5 valid final submissions, underscoring both the difficulty of the task and the selectivity of the competition phase (Ding et al., 2024).

By 2025, the challenge report characterizes MeViS as a benchmark for language-guided segmentation in crowded and dynamic environments, with motion-centric expressions rather than static object attributes. The 2025 edition used the MeViS testing set with newly added videos and confidential ground truth, and the report records 77 teams registered for MeViS and 31 teams submitted in the testing phase. The same report highlights a methodological shift: top-performing systems increasingly combined multimodal LLMs with segmentation foundation models such as SAM2 (Ding et al., 15 Apr 2025).

In 2026, the track was explicitly presented as MeViS-Text, alongside MOSE and the new MeViS-Audio track. The 2026 report describes MeViS-Text as “text-based Motion Expression Video Segmentation,” where referring expressions explicitly describe motion patterns and temporal dynamics, and where the evaluation set includes No-target samples. The evaluation platform is reported as CodaBench, and the report emphasizes that the test set uses fresh video material and undisclosed annotations, preserving the benchmark’s role as a hidden-test evaluation of motion-aware grounding (Liu et al., 28 Apr 2026).

3. Evaluation protocol and official rankings

Across PVUW editions, MeViS evaluation centers on $\mathcal{J}$ for region similarity and $\mathcal{F}$ for contour accuracy, with $\mathcal{J}\mathcal{F}$ as the standard segmentation summary metric. The 2026 MeViS-Text protocol extends this by adding N-acc. for no-target accuracy and T-acc. for target accuracy, and defines the official ranking score as

$\text{Final} = \frac{1}{3}\left(\text{mean } \mathcal{J}\mathcal{F} + \text{N-acc.} + \text{T-acc.}\right).$

This change is consequential because it prevents systems from optimizing only positive-case mask quality while ignoring null queries or target-presence errors (He et al., 1 Apr 2026, Liu et al., 28 Apr 2026).

The leaderboard progression across the first three PVUW editions shows both continuity in benchmark focus and discontinuity in methodology. Scores are not directly comparable across all years, because the 2026 ranking adds no-target-sensitive metrics and uses a different final score definition.

Edition	First place	Reported result
PVUW 2024	Tapall.ai	J&F = 0.5447
PVUW 2025	MVP-Lab	61.98 $\mathcal{J}\mathcal{F}$
PVUW 2026	HITsz_Dragon	Final score = 0.909064, $\mathcal{J}\mathcal{F} = 0.7897$

In 2024, the final MeViS leaderboard placed Tapall.ai first with 54.5 $\mathcal{J}\mathcal{F}$ in the challenge report, while the dedicated solution report gives J&F = 0.5447, J = 0.5048, and F = 0.5846 for the winning submission. In 2025, the challenge report lists MVP-Lab first at 61.98, ReferDINO-Plus second at 60.43, and Sa2VA third at 56.26. In 2026, the winning method reported Final score: 0.909064 and $\mathcal{J}\mathcal{F}$ : 0.7897, ranking first overall on the PVUW 2026 MeViS-Text test set (Ding et al., 2024, Gao et al., 2024, Ding et al., 15 Apr 2025, He et al., 1 Apr 2026).

4. Methodological trajectory from 2024 to 2025

The 2024 competition was dominated by adapted RVOS architectures rather than fully foundation-model-based pipelines. The first-place Tapall.ai solution built on MUTR, retained earlier “static-dominant” RVOS data such as Ref-COCO, Ref-COCO+, Ref-COCOg, and Ref-YouTube-VOS, and addressed long videos by splitting them into sub-videos and using frame sampling. Its ablations reported a clear gain from adding previous datasets, with MUTR improving from 0.4343 to 0.4857, and showed that moderate sub-video lengths and sampling improved over no sampling. The second-place 2024 solution also used MUTR, but augmented it with DVIS instance masks for query initialization, global sampling rather than local sampling, and HQ-SAM for spatial refinement, reaching 54.20 $_{17}$ 0 on the test phase. The third-place solution moved toward stronger cross-modal feature alignment by using a frozen ConvNeXt-Large CLIP backbone from OpenCLIP, adding repeated cross-modal attention and a Hungarian-matching-based video-query initializer, and achieved 51.5 $_{17}$ 1 on the test set (Gao et al., 2024, Cao et al., 2024, Pan et al., 2024).

The 2025 competition marks a clear transition toward multimodal large models and segmentation foundation models. The first-place solution, “Unleashing the Potential of Large Multimodal Models for Referring Video Segmentation,” used Sa2VA as a baseline, replaced first-five-frame inference with uniformly sampled key frames across the entire video, and fused multiple expert models by pixel-level binary mask voting. On validation, uniform sampling with a 26B model improved from 52.11 to 55.69 $_{17}$ 2 with five key frames, and the best validation score was 58.06 at 20 sampled frames; the final test result was 61.98 $_{17}$ 3 (Fang et al., 7 Apr 2025).

Other 2025 methods refined this foundation-model template rather than departing from it. ReferDINO-Plus used a two-stage pipeline in which ReferDINO produced candidate masks and scores, the highest-scoring mask prompted SAM2, and a frame-wise Conditional Mask Fusion rule decided whether to trust SAM2 alone or combine it with ReferDINO. The reported test result was 60.43 $_{17}$ 4 with $_{17}$ 5 and $_{17}$ 6. The third-place Sa2VA report introduced Long-Interleaved Inference (LII), replacing the default first five frames with key frames 1, 4, 7, 10, 13, which improved Sa2VA-26B from 54.1 to 56.3 $_{17}$ 7 without additional training; notably, an attempted SAM-2 ensembling strategy reduced performance rather than improving it (Liang et al., 30 Mar 2025, Yuan et al., 1 Apr 2025, Ding et al., 15 Apr 2025).

5. The 2026 winning solution: strong MLLMs meet SAM3

The 2026 first-place method, “Strong MLLMs Meet SAM3 for Referring Video Object Segmentation,” formalized the shift from prompt-enhanced segmentation to an explicitly reasoning-driven, fully training-free pipeline. The paper identifies three practical weaknesses in prior RVOS systems: weaker open-source MLLMs can struggle with difficult motion-language reasoning, many prior approaches rely on SAM2 rather than SAM3, and intermediate boxes or special tokens can discard fine-grained referential information before segmentation. Its response is a three-stage system that preserves language semantics, converts the event into more tractable grounding tasks, and delegates precise mask generation and tracking to SAM3 (He et al., 1 Apr 2026).

In Stage 1, Gemini-3.1 Pro decomposes the original motion-centric query into instance-level grounding targets, isolates the central subject from auxiliary referents, selects the frame where the target is most clearly visible, and generates a discriminative description for that specific instance. The paper frames this as converting a difficult video motion-expression problem into a set of image grounding problems. This is particularly important when a query depends on behavior over time, such as identifying which visually similar instance actually satisfies the event (He et al., 1 Apr 2026).

In Stage 2, the selected frame and refined text prompt are passed to SAM3-agent to produce a pixel-accurate seed mask, rather than a box or token proxy. The agent is described as a reasoning loop: a multimodal planner, again Gemini-3.1 Pro, repeatedly chooses among SAM3 tools based on the image, text prompt, and intermediate outputs until a satisfactory mask is obtained or the object is judged absent. The official SAM3 tracker then propagates the seed through the entire video in both temporal directions. For multiple valid instances, each instance is segmented and tracked independently, and masks are merged only after propagation (He et al., 1 Apr 2026).

In Stage 3, the system performs refinement with Qwen3.5-Plus and behavior-level verification. The method flags structurally unreliable cases such as empty predictions or high overlap between different targets, regenerates more precise descriptions, and reruns grounding. For directional or negative expressions, it samples frames, overlays or highlights mask boundaries, and asks an MLLM to judge whether the tracked object actually conforms to the original event; inconsistent predictions are sent back for another grounding pass. This creates a self-correcting loop oriented toward semantic consistency rather than geometry alone (He et al., 1 Apr 2026).

The reported outcome is first place on the PVUW 2026 MeViS-Text test set without task-specific fine-tuning, with a Final score of 0.909064 and $_{17}$ 8. The paper also states that the method did not achieve the absolute best N-acc. or T-acc. individually, but achieved the best overall balance and the best region/boundary quality, outperforming the second-place method by a substantial margin in $_{17}$ 9 (He et al., 1 Apr 2026).

6. Recurrent technical themes and unresolved issues

Across PVUW editions, several technical motifs recur. First, temporal coverage is repeatedly treated as a bottleneck. The 2024 winner used sub-video partitioning and sampling because temporal modules were trained on short pseudo-videos; the 2024 second-place method replaced local with global sampling; the 2025 Sa2VA winner uniformly sampled frames across the whole video; and the 2025 third-place Sa2VA report showed gains from distributing key frames across a longer temporal window. This convergence suggests that motion-expression grounding is highly sensitive to what part of the video is visible to the grounding module at inference time (Gao et al., 2024, Cao et al., 2024, Fang et al., 7 Apr 2025, Yuan et al., 1 Apr 2025).

Second, the benchmark increasingly rewards methods that preserve language semantics until mask generation. Earlier systems often used text to initialize queries or select a target trajectory, whereas later methods either projected special segmentation tokens into mask prompts, as in Sa2VA, or avoided proxy representations entirely by generating a seed mask directly from a discriminative description, as in the 2026 SAM3-agent pipeline. This suggests a shift from indirect language-to-mask transfer toward tighter semantic conditioning of the segmentation stage (Fang et al., 7 Apr 2025, Yuan et al., 1 Apr 2025, He et al., 1 Apr 2026).

Third, multi-object and no-target reasoning have become central rather than peripheral. MeViS was designed from the outset to allow one expression to refer to several targets, and MeViSv2 adds no-target cases explicitly. The 2025 ReferDINO-Plus paper identifies SAM2’s tendency to collapse multi-object prompts into a single-object mask, motivating Conditional Mask Fusion. The 2026 challenge report further emphasizes existence verification and false-positive suppression in top systems, and the official 2026 metric integrates N-acc. and T-acc. into the ranking criterion (Ding et al., 2023, Ding et al., 11 Dec 2025, Liang et al., 30 Mar 2025, Liu et al., 28 Apr 2026).

The challenge literature also makes clear that progress is accompanied by a growing reliance on heuristics, agentic loops, and test-time orchestration. Examples include area-ratio rules in ReferDINO-Plus, majority voting across expert models in the 2025 winner, key-frame schedules such as 1, 4, 7, 10, 13 in LII, and self-correcting verification loops in the 2026 winner. A plausible implication is that MeViS-Text remains insufficiently addressed by a single monolithic end-to-end architecture; current high-performing systems often decompose the task into grounding, propagation, and verification subproblems rather than solving it in one pass (Fang et al., 7 Apr 2025, Liang et al., 30 Mar 2025, Yuan et al., 1 Apr 2025, He et al., 1 Apr 2026).

The broader significance of PVUW MeViS-Text is therefore twofold. Empirically, it serves as a stress test for whether video-language systems can ground motion-focused referring expressions in long, cluttered, realistic videos rather than relying on static appearance priors. Methodologically, it has traced an arc from adapted RVOS transformers, through LMM-plus-SAM2 hybrids, to reasoning-centric MLLM-plus-SAM3 systems with explicit existence checking and semantic verification. The challenge reports state that future progress is likely to involve stronger multimodal LLM integration, better temporal grounding, more robust handling of null-target and multi-target queries, and continued expansion of MeViS-style evaluation with refreshed hidden test data and more diverse modalities (Ding et al., 2024, Ding et al., 15 Apr 2025, Liu et al., 28 Apr 2026).