The 1st Winner for 5th PVUW MeViS-Text Challenge: Strong MLLMs Meet SAM3 for Referring Video Object Segmentation

Abstract: This report presents our winning solution to the 5th PVUW MeViS-Text Challenge. The track studies referring video object segmentation under motion-centric language expressions, where the model must jointly understand appearance, temporal behavior, and object interactions. To address this problem, we build a fully training-free pipeline that combines strong multimodal LLMs with SAM3. Our method contains three stages. First, Gemini-3.1 Pro decomposes each target event into instance-level grounding targets, selects the frame where the target is most clearly visible, and generates a discriminative description. Second, SAM3-agent produces a precise seed mask on the selected frame, and the official SAM3 tracker propagates the mask through the whole video. Third, a refinement stage uses Qwen3.5-Plus and behavior-level verification to correct ambiguous or semantically inconsistent predictions. Without task-specific fine-tuning, our method ranks first on the PVUW 2026 MeViS-Text test set, achieving a Final score of 0.909064 and a J&F score of 0.7897. The code is available at https://github.com/Moujuruo/MeViSv2_Track_Solution_2026.

• The paper introduces a training-free, modular pipeline that uses Gemini-3.1 Pro for event decomposition and SAM3 for precise mask generation.
• It achieves a top final score of 0.909064 and a leading JF score of 0.7897, demonstrating robust segmentation in dynamic scenes.
• The approach highlights full-cycle reasoning with iterative mask verification, effectively addressing occlusions and similar-instance challenges.

Training-free Referring Video Object Segmentation with MLLMs and SAM3: The HITsz_Dragon PVUW MeViS-Text Challenge Solution

Introduction

The task of referring video object segmentation (RVOS) under motion-centric language expressions requires joint modeling of spatial appearance, temporal dynamics, and intricate object interactions. Unlike traditional VOS, the semantic grounding here is defined by natural language referencing dynamic behavior, leading to increased ambiguity, especially in real-world settings with occlusions, viewpoint shifts, and numerous similar instances. The PVUW 2026 MeViS-Text Challenge further intensifies the problem by focusing on language referring to motion, interaction, and relational properties, demanding temporally compositional reasoning and robust model generalization.

This essay analyzes the HITsz_Dragon team's winning solution "The 1st Winner for 5th PVUW MeViS-Text Challenge: Strong MLLMs Meet SAM3 for Referring Video Object Segmentation" (2604.00404), which introduces a fully training-free and modular pipeline that integrates state-of-the-art multimodal LLMs (MLLMs) with the third generation Segment Anything Model (SAM3). The design strategically addresses key weaknesses in existing pipelines, such as inferior compositional reasoning from smaller MLLMs, information loss from box-based intermediate representations, and the necessity for post-hoc consistency refinement.

Methodology

Stage-wise Pipeline Architecture

The framework is composed of three sequential stages: instance-level event decomposition and description generation, discriminative mask grounding via SAM3-agent, and an iterative refinement/verification stage. Central to this architecture is the principle of maximal semantic preservation: linguistic granularity is explicitly maintained throughout segmentation and tracking by avoiding coarse intermediate structures.

Figure 1: End-to-end overview: Key-frame event decomposition, seed mask generation, video-level propagation, and semantic refinement.

Stage 1: Key-frame Reasoning and Event Decomposition

A strong MLLM (Gemini-3.1 Pro) analyzes each motion-referential query, decomposes events into instance-level targets, selects optimal frames with maximal target visibility, and generates highly discriminative per-instance descriptions. This explicit logical disentanglement disambiguates competing referents and enhances subsequent segmentation input specificity, clearly outperforming strategies reliant on direct or box-level prompts. Each event is recast as a sequence of grounded image-level problems, facilitating both clarity and modular processing in visually cluttered environments.

Stage 2: Direct Mask Grounding with SAM3-agent

Seed mask generation is performed by the SAM3-agent, with Gemini-3.1 Pro orchestrating segmentation through multi-step visual-linguistic reasoning. Crucially, intermediate representations retain full pixel-level precision, in contrast to prevalent methods that regress to bounding boxes or singular token prompting. The agent dynamically invokes SAM3's segmentation primitives conditioned on both the language prompt and ongoing results, iteratively refining the mask until target alignment is achieved or absence is declared. Once obtained, the SAM3 video tracker propagates the mask bidirectionally across the video, independently handling multiple instances before union at inference end. This pipeline enables semantic consistency and precision that box-based or heuristic agent strategies cannot robustly sustain.

Stage 3: Mask Reliability Refinement and Behavior Verification

Residual errors often emerge from insufficient description discrimination or failure to respect global semantic constraints (e.g., negative, relational, or directional content). The system introduces a self-refinement loop to address these – masks are systematically inspected for emptiness or significant overlap, which serve as triggers for linguistic description regeneration and agent restart (using Qwen3.5-Plus for enhanced textual precision). A secondary verification step, framed as a question-answering task to a MLLM over sampled video frames and predicted mask boundaries, enforces behavioral consistency with the source event expression. Only predictions passing this verification are retained, further enhancing result robustness and reducing semantic drift.

Figure 2: Qualitative samples: Instance-specific frame extraction, discriminative language descriptions, and high-fidelity mask prediction in dynamic scenes.

Experimental Results

Challenge and Metric Protocol

All evaluations were conducted under the constraint of no task-specific fine-tuning or model ensembling. The methods were benchmarked on the MeViSv2-Text track, which is grounded in complex videos with numerous challenges, including occlusion and dense distractors. The principal metrics are mean $\mathcal{J}\mathcal{F}$, capturing region and boundary segmentation performance, No-target accuracy (N-acc.), Target accuracy (T-acc.), and a Final composite score synthesized across these aspects.

Numerical Performance

This training-free approach achieved a Final score of 0.909064 and the top $\mathcal{J}\mathcal{F}$ score of 0.7897, leading the nearest competitor by +0.0791 absolute. While not maximally optimal in N-acc. or T-acc. individually, the holistic centering of semantic information in mask propagation produced a method that consistently delivers across all benchmarks, resulting in the highest aggregate score.

Theoretical and Practical Implications

The results substantiate three key points. First, leveraging state-of-the-art MLLMs for both input decomposition and mask generation planning substantially improves disambiguation and fine-grained referent identification, even under the zero-shot constraint. Second, the direct integration with SAM3 and avoidance of information-reducing representations (e.g., boxes, special tokens) preserves discriminative details necessary for challenging multi-instance scenes. Third, the introduction of a self-refining verification loop using strong LLMs highlights the necessity of full-cycle reasoning and semantic alignment in high-level VOS tasks.

These outcomes suggest that training-free, modular pipelines – when built on sufficiently strong foundation models – are becoming increasingly competitive for applied and theoretical RVOS. This holds practical significance for real-world scenarios where labeled data is scarce or generalization is prioritized over adaptation cost.

Future Directions

Moving forward, the integration of autonomous event decomposition, multi-modal memory, and iterative feedback can be further investigated in both the RVOS and broader video-language grounding domains. Attention to the interpretability of multi-agent reasoning strategies and more robust evaluation of failure modes under rapidly evolving motion-centric expressions remain open challenges. Additionally, with continual improvements in MLLM compositionality and segmentation accuracy, architectures eschewing task-specific training may become standard for a variety of dense vision-language tasks.

Conclusion

This essay has rigorously summarized and contextualized the HITsz_Dragon team's training-free solution to motion-centric RVOS for the PVUW 2026 MeViS-Text Challenge (2604.00404). The pipeline demonstrates that careful integration of advanced MLLMs for event decomposition, SAM3-based direct mask grounding, and proactive semantic verification strategies yield state-of-the-art results without recourse to task-tuned adaptation. The approach provides a compelling baseline and reference for continued research into efficient, generalizable video-language understanding systems.