2nd of the 5th PVUW MeViS-Audio Track: ASR-SaSaSa2VA

Abstract: Audio-based video object segmentation aims to locate and segment objects in videos conditioned on audio cues, requiring precise understanding of both appearance and motion. Recent audio-driven video segmentation methods extend MLLMs by fusing audio and visual features for end-to-end localization. Despite their promise, these approaches are computationally intensive, struggle with aligning temporal audio cues to dynamic video content, and depend on large paired audio-video datasets. To address these challenges, we present ASR-SaSaSa2VA, a resource-efficient framework for audio-guided video segmentation. The key idea is to convert audio inputs into textual motion descriptions via automatic speech recognition (ASR) models and then leverage pre-trained text-based referring video segmentation models (e.g., SaSaSa2VA) for pixel-level predictions. To further enhance robustness, we incorporate a no-target expression detection module, implemented by a fine-tuned audio-based MLLM, which filters out audio clips that do not refer to any target object. This design allows the system to exploit strong pre-trained models while effectively handling ambiguous or irrelevant audio inputs. Our approach achieves a final score of 80.7 in the 5th PVUW Challenge (MeViS-v2-Audio track), earning the second-place ranking.

• The paper achieves a final score of 80.7 by converting audio motion cues to text and leveraging advanced text-video segmentation models.
• It introduces a modular pipeline combining Qwen3-ASR for transcription, SaSaSa2VA for segmentation, and Qwen2.5-Omni for filtering irrelevant audio.
• Experimental ablations confirm improved temporal modeling, reduced false positives, and efficient reuse of pre-trained components for robust segmentation.

ASR-SaSaSa2VA: Modular Audio-Driven Referring Video Object Segmentation

Task Setting and Challenges

The MeViS-v2-Audio track of the 5th PVUW Challenge presents a technically demanding scenario: pixel-level video object segmentation conditioned on audio descriptions of object motion. Unlike text-based referring expression segmentation, audio signals encode temporal and ambiguous motion cues, complicating direct cross-modal alignment. SOTA end-to-end approaches fuse audio and visual features with MLLM architectures, but these methods suffer from high computational demands, limited generalization due to scarce audio-video paired datasets, and difficulties with temporal cue alignment, especially for long or multi-object videos.

Framework Overview

ASR-SaSaSa2VA addresses these key challenges through a modular decomposition (Figure 1). Its pipeline consists of:

1. Audio-to-Text Conversion: Transcribes audio input to textual motion descriptions using Qwen3-ASR, converting complex audio temporal signals into explicit text prompts.
2. Text-Based Video Segmentation: Leverages powerful pre-trained text-video models (SaSaSa2VA) to achieve pixel-level segmentation guided by transcribed motion prompts.
3. No-target Expression Detection: Applies a fine-tuned audio-based MLLM (Qwen2.5-Omni) to filter out audio clips lacking valid referring expressions, improving robustness to ambiguous and irrelevant queries.

   Figure 1: ASR-SaSaSa2VA decomposes audio-driven video object segmentation into audio transcription, text-based segmentation, and no-target filtering.

Component Details

Automatic Speech Recognition

The first stage employs Qwen3-ASR, an efficient multilingual ASR backbone capable of high-fidelity transcription in diverse environments. This removes modality barriers, translating audio motion cues into structured textual instructions suitable for downstream segmentation.

Text-Based Referring Video Segmentation

ASR-SaSaSa2VA relies on SaSaSa2VA, an enhanced vision-language segmentation model. SaSaSa2VA extends Sa2VA with Key Frame Compression and multiple [SEG] tokens per clip, strengthening temporal modeling and granular prompt extraction. The MLLM generates context-aware token embeddings for each segment, which are used as implicit SAM2 prompts to generate accurate object masks throughout the video sequence.

No-target Expression Detection

Not all audio inputs correspond to valid targets. To maximize robustness, the framework integrates a no-target detection module utilizing Qwen2.5-Omni, fine-tuned for binary classification of audio-visual pairs. This module effectively rejects irrelevant or ambiguous audio queries, reducing spurious false positive segmentations.

Experimental Evaluation

ASR-SaSaSa2VA achieves a final score of 80.7, ranking second in the MeViS-v2-Audio track. It demonstrates strong $\mathcal{J{content}F}$, N-acc, and T-acc metrics, outperforming most participants and validating the modular design approach. Significant ablations are reported:

• ASR Backbone: Qwen3-ASR outperforms Fun-ASR in $\mathcal{J{content}F}$, confirming that high-quality audio transcription is critical for generating effective textual motion prompts.
• Model Scaling: Larger SaSaSa2VA models (26B) provide substantial gains in segmentation accuracy, underscoring the importance of temporal-context-aware models for aligning motion and appearance features.
• No-target Detection: The inclusion of no-target expression filtering markedly improves N-acc and reduces false positives, further boosting overall performance and robustness.

Practical and Theoretical Implications

The modular architecture enables highly efficient reuse of pre-trained models, sidestepping the bottleneck of end-to-end audio-visual training and the need for large-scale paired datasets. This result suggests future pipelines for multimodal video understanding tasks should explore modality conversion and plug-and-play leveraging of mature segmentation and LLMs. Furthermore, precise detection of irrelevant queries is essential for robust deployment in real-world settings, indicating a growing role for cross-modal reasoning modules.

Theoretically, transforming audio cues into textual representations circumvents modality mismatches, allowing established text-video models to address audio-guided segmentation with minimal adaptation. The architecture is extensible to other modalities (e.g., haptic or sensor input), provided sufficient modality conversion and context filtering.

Future Directions

Potential future research includes:

• Advancing modality conversion strategies for richer audio-to-text transcription, capturing nuanced temporal and spatial cues.
• Integrating prompt engineering for text-based segmentation models, dynamically adapting to various description complexities.
• Scaling cross-modal detection for multi-object and overlapping action scenarios, leveraging novel multimodal fusion architectures.
• Reducing computational overhead through lightweight, efficient segmenters and detection modules suitable for real-time deployment.

Conclusion

ASR-SaSaSa2VA demonstrates a resource-efficient and robust paradigm for audio-driven referring video object segmentation, combining ASR-based motion transcription, strong pre-trained text segmentation models, and rigorous filtering of irrelevant queries for competitive performance. This modular pipeline offers practical insights for future multimodal vision systems, highlighting the value of modality conversion and targeted cross-modal reasoning.