OmniAVS: Omnimodal Referring Audio-Visual Segmentation

Updated 3 July 2026

OmniAVS is a task that integrates video, audio, and various omnimodal prompts to achieve pixel-accurate segmentation with robust temporal and semantic reasoning.
State-of-the-art architectures employ unified feature fusion, explicit reasoning, and adaptive modality suppression to handle diverse, reasoning-intensive scenarios.
Key challenges include null/distractor disambiguation and temporal coherence, while future directions focus on end-to-end multimodal reasoning and scalable sensor integration.

Omnimodal Referring Audio-Visual Segmentation (OmniAVS) generalizes the referring segmentation paradigm by requiring pixel-accurate segmentation of target objects in video based on omnimodal prompts that can include combinations of free-form text, speech, audio, sound cues, and image examples. This task necessitates explicit integration of video, audio, and prompt streams, demanding cross-modal semantic grounding and robust temporal reasoning. OmniAVS extends beyond classic multimodal and referring audio-visual segmentation by supporting open-world, cross-modality, and reasoning-intensive scenarios, as codified in benchmarks such as OmniAVS and evaluated by mask quality (IoU and contour F), generalization across referring split types, and robustness to distractors and composite instructions (Ying et al., 30 Jul 2025, Ding et al., 1 Aug 2025).

1. Task Definition and Problem Formalization

The core objective of OmniAVS is to learn a mapping

$(V,\,A,\,E) \rightarrow \{M_t\}_{t=1}^T$

where $V = \{V_t\}$ is a sequence of video frames, $A = \{A_t\}$ is the synchronized audio stream, and $E$ is the omnimodal referring prompt, flexibly comprised of text, speech, audio event, or image cues (Ding et al., 1 Aug 2025, Ying et al., 30 Jul 2025). The model must predict temporally-aligned binary mask sequences $M_t \in \{0,1\}^{H\times W}$ , such that each mask isolates the visual object(s) uniquely identified by the prompt at time $t$ .

Prompts $E$ may be of eight combinatorial types (text, speech, sound, image, and their unions), with inference required on both semantic content (e.g., causality, function, compositionality) and low-level multimodal cues (e.g., onset, timbre, spatial position) (Ying et al., 30 Jul 2025). Models must operate beyond direct surface-level matching, supporting expressions with explicit reasoning (e.g., "the one who coughed, which indicates sickness"), ambiguous attributes, and null targets (no referred object present).

2. Benchmarks and Data Resources

The principal OmniAVS dataset comprises 2,104 videos and 61,095 multimodal referring expressions, structured to stress test omnimodal understanding (Ying et al., 30 Jul 2025). Ref-AVS (Wang et al., 2024) (4,002 videos, 20,261 expressions) is earlier but includes only multimodal-cue-enriched text and audio. Benchmarks provide per-frame pixel-level masks, diverse expression modalities, and multi-split evaluation (seen/unseen categories, null prompts).

Benchmark characteristics:

Benchmark	#Vids	#Expressions	Modalities Supported	Reasoning Complexity
OmniAVS	2,104	61,095	Text, speech, sound, image (8 types)	High (world knowledge, compositional, causal)
Ref-AVS	4,002	20,261	Text+audio (no image/speech prompt)	Moderate

OmniAVS annotates not only mask tracks, but also detailed referring explanations for reasoning-intensive expressions (Ying et al., 30 Jul 2025). The Ref-AVS R²-AVSBench extension introduces longer, indirect references requiring multi-hop inference and abstract world knowledge (Zhou et al., 6 Aug 2025). Evaluation uses region similarity ( $\mathcal{J}$ ), contour similarity ( $\mathcal{F}$ ), and explanation quality metrics (e.g., METEOR) (Ying et al., 30 Jul 2025).

3. Meta-Architectures and Methodological Advances

State-of-the-art OmniAVS systems employ four main classes of architecture:

Unified Feature Fusion Pipelines: The canonical pipeline consists of specialized encoders for visual ( $\Phi_v$ ), audio ( $V = \{V_t\}$ 0), and omnimodal prompt ( $V = \{V_t\}$ 1) inputs, cross-modal fusion (e.g., attention or interleaving), followed by a mask decoder (typically Mask2Former, SAM2, or custom heads) (Ding et al., 1 Aug 2025, Ying et al., 30 Jul 2025).
MLLM-Based Reasoning and Query Propagation: Omnimodal Instructed Segmentation Assistant (OISA) uses a multimodal LLM to generate a [SEG] token that semantically summarizes the prompt and context. This token or its propagated variant guides mask queries through the temporal video (Ying et al., 30 Jul 2025). Audio-Visual Interleaving, as in OISA, improves alignment of temporally synchronized content.
Explicit Reasoning Decomposition: TGS-Agent implements a three-stage Think-Ground-Segment pipeline: (i) textual/audio/visual reasoning over inputs to derive fine-grained object descriptions, (ii) open-vocabulary detector grounding (e.g., Grounding-DINO), and (iii) prompt-conditioned segmentation decoding (SAM2) (Zhou et al., 6 Aug 2025). This decouples semantic reference understanding from the low-level mask prediction.
Adaptive Modality Suppression and Biased Competition: PRIMED introduces a modality prior decoder that learns when to suppress or prioritize audio/visual/text cues. Fusion is modulated by this prior, using global context distilled via token aggregation and competition-aware cross-modal fusion (He et al., 8 May 2026). Foreground-background discrimination is reinforced by spatial-aware semantic alignment loss.

Loss functions universally include pixel-wise binary cross-entropy ( $V = \{V_t\}$ 2), Dice overlap ( $V = \{V_t\}$ 3), and in some settings, cross-modal alignment or contrastive objectives to encourage semantic grounding (Ding et al., 1 Aug 2025, He et al., 8 May 2026).

4. Experimental Results and Comparative Analysis

System performance is reported across multiple splits, aggregates, and ablations:

Model	Benchmark	(J+F)/2 (%)	Explanation METEOR	Notable Techniques
OISA-1B	OmniAVS	41.1	21.7	Audio-Visual Interleaving, MLLM [SEG], Query Prop
PRIMED	Ref-AVS	70.9 (mix)	—	Modality prior, global context distillation
Omni-R1-7B	Ref-AVS	47.2 (seen)	—	RL-based keyframe selection and task rewriting
TGS-Agent	Ref-AVSBench	65.9 (mix)	—	Explicit reasoning (Think-Ground-Segment)

Ablation studies in OISA and PRIMED demonstrate that omnimodal cues, when adaptively fused, yield substantial improvements over uni-modal or naively fused baselines. For instance, audio-visual interleaving outperforms classic cross-attention (+3.4 $V = \{V_t\}$ 4), and adaptive suppression (PRIMED) further enhances unseen class generalization (Ying et al., 30 Jul 2025, He et al., 8 May 2026).

Notably, explicit reasoning modules (as in TGS-Agent) provide resilience on linguistically and semantically diverse expressions (R²-AVSBench), with performance drops less severe than blackbox fusion architectures (Zhou et al., 6 Aug 2025).

5. Challenges, Limitations, and Open Problems

OmniAVS remains challenged by several factors:

Null and Distractor Disambiguation: Models exhibit hallucination when the prompt targets objects absent from the scene or when distractor sound/visual events are present. Specialized losses and prior estimation (e.g., PRIMED’s modality prior or OISA’s null-split evaluation protocol) partially address, but do not eliminate, this failure mode (Ying et al., 30 Jul 2025, He et al., 8 May 2026).
Temporal and Semantic Coherence: Accurately tracking objects through occlusions, reappearing entities, and events with lagged or concurrent cues requires robust temporal modeling; naïve framewise or prompt-centric approaches underperform on long videos (Ying et al., 30 Jul 2025, Zhou et al., 6 Aug 2025).
Multimodal Reasoning and Explanation: Real-world deployment necessitates not only correct mask prediction, but also semantic justification (e.g., “this mask matches the person who coughed, consistent with sickness”). Only a subset of models (OISA, TGS-Agent) produce textual explanations aligned to masks (Ying et al., 30 Jul 2025, Zhou et al., 6 Aug 2025).
Scalability to Additional Modalities: Extension to depth, thermal, or haptic sensors demands further architectural generalization, particularly in the modality prior and cross-modal fusion design (He et al., 8 May 2026, Ding et al., 1 Aug 2025).

6. Future Directions and Prospects

Research is converging on several promising fronts:

Explicit Reasoning Integration: End-to-end joint training of MLLM-based reasoning with mask head modules is anticipated to yield better cross-modal alignment and interpretable outputs (Zhou et al., 6 Aug 2025, Ying et al., 30 Jul 2025).
Scalable Modality Priors: Further development of hierarchical or learned modality priors—potentially incorporating curriculum schedules for ramping in new sensors—can provide robust suppression of noise and improved generalization to real-world conditions (e.g., audio-visual-depth fusion) (He et al., 8 May 2026).
Foundation RL Methods: RL-based selection frameworks, such as Omni-R1, have demonstrated improved out-of-domain generalization and reduced hallucination, suggesting an avenue for scalable universal OmniAVS agents (Zhong et al., 26 May 2025).
Efficient Cross-Modal Attention: Investigating light-weight adapters and early-exit pipelines for real-time processing without sacrificing contextual understanding is of interest for deployable systems (Ying et al., 30 Jul 2025, Ding et al., 1 Aug 2025).
Rich Annotation and Multi-Turn Interaction: Datasets with full per-frame explanation, conversational prompts, and ambiguous/multiple-object references are under development to stress reasoning under uncertainty and interaction (Ying et al., 30 Jul 2025).

OmniAVS, via its comprehensive benchmark protocol, extensible omnimodal prompt structure, and an evolving ecosystem of architectures, sets the foundation for advancing segment-anything agents capable of instruction-aware, explainable, and robust segmentation in real-world multisensor environments (Ding et al., 1 Aug 2025).