Multimodal Referring Segmentation

• Multimodal referring segmentation is the task of identifying precise regions in images, videos, or 3D scenes using language, audio, or other modality cues.
• It employs deep CNNs, vision–language transformers, and contrastive alignment techniques to fuse visual and textual information efficiently.
• The approach underpins applications in robotics, AR/VR, and video surveillance while driving research in scalable, real-time multimodal understanding.

Multimodal referring segmentation is a task that requires segmenting the precise regions of visual scenes—within images, video, audio-visual streams, or 3D environments—according to user- or system-supplied referring expressions, which may be formulated in text, speech, or other modalities. This research domain combines challenges in fine-grained recognition, cross-modal alignment, and robust perception under language-guided specification. The field has evolved rapidly, leveraging advances in deep convolutional networks, vision–language transformers, contrastive multimodal models, and large multimodal LLMs (MLLMs) to achieve robust segmentation and tracking across increasingly complex scenarios (Ding et al., 1 Aug 2025).

1. Formal Problem Definition and Task Variants

Let $V$ denote a visual input—either a 2D image, a clip of video frames, audio-visual video, or a 3D point cloud—and $E$ a referring expression, which can be text, speech, or audio. The core objective is to learn a function

$f:\; V\times E \to M$

where $M$ is a segmentation mask identifying the object(s) or region(s) described by the expression (Ding et al., 1 Aug 2025). The mask may be:

• 2D: For images, $M \in \{0,1\}^{H \times W}$
• Video: $M = \{M_t\}_{t=1}^T$, spatio-temporal sequence of masks
• Audio-visual: multimodal cues supplied as audio, image, or speech (Ying et al., 30 Jul 2025)
• 3D: Point-labels for 3D scenes, $M \in \{0,1\}^N$ for point clouds.

Recent research generalizes to Generalized Referring Expression Segmentation (GRES), in which the mask $M$ may cover zero, one, or multiple regions, accommodating the "one-to-many" and "one-to-zero" cases (Hu et al., 2023).

Common task configurations:

• Image-based: Predict mask in a single image based on a text (or audio) query
• Video-based (RVOS/MeViS): Segment and track the referred objects across frames, requiring consistent identity and temporal reasoning (Fang et al., 7 Apr 2025, Botach et al., 2021, Xiao et al., 2024)
• Audio-visual/Omnimodal: Integrate multiple modalities of expressions with visual and acoustic signals (Ying et al., 30 Jul 2025)
• 3D scene: Segment objects in point clouds according to language (Ding et al., 1 Aug 2025)

2. Meta-Architectures and Multimodal Fusion Strategies

Most competitive methods follow a unified architecture structured in four main stages (Ding et al., 1 Aug 2025):

1. Visual Encoder: Extract visual features.
   • 2D images: CNNs (e.g., ResNet, Swin) (Liu et al., 2023, Jain et al., 2021, Wei et al., 2023, Shi et al., 14 Mar 2025)
   • Videos: 3D-CNNs or vision transformers (e.g., Video Swin) (Botach et al., 2021)
   • 3D: PointNet++ or sparse ConvNets
2. Language/Audio Encoder: Embed the expression or auxiliary cues (text, audio, speech) (Wei et al., 2023, Zhang et al., 2023).
   • Text: BERT, RoBERTa, CLIP-Text (Lu et al., 16 Sep 2025, Zhang et al., 2023)
   • Audio: wav2vec, Whisper (Ying et al., 30 Jul 2025)
3. Cross-Modal Fusion:
   • Early and deep fusion: Transformer-based cross-attention (e.g., Synchronous Multi-Modal Fusion, mutual-aware attention) (Jain et al., 2021, Zhang et al., 2023, Liu et al., 2023, Wei et al., 2023, Lu et al., 16 Sep 2025)
   • Contrastive alignment: Position-aware modules and InfoNCE losses explicitly align text–vision pairs, often using bounding box priors (Chen et al., 2022)
   • Graph-based context modeling: Propagating information along linguistically-structured word graphs (Hui et al., 2020)
   • Prompt-based fusion: MLLM generates prompts (e.g., point, box) for foundation models like SAM (Chen et al., 2024)
4. Segmentation Head:
   • CNN-based decoding and upsampling (e.g., ASPP, FPN, ConvLSTM) (Jain et al., 2021, Yang et al., 2021, Hui et al., 2020)
   • Transformer-based decoders: mask tokens or query-based segmentation via cross-attention (Wei et al., 2023, Xiao et al., 2024, Fang et al., 7 Apr 2025)
   • Promptable: Decouple decision and mask extraction (e.g., SAM4MLLM, OISA) (Chen et al., 2024, Ying et al., 30 Jul 2025)

A canonical instantiation (image case) computes: $$M = \sigma(\mathsf{ConvUp}(\phi(F_{\mathrm{vis}},F_{\mathrm{txt}})))$$ or, in transformer-based settings: $$F_{\mathrm{fused}}^{(l+1)} = \mathrm{MultiHeadAttn}(F_{\mathrm{vis}}^{(l)}, F_{\mathrm{txt}})$$ with mask prediction via pixel-wise or token-to-image projection (Ding et al., 1 Aug 2025, Zhang et al., 2023).

3. Methodological Innovations and Design Principles

Multiscale and Progressive Fusion

• Cascaded Multi-modal Fusion (CMF): Stacks atrous/dilated convolutional fusion blocks, aligning vision and language at multiple scales during both contraction and expansion (Yang et al., 2021).
• Hierarchical Cross-Modal Aggregation (HCAM): Facilitates contextual information exchange across vision–language feature pyramids and hierarchies (Jain et al., 2021).
• Recurrent Multimodal Interaction (RMI): mLSTM-based word-by-word sequential modulation enables temporal accumulation of multimodal cues (Liu et al., 2017).

Deep Bidirectional and Structural Context Alignment

• Dual Multi-Modal Interaction (DMMI): Joint image-to-text and text-to-image decoders ensure cycle-consistency between visual and linguistic grounding, supporting one-to-one, one-to-many, and one-to-zero mappings (Hu et al., 2023).
• Linguistic Structure-Guided Context Modeling (LSCM): Explicitly injects dependency parsing–induced structural priors into multimodal feature propagation (Hui et al., 2020).

Position and Instance Awareness

• Position-Aware Contrastive Alignment: Incorporates bounding box or detector-predicted location priors for both feature alignment and contrastive discrimination, especially vital in cluttered scenes (Chen et al., 2022).
• Instance Mask Modality: Approaches such as MaIL introduce instance masks as a third input modality, enhancing instance sensitivity and mask fidelity (see (Li et al., 2021) abstract).

Promptable Segmentation and MLLM Integration

• SAM4MLLM and OISA: Bridge MLLMs and foundation segmentation models by asking the MLLM to generate discrete localization prompts (points, boxes) interpretable by SAM, supporting segmentation tasks with minimal architectural changes (Chen et al., 2024, Ying et al., 30 Jul 2025).
• Inference Decoupling: Multistage pipelines let text/MLLM generate guidance, while a strong visual decoder (often frozen) produces masks; enables scaling to new expression modalities or domains with minimal fine-tuning.

4. Datasets, Metrics, and Benchmark Results

Image-centric Benchmarks

• RefCOCO, RefCOCO+, RefCOCOg: MSCOCO-derived splits supporting spatial and appearance-based queries; typical mIoU for top image models exceeds 83% (Ding et al., 1 Aug 2025).
• ReferIt, G-Ref: Diverse objects ("things" and "stuff"), long queries; multi-modal and multi-instance evaluation (Jain et al., 2021, Hui et al., 2020).

Video & Audio-Visual Segmentation

• RVOS benchmarks: MeViS, Refer-YouTube-VOS, A2D-Sentences, JHMDB-Sentences; evaluate region similarity (J), contour accuracy (F), and composite J&F (Fang et al., 7 Apr 2025, Botach et al., 2021).
• OmniAVS: 2,104 videos, 61,095 expressions; eight expression types combining text, speech, sound, image (Ying et al., 30 Jul 2025). OISA achieves 41.1% J∧F overall.

Generalized/Multimodal and 3D Tasks

• GREx/gRefCOCO: Evaluate multi-object or absent-object (zero-shot) detection; use cumulative IoU, mean IoU, and "no-object" accuracy (Hu et al., 2023, Ding et al., 1 Aug 2025).
• 3D-RES: Point cloud segmentation with language grounding; ScanRefer, Multi3DRes.

Performance Highlights

Task/Benchmark	Top Result (2025–2026)	Noteworthy Model or Method
RefCOCO mIoU	~83%	OneRef-L, UNINEXT (Ding et al., 1 Aug 2025)
RefCOCO+ mIoU	~77%	OneRef-L, TFANet, LGFormer
G-Ref (Google)	>70%	TFANet (Lu et al., 16 Sep 2025)
MeViS (video)	61.98% J&F	MVP-Lab (Sa2VA+) (Fang et al., 7 Apr 2025)
OmniAVS (AV)	41.1% J&F	OISA-1B (Ying et al., 30 Jul 2025)
Ref-KITTI (video)	HOTA 46.0	TenRMOT (Xiao et al., 2024)
3D (ScanRefer)	[email protected] ~55–60%	IPDN, TGNN (Ding et al., 1 Aug 2025)

5. Practical Applications, Related Tasks, and Limitations

Applications

• Embodied AI and Robotics: Referring segmentation underpins fine-grained manipulation, grasping, and language-driven navigation (Ding et al., 1 Aug 2025).
• Human–Computer Interaction: Region-level, language-guided editing for AR/VR, assistive technologies, and collaborative annotation.
• Video Analysis and Surveillance: Temporal tracking of specified objects across frames for retrieval, monitoring, or event understanding.
• Remote Sensing: Language-guided segmentation of high-resolution aerial/satellite imagery, as in RRSIS-D (Shi et al., 14 Mar 2025).

Related Tasks

• Referring Expression Comprehension (REC): Bounding box localization (Ding et al., 1 Aug 2025).
• Phrase/Panoptic Grounding, Reasoning-based Segmentation: Extensions to open-vocabulary, multi-object, and inference-driven masks (Ying et al., 30 Jul 2025).

Limitations and Challenges

• Scalability: Heavyweight transformers and MLLMs limit real-time and mobile deployment; quantization and distillation are active directions (Ding et al., 1 Aug 2025, Lu et al., 16 Sep 2025).
• Domain Generalization: Adapting to non-standard scenes (medical, night, remote sensing) or noise-heavy modalities remains a major bottleneck.
• Semantic Loss/Alignment: Maintaining fine linguistic cues deep in fusion pipelines (mitigated via modules such as WFDM (Lu et al., 16 Sep 2025), LFR (Liu et al., 2023)).
• Zero-shot and Open-world: Reasoning with novel expressions or objects unseen during training is not yet robust.
• Ambiguous/Complex Queries: Handling expressions with nuanced reasoning, multi-step relationships, or ambiguous references (e.g., "the man who arrived last").

6. Research Trends and Future Directions

• Unifying Modalities: Movement toward architectures supporting joint text, speech, image, video, audio, and even tactile signals (omnimodal) (Ying et al., 30 Jul 2025).
• Promptable Foundation Models: Integrating prompt generation with frozen foundation models (e.g., SAM, Mask2Former) for task scalability (Chen et al., 2024, Ding et al., 1 Aug 2025).
• LLM Integration: Leveraging LLMs for high-level reasoning, multi-instruction, and dialog-based interactive segmentation (Ying et al., 30 Jul 2025, Chen et al., 2024).
• Multi-stage and Bidirectional Pipelines: Enforcing cycle consistency and deep cross-modal information flow (e.g., DMMI (Hu et al., 2023), RISAM (Zhang et al., 2023)).
• Scale/Speed/Domain Tradeoffs: Tuning networks for embedded/multi-task operation, real-time performance, and robustness to data distribution shifts (Ding et al., 1 Aug 2025, Lu et al., 16 Sep 2025).
• Benchmark Expansion: Datasets such as OmniAVS and gRefCOCO foreground expression diversity, complexity, and real-world grounding.

Research in multimodal referring segmentation continues to push boundaries in robust alignment, complex reasoning, and open-vocabulary generalization, with unified benchmarks and foundation-model-based paradigms anticipated to drive subsequent advances (Ding et al., 1 Aug 2025, Ying et al., 30 Jul 2025, Chen et al., 2024).