Omnimodal Instructed Segmentation Assistant (OISA)

• OISA is a unified segmentation framework that integrates diverse data modalities and natural language instructions to produce instance- or region-level masks.
• It employs a novel set-structured reasoning-to-instance interface with hybrid attention, ensuring efficient and coherent instance predictions.
• OISA utilizes modality-specific encoders and dynamic cross-modal fusion strategies, enabling robust segmentation performance across images, videos, audio, and 3D data.

An Omnimodal Instructed Segmentation Assistant (OISA) is a unified system for segmentation across multiple input modalities, capable of following arbitrary, open-form natural language instructions and reasoning over multi-modal sensory inputs to generate instance- or region-level segmentation outputs. OISA represents the convergence of advances in instruction-driven segmentation, multimodal LLMs (MLLMs), differentiable query interfaces, and agentic or function-calling frameworks for modular task orchestration. The OISA paradigm extends segmentation from static 2D images to video, 3D volumes, audio-visual sequences, and multi-sensor data, integrating reasoning, temporal context, and multimodal grounding (Yuan et al., 25 May 2026, Chen et al., 2024, Ying et al., 30 Jul 2025, Zhong et al., 26 May 2025, Zheng et al., 7 Dec 2025, Li et al., 4 Dec 2025).

1. Core Problem Definition and Objectives

OISA formalizes segmentation as the task of predicting one or more masks $$M = \{M_i\}_{i=1}^N,\, M_i \in \{0,1\}^{H \times W}$$ or per-frame/per-modality generalizations, conditioned on an input $X$ comprising data from any combination of the following:

• RGB images or frames, $I_t \in \mathbb{R}^{H\times W\times 3}$
• Videos or temporal sequences, $V = \{I_1, ..., I_N\}$
• Audio streams $A$ or speech/text utterances
• Expression tokens $E$ combining text, speech, sound, image examples, or sketches
• Reference images with spatial prompts (masks, boxes, scribbles)
• Depth, 3D point clouds, or volumetric scans

OISA is instructed via rich, unconstrained language or multimodal expressions $E$, supporting compositional queries, temporal and spatial reasoning, and corrective feedback. The system must return segmentation masks and, in some architectures, natural-language explanations of the reasoning process (Ying et al., 30 Jul 2025, Li et al., 4 Dec 2025).

OISA must generalize across:

• Input modalities (images, video, audio, 3D, etc.)
• Instruction modalities (text, speech, exemplars, reference masks)
• Task settings (single/multi-object, panoptic vs. instance segmentation, semantic vs. referring)
• Output forms (masks, region IDs, explanations, or function calls)

2. Unified Architectural Principles

Set-Structured Reasoning-to-Instance Interface

A fundamental architectural innovation for OISA is the set-structured reasoning-to-instance query interface, exemplified by the InstructSAM approach. Here, segmentation is formulated as a set prediction problem: a bank of learnable instance queries (slots) is contextualized by both the instruction and multimodal input tokens, and each query slot is refined through hybrid-attention across the multimodal feature space. This design enables explicit slot interaction that promotes coherent instance enumeration and reduces duplicate predictions. Final queries are projected into a decoder (e.g. SAM3 mask decoder) for mask generation (Yuan et al., 25 May 2026).

Hybrid Attention and Query Bank

OISA architectures incorporate custom attention masks. Text tokens follow causal attention for autoregressive language reasoning, while instance query tokens have full bidirectional attention with visual and instruction tokens, facilitating tight slot-based multimodal coordination. Query banks can be scaled (increased $K$) for capacity, with a trade-off in inference efficiency (Yuan et al., 25 May 2026).

Modality Integration and Multimodal Prompt Encoding

To support omnimodal scenarios, OISA systems leverage:

• Modality-specific encoders (e.g., ViT, Swin3D, DGCNN, Whisper, LiDAR backbones) (Chen et al., 2024, Ying et al., 30 Jul 2025, Zheng et al., 7 Dec 2025)
• Mixture of Projectors (MOP) modules mapping each modality’s features into a unified LLM embedding space, with router networks to dynamically select domain-relevant projections (Chen et al., 2024)
• Cross-modal fusion strategies: Audio-Visual Interleaving (AVI), additive or deformable cross-attention fusion, or Mixture-of-Experts gating (Ying et al., 30 Jul 2025, Zheng et al., 7 Dec 2025, Li et al., 4 Dec 2025)

Instruction Taxonomy and Cascaded Adapters

OISA must contend with diverse instruction complexity. The SAM3-I framework distinguishes between (i) concept-level (NP) prompts (ii) simple compositional instructions with explicit modifiers, and (iii) complex instructions requiring multi-step or implicit reasoning (Li et al., 4 Dec 2025). The model adapts processing depth—cascaded adapters for simple and complex instruction types—to handle this spectrum, aligning semantic granularity with the instruction structure.

Agentic and Function-Calling Architectures

Some OISA systems utilize agentic pipelines with explicit function-calling APIs. The LLM core outputs structured “Thinking,” “Calling,” and “Replying” streams: generation of reasoning tokens, parsing of function-call requests (e.g., segment(…)), asynchronous invocation of external backends, and conversational feedback to the user (Chen et al., 2024). This orchestrates multimodal and domain-specialized segmentation modules, and enables human-in-the-loop refinement.

3. Learning Frameworks and Optimization

Stagewise and Curriculum Training

Robust OISA systems rely on multi-stage training. Best practices from OmniSegNet and SAM3-I include:

1. Warm-up with unimodal (text or reference) RIS/RAVS datasets for vision-language alignment (Zheng et al., 7 Dec 2025, Ying et al., 30 Jul 2025)
2. Visual or domain instruction tuning with spatial prompts or domain-specific data
3. Joint training/fine-tuning with multimodal or omni-prompt batches, with balanced ratios for modality coverage (Zheng et al., 7 Dec 2025, Li et al., 4 Dec 2025)

Loss Functions

Typical loss components span:

• Segmentation (per-pixel BCE, Dice; DETR-style set prediction for instance masks) (Yuan et al., 25 May 2026, Zheng et al., 7 Dec 2025, Li et al., 4 Dec 2025)
• Presence/classification loss for slot validity (Yuan et al., 25 May 2026)
• Language modeling for explanation (Ying et al., 30 Jul 2025)
• Alignment losses: cross-modal CLIP losses or distributional consistency (e.g., $L_\mathrm{align}$, KL-divergence between mask distributions) (Li et al., 4 Dec 2025, Quenum et al., 5 May 2025)
• Hard-region uncertainty supervision (Li et al., 4 Dec 2025)
• Function-call action or policy learning for agentic architectures (Chen et al., 2024)

Reinforcement and Hierarchical Policy Optimization

For event-based, long-horizon video or audio-visual reasoning, OISA can leverage a two-system decomposition with RL-based training (Omni-R1). A global reasoning module rewrites instructions and selects keyframes via a policy $\pi_\theta$ learned under hierarchical rewards capturing selection diversity, alignment, and temporal segmentation consistency. A detail system executes pixel-level segmentation. Training employs Group Relative Policy Optimization (GRPO), using PPO-style updates with advantage normalization and KL penalties (Zhong et al., 26 May 2025).

4. Domain Adaptation and Data Construction

Multi-Domain, Omnimodal Data

High-quality, large-scale datasets are an enabler for OISA. Representative corpora include:

• Inst²Seg: 100k images / 500k instruction–mask pairs from exocentric and egocentric sources (Yuan et al., 25 May 2026)
• OmniRef: 30k+ images with 186,939 textual/visual/mixed prompts for omniprompts evaluation (Zheng et al., 7 Dec 2025)
• PACO-LVIS-Instruct: 843k positive instructions auto-generated and verified (Li et al., 4 Dec 2025)
• OmniAVS: 2,104 videos, 61,095 multimodal expressions (text, speech, sound, images) (Ying et al., 30 Jul 2025)
• GRES: 27,615 queries over 9,205 satellite images, leveraging large-LLMs for attribute-specific query synthesis (Quenum et al., 5 May 2025)

Annotation strategies may combine MLLM-assisted captioning, agentic quality inspection, and human-in-the-loop curation to guarantee instruction diversity and precision (Li et al., 4 Dec 2025).

Cross-Domain and Plugin Extensions

OISA can add new modality encoders or projectors as “plugins”—for example, 3D, video, or audio backends—supported by extending the fusion/router schema and collecting domain-specific data with per-modality prompt templates. Parameter-efficient tuning (LoRA/prompt tuning) facilitates rapid adaptation to specialized domains without retraining the full model (Chen et al., 2024, Li et al., 4 Dec 2025).

5. Representative Implementations and Benchmarks

Table: OISA Implementations and Key Attributes

System	Modalities	Instruction Handling	Segmentation Outputs
InstructSAM (Yuan et al., 25 May 2026)	Image (RGB), text	Complex, phrase, multi-slot	Multi-instance, set-structured
VS-Assistant (Chen et al., 2024)	2D/3D, video, text/audio	Agentic function-calling	Modular, via function API
OISA-OmniAVS (Ying et al., 30 Jul 2025)	Video, audio, text/image/sound	Multimodal, explanatory	Per-frame mask, text explain
OmniSegNet (Zheng et al., 7 Dec 2025)	Image + mask/box/scribble	Omni-prompt, multimodal	Multi-vs-multi/no-target
SAM3-I (Li et al., 4 Dec 2025)	Image, text, (audio/sketch*)	Instruction taxonomy, adapters	Mask, interactive refinement
LISAT (Quenum et al., 5 May 2025)	Remote-sensing, text	Complex queries, LoRA tuning	Mask via SAM decoder

*star = planned/blueprint feature

Key results underline strong performance on benchmarks: OISA-1B achieves 41.1% average $X$0 on OmniAVS (∆+5.0 over LISA13B), gIoU=60.4 and mAP=31.5 on Inst²Seg, and robust multimodal explanation metrics (Ying et al., 30 Jul 2025, Yuan et al., 25 May 2026). Ablations show necessity for hybrid attention, query bank scaling, and fusion routing.

6. Challenges, Limitations, and Future Directions

OISA’s principal challenges include scaling prompt-encoders to higher-order modalities and larger models, balancing multi-task learning across divergent input types, extending to real-time and streaming data, and addressing ambiguities or hallucinations in keyframe selection or multimodal reference resolution (Zhong et al., 26 May 2025, Ying et al., 30 Jul 2025, Zheng et al., 7 Dec 2025). Additional difficulties arise from mask noise propagation, extreme fine-grained detail segmentation, and rare-class generalization (Quenum et al., 5 May 2025).

Suggested next steps encompass:

• Advanced audio-visual grounding and disentanglement beyond token interleaving (Ying et al., 30 Jul 2025)
• Joint optimization for segmentation, detection, narration, and cross-modal alignment (Quenum et al., 5 May 2025)
• Function-calling extensions for volumetric, panoptic, or real-world robotics applications (Chen et al., 2024)
• Pretraining with large-scale omnimodal corpora, hierarchical patching for large/long-range inputs (Quenum et al., 5 May 2025)
• Interactive multi-turn dialogue with user feedback, looped refinement, and policy-driven action selection (Li et al., 4 Dec 2025, Zhong et al., 26 May 2025)

A plausible implication is that OISA frameworks will underpin future universal, “instructable” segmentation agents compatible with heterogeneous input/output pipelines, scalable function sets, and seamless plug-and-play extension to novel environments and modalities.