Omni-modal Large Language Models

• Omni-modal Large Language Models are unified neural systems that process text, vision, and audio to achieve modality-invariant semantic inference.
• They employ modality-specific encoders with a shared Transformer backbone to map diverse sensory inputs into a common embedding space.
• Benchmark evaluations like XModBench and Omni-SafetyBench highlight key challenges in cross-modal reasoning, safety, and alignment.

Omni-modal LLMs (OLLMs) are an evolution of foundational LLMs that extend unified representation and reasoning to text, vision, and audio within a single neural architecture. This class of models seeks modality-invariant semantic inference: the ability to reach consistent conclusions regardless of whether the same content is presented through text, images, or sound. The OLLM paradigm is foundational for developing artificial intelligence systems capable of human-like perception, flexible cross-modal understanding, and robust interaction in open worlds.

1. Core Definition and Objectives

An OLLM is a single neural model trained to process audio waveforms, images or video frames, and natural language text as interchangeable vehicles for semantic information. The essential objective is cross-modal consistency: the model must rely on aligned, high-level concepts rather than modality-specific cues. Ideal OLLMs support modality-agnostic inference, where:

• The same question delivers stable, correct answers across modality permutations (e.g., A→T, A→V, T→A, T→V, V→A, V→T).
• There is no systematic bias favoring or penalizing any single sensory stream.
• Predictions remain invariant when context and answer modalities are swapped.

XModBench formalizes these goals by systematically evaluating all pairwise modality configurations for each semantic instance, enabling granular diagnosis of cross-modal reasoning failures and asymmetries (Wang et al., 16 Oct 2025).

2. Model Architectures and System Design

Leading OLLMs integrate modality-specific encoders (e.g., Whisper-large-v3 for audio, Vision Transformers for images/video) with a shared Transformer backbone for joint sequence modeling. Key system-level innovations include:

• Modality Encoders and Projection: Each raw modality undergoes feature extraction and projection into a shared embedding space compatible with the LLM's token inputs (Liu et al., 6 Feb 2025).
• Unified Model Foundation: Architectures range from dense Transformers to dynamic-capacity Mixture-of-Experts (MoE), e.g., Uni-MoE-2.0-Omni that interleaves routed, shared, and null experts with 3D rotary positional encodings for fine-grained spatio-temporal alignment (Li et al., 16 Nov 2025).
• Modular and Highly Parallelized Training: Systems such as VeOmni decouple computation from communication, leveraging multi-dimensional (data/tensor/expert) parallelism for efficient scaling of large OLLMs (up to 30 B parameters and 160 K context length) (Ma et al., 4 Aug 2025).
• Innovations in Decoding: MGM-Omni’s dual-track “brain–mouth” design decouples multimodal reasoning (MLLM) from real-time speech synthesis (SpeechLM), supporting parallel chunk-based decoding for low-latency, high-fidelity audio output (Wang et al., 29 Sep 2025).
• Unified Token Space and Diffusion: Models like Dynin-Omni push further by adopting masked diffusion over a joint discrete vocabulary spanning all modalities, enabling any-to-any input–output generation, bidirectional context, and simultaneous editing/inference (Kim et al., 9 Mar 2026).

3. Cross-Modal Reasoning: Evaluation, Bottlenecks, and Alignment

OLLMs are systematically tested on large-scale multi-modal benchmarks that disambiguate modality-invariant reasoning from modality-specific or direction-specific biases:

• Task Taxonomy: Core competencies cover perception (object/sound/event recognition), spatio-temporal reasoning, linguistic understanding (OCR/ASR/cross-modal translation), and knowledge integration.
• Benchmarking:
  • XModBench: 60,828 tri-modal MCQs across 17 subtasks in five families, with all context–candidate modality permutations (Wang et al., 16 Oct 2025).
  • MMAO-Bench: 44+ task types, quantifying compositional law between uni-modal and omni-modal skills; S_omni ∝ (S_vis × S_aud)^γ (Chen et al., 21 Oct 2025).
• Metrics:
  • Task accuracy averaged across modality pairs:

  $$\text{TaskAcc} = \frac{1}{6}\sum_{(X,Y)\in\{A,T,V\}^2,\,X\neq Y} \mathrm{Acc}_{X\to Y}$$ - Modality disparity (e.g., Δ{T\vs A}), directional imbalance (Δ{X↔Y} = Acc_{X→Y} − Acc_{Y→X}).

• Empirical Findings:

  • Spatial/temporal reasoning is a critical bottleneck; state-of-the-art models (e.g., Gemini 2.5 Pro) remain near or below 60% accuracy in these domains.
  • Severe modality disparities exist: for leading models, accuracy drops by ~49 points when switching from text to audio, ~33 for vision to audio, and ~15 for text to vision.
  • Directional imbalances appear: vision–text pairs exhibit pronounced asymmetric consistency, with models systematically favoring reading texts over interpreting vision as context.
  • Robust cross-modal alignment is not achieved: the weakest modality constrains cross-modal performance in weaker OLLMs (“bottleneck effect”), while only strong models achieve demonstrable synergy (“super-multiplicative” gains) (Chen et al., 21 Oct 2025).

4. Training Strategies and Data Curation

To achieve modality-agnostic generalization, OLLMs employ complex data curation and curriculum strategies:

• Multi-Stage and Progressive Training: Most state-of-the-art OLLMs follow a progression—modality-specific warm-up (e.g., ASR for audio, image captioning for vision), then joint supervised fine-tuning on balanced, high-quality multi-modal instruction datasets, followed by generative, reinforcement, or preference-based optimization (e.g., GSPO–DPO) to further calibrate cross-modal and generative outputs (Li et al., 16 Nov 2025, Wang et al., 29 Sep 2025).
• Synthetic, Curated, and Multi-turn Evaluations: Benchmarks are rigorously curated for clarity and correctness, employing LLM-assisted template generation, human QA, and competitive OLLM filtering, with additional multi-turn and memory-oriented evaluations (e.g., InteractiveOmni's Multi-Turn Memory Benchmarks) (Tong et al., 15 Oct 2025).
• Curriculum Complexity: Advanced pipelines incorporate multi-step open-ended tasks, gradual increases in reasoning complexity, and modality-ablation checks to expose failure modes at scale (Wang et al., 16 Oct 2025, Chen et al., 21 Oct 2025).
• Balanced Sampling and Data Efficiency: Leading OLLMs reduce training cost and dataset size by dynamic, length-aware batching and progressive integration of modalities, achieving top-level performance with markedly fewer samples or compute than previous generations (Wang et al., 29 Sep 2025).

5. Safety, Hallucination, and Robustness in OLLMs

OLLMs present unique safety and alignment challenges due to their cross-modal capabilities:

• Benchmarks and Metrics:
  • Omni-SafetyBench sets a standard for multi-modal safety evaluation with 24 subcategories and >20,000 test cases, introducing metrics such as C-ASR (Conditional Attack Success Rate) and CMSC-score (Cross-Modal Safety Consistency Score) (Pan et al., 10 Aug 2025).
  • AdvBench-Omni exposes cross-modal safety gaps via modality-semantics decoupling, revealing that OLLMs are far less robust when harmful payloads are split across modalities, with cross-modal refusal rates dropping as much as 27% compared to text-only settings (Wang et al., 10 Feb 2026).
• Mechanistic Insights:
  • Mid-layer “dissolution” of refusal signals—attenuation of safety representations in the encoder stack—explains much of the vulnerability to cross-modal jailbreaks.
  • Extraction of “golden refusal vectors” (via SVD) enables lightweight, adapter-based steering (OmniSteer) at inference, boosting cross-modal refusal rates without degrading general capabilities.
• Alignment and Hallucination Mitigation:
  • Modality-conditioned preference learning frameworks (e.g., OmniDPO) directly penalize over-reliance on textual priors and enforce audio-video correlation, reducing multimodal hallucinations and increasing cross-modal F1 by up to 6.1% (Chen et al., 31 Aug 2025).
  • Open- and closed-world tradeoffs: combining consistency-based losses, deliberate reasoning (OmniGuard), and dynamic policy updates offers robust safety across unseen modalities and tasks (Zhu et al., 2 Dec 2025).
  • OLLMs exhibit critical failure cases where a model is safe on text (high CMSC-score) yet unsafe in audio or vision on equivalent prompts, indicating unsolved consistency gaps (Pan et al., 10 Aug 2025, Wang et al., 10 Feb 2026).

6. Open Challenges and Practical Implications

Current OLLMs, while demonstrating strong perceptual and linguistic competence, are demonstrably not modality-invariant: systematic disparities, bottlenecks on weak modalities, and high error rates on spatial/temporal reasoning persist (Wang et al., 16 Oct 2025, Chen et al., 21 Oct 2025). Common limitations include:

• Audio understanding is consistently subpar; accuracy drops sharply whenever semantic decoding relies exclusively on audio input.
• Spatial-temporal reasoning is weakly addressed due to limited training signals and insufficiently integrated feature extraction pipelines.
• Safety vulnerabilities emerge from under-aligned modality interactions, requiring continued innovation in both architectural and training objective design.

Robust omni-modal reasoning will require:

• Enhanced and diversified pretraining—especially for audio modalities, with rich spatial/temporal and speaker-variant data.
• Joint contrastive/generative alignment objectives to force co-location of multi-modal embeddings, combined with explicit consistency-penalizing losses.
• Architectural advances supporting balanced, bidirectional cross-modal attention and information flow.
• Modular, model-centric system designs for scalable large-model training with minimal engineering overhead (Ma et al., 4 Aug 2025).
• Standardized, multi-faceted benchmarks (e.g., XModBench, MMAO-Bench, Omni-SafetyBench) to track progress in modality invariance, consistency, synergy, safety, and emergent reasoning.

In summary, OLLMs constitute a critical research frontier for artificial general perception, with significant foundational and practical challenges remaining before true modality-invariant, safe and robust AI agents are realized (Wang et al., 16 Oct 2025, Pan et al., 10 Aug 2025, Chen et al., 21 Oct 2025, Li et al., 16 Nov 2025, Wang et al., 29 Sep 2025, Kim et al., 9 Mar 2026, Wang et al., 10 Feb 2026, Chen et al., 31 Aug 2025).