Audio-Omni: Native Omni-Modal Audio Systems

• Audio-Omni is a design objective that treats audio as a native modality, integrating it with text, image, and video for unified representation and reasoning.
• It enables omni-modal understanding, generation, shared-space retrieval, and evaluation through unified sequence modeling and decoupled reasoning architectures.
• Recent advances address audio’s temporal density and modality imbalance with techniques like explicit alignment, audio-aware token compression, and latent generative systems.

Searching arXiv for papers on Audio-Omni and closely related omni-modal audio systems. Audio-Omni denotes a family of omni-modal systems in which audio is treated as a first-class modality for representation learning, perception, reasoning, interaction, generation, and editing alongside text, image, and video. In recent work, the term encompasses shared embedding models for text–image–video–audio retrieval, multimodal LLMs that jointly listen and speak, unified encoders for synchronized audio–visual understanding, and latent generative systems that synthesize or edit sound effects, music, speech, and full video soundtracks from heterogeneous conditions (Chen et al., 7 Jan 2026, Li et al., 26 Jan 2025, Tian et al., 12 Apr 2026, Xu et al., 22 Sep 2025).

1. Definition and research scope

In the current literature, Audio-Omni is not a single architecture but a design objective: to make audio native to omni-modal computation rather than an auxiliary branch. In Baichuan-Omni-1.5, this is described as making audio “a first-class citizen in an omni-modal MLLM that sees, listens, and speaks, with end-to-end speech I/O integrated into the same sequence modeling backbone that handles text, images, and video” (Li et al., 26 Jan 2025). In e5-omni, the same objective appears in embedding form: off-the-shelf vision–language backbones are adapted into omni-modal embedding models spanning text, image, video, and audio, with particular emphasis on stabilizing cross-modal alignment in mixed-modality training (Chen et al., 7 Jan 2026).

This scope now spans at least four partially overlapping problem settings. The first is omni-modal understanding, where audio must be jointly interpreted with vision and language in tasks such as ASR, audio QA, audio-visual reasoning, and multi-turn dialogue. The second is omni-modal generation, where systems produce speech, sound effects, music, or full mixed soundtracks from text, video, audio, or their combinations. The third is shared-space retrieval and compression, where audio participates in common embedding spaces or token-reduction policies. The fourth is evaluation, where work such as AVI-Bench argues that audio-visual intelligence should be measured across perception, understanding, reasoning, and unfamiliar low-semantic stimuli rather than by isolated benchmarks alone (Wang et al., 1 Jun 2026).

A recurrent theme across these lines of work is that audio introduces temporal density, ambiguity, and modality imbalance. That pressure has produced several distinct architectural responses: unified sequence modeling, decoupled reasoning–speech systems, explicit alignment modules, audio-aware token compression, and latent reasoning mechanisms that avoid collapsing sensory evidence into text too early.

2. Architectural patterns for treating audio as a native modality

One major pattern is single-backbone sequence modeling. Baichuan-Omni-1.5 uses a visual branch, an audio branch, and a 7B LLM backbone that jointly processes interleaved multimodal tokens; for generation, the LLM alternates between predicting text tokens and audio tokens in one sequence, with a special modality-switch token gating prediction between text and audio (Li et al., 26 Jan 2025). HyperCLOVA X 8B Omni similarly consolidates text, vision, and audio inputs and outputs into a single 8B decoder-only Transformer with a shared next-token prediction interface over an interleaved multimodal sequence, while also using continuous audio embeddings for understanding and discrete audio tokens for generation (Team, 5 Jan 2026). InteractiveOmni follows the same unification principle with InternViT-300M, Whisper-large-v3, Qwen3, and a CosyVoice2-based speech stack, using a fixed 5:25 text:speech interleaving ratio for streaming synthesis (Tong et al., 15 Oct 2025).

A second pattern is decoupled reasoning and speech generation. Qwen3-Omni introduces a Thinker–Talker MoE architecture in which the Thinker handles multimodal reasoning and the Talker generates streaming speech via discrete codec tokens, multi-token prediction, and a causal ConvNet called Code2Wav (Xu et al., 22 Sep 2025). Qwen3.5-Omni extends this design with a Hybrid Attention Mixture-of-Experts framework for both Thinker and Talker, a 256k context length, and ARIA, which dynamically aligns text and speech units to improve the stability and prosody of conversational speech (Team, 17 Apr 2026). MGM-Omni adopts a related “brain-mouth” design, in which a Qwen2.5-VL-based MLLM performs reasoning and a Qwen3-based SpeechLM performs chunk-based, parallel speech decoding for low-latency, streaming zero-shot voice cloning with stable timbre over extended durations (Wang et al., 29 Sep 2025).

A third pattern is encoder–LLM–decoder modularity. Nexus places Whisper-large-v3 behind a two-layer MLP adapter on top of Qwen2.5-VL-7B, then couples the LLM to a 6-layer autoregressive audio decoder using Fish-Speech or CosyVoice2.0 as the speech generator (Liu et al., 26 Feb 2025). River-Omni, introduced in the Baichuan-Omni technical report, also uses Whisper-large-v3 as the audio front end, but replaces simple pooling with Conv-GMLP to preserve more information under down-sampling before projection into the LLM embedding space (Li et al., 2024).

A fourth pattern is frame-synchronous unified encoding. OmniEncoder argues that prevailing systems use a “video-coarse, audio-dense” design and instead co-embeds visual and audio signals at a symmetrical 25 fps within a single 24-layer Transformer, using Audio tokens, Visual Continuous tokens, Visual Base tokens, Omni-RoPE, and Temporal Window Shifting (Bai et al., 2 May 2026). This is an explicitly encoder-side response to the claim that many omni-modal systems still perceive video frame by frame and modality by modality rather than holistically.

Taken together, these designs indicate that Audio-Omni is not converging on a single canonical blueprint. The field instead distinguishes between systems that unify audio at the token interface, systems that separate reasoning from speech rendering, and systems that move fusion earlier into the encoder.

3. Alignment, shared spaces, and audio-aware efficiency

Audio frequently destabilizes shared-space learning. e5-omni identifies three failure modes when audio is added to a shared embedding space: modality-dependent similarity sharpness or scale, diminishing efficacy of in-batch negatives in mixed-modality batches, and mismatched first- and second-order statistics across modalities (Chen et al., 7 Jan 2026). Its alignment recipe adds modality-aware temperature calibration, a controllable negative curriculum with debiasing, and batch whitening with covariance regularization. The calibration step defines

$$\tau(x)=\max(w(x)^\top\tau,10^{-6}), \qquad \ell(q,p)=\frac{\mathrm{sim}(e(q),e(p))}{\tau(q,p)},$$

and the final training objective is

$L=L_{DCL}+\lambda_{coral}L_{coral}.$

In e5-omni-7B, the learned temperatures are $[0.0130, 0.0127, 0.0219, 0.0223]$ for $\{text, image, audio, video\}$, and on AudioCaps the model reaches Recall@1 of 37.7, compared with 34.0 for Tevatron-Omni, 24.2 for LCO-EMB, and 20.5 for Omni-Embed-Nemotron (Chen et al., 7 Jan 2026).

A complementary line asks how to reduce the cost of audio-visual omni models without destroying context. ContextGuard reframes token pruning as preserving broad audio-visual context while removing cross-modal redundancy. Its principle is to “keep what audio cannot say”: video tokens whose coarse semantics are explainable by audio are pruned, localized visual details are retained via a spatial coverage criterion, and temporally similar chunks are merged (Jung et al., 12 May 2026). On Qwen2.5-Omni 7B, ContextGuard achieves 55% token reduction, matches full-token performance on five of six benchmarks, reduces peak memory from 27.1 GB to 24.6 GB, prefill time from 5.0 s to 3.1 s, and end-to-end latency from 6.7 s to 4.5 s (Jung et al., 12 May 2026).

LatentOmni targets a different bottleneck: the conversion of dense audio-visual evidence into explicit text chain-of-thought. It interleaves textual reasoning with continuous latent states and introduces Omni-Sync Position Embedding,

$$\operatorname{OSPE}(h,t)=h\odot\cos(t\Theta)+\mathcal{R}(h)\odot\sin(t\Theta),$$

together with a latent grounding loss and a symmetric temporal synchronization loss (Dai et al., 21 May 2026). On Daily-Omni, WorldSense, OmniVideoBench, and LVOmniBench, it outperforms both the Qwen2.5-Omni-7B base model and an Explicit Text CoT baseline (Dai et al., 21 May 2026). OmniEncoder addresses the same alignment problem at the encoder level by co-embedding audio and visual streams at 25 fps and preserving dense motion-sensitive visual tokens under the same downstream LLM token budget (Bai et al., 2 May 2026).

These systems collectively suggest that Audio-Omni requires more than generic multimodal fusion. Audio changes similarity scales, negative hardness, temporal sampling, and token budgets, and recent work increasingly treats these as primary optimization targets rather than implementation details.

4. Audio generation, speech synthesis, and editing

Generative Audio-Omni systems divide broadly between autoregressive codec models and latent diffusion or flow-matching models.

Among autoregressive systems, HyperCLOVA X 8B Omni uses a Whisper-large-v3–initialized encoder for continuous audio embeddings, a 6,561-token FSQ audio tokenizer at 25 Hz for generation, and Unit-BigVGAN conditioned on an ECAPA-TDNN speaker embedding for waveform synthesis (Team, 5 Jan 2026). Qwen3-Omni and Qwen3.5-Omni use Talker modules that generate discrete speech codec tokens and stream waveforms from the first codec frame, with Qwen3.5-Omni further introducing ARIA to align text and speech rates during streaming synthesis (Xu et al., 22 Sep 2025, Team, 17 Apr 2026). Baichuan-Omni-1.5 integrates an 8-layer residual vector quantizer at 12.5 Hz, an audio head inside the LLM, a spectrogram reconstruction decoder, a flow-matching U-Net, and a HiFi-GAN vocoder, thereby unifying understanding and end-to-end speech generation within one sequence-modeling backbone (Li et al., 26 Jan 2025).

Among latent generative systems, Audio-Omni couples a frozen Qwen2.5-Omni-3B with a trainable Diffusion Transformer for synthesis and editing across general sound, music, and speech, using Rectified Flow in VAE latent space with

$$x_t=(1-t)x_0+tx_1, \qquad L=\mathbb{E}\big[\|v_\theta(x_t,t,c)-(x_1-x_0)\|^2\big].$$

It also introduces AudioEdit, a dataset comprising over one million editing pairs, and reports competitive or superior results across T2A, T2M, V2A, V2M, TTS, and editing benchmarks (Tian et al., 12 Apr 2026).

Omni2Sound addresses unified video-to-audio, text-to-audio, and joint video-text-to-audio generation within a single diffusion transformer, using SoundAtlas with 470,000 video–audio–text pairs and a three-stage multi-task progressive training schedule to resolve cross-task competition and modality bias in VT2A (Dai et al., 6 Jan 2026). UniFlow-Audio instead unifies time-aligned and non-time-aligned tasks by combining additive fusion of time-aligned conditioning with cross-attention over non-time-aligned conditioning inside a flow-matching transformer, and reports strong results across TTS, SVS, T2A, T2M, SE, SR, and V2A with fewer than 8K hours of public training data and under 1B trainable parameters (Xu et al., 29 Sep 2025). AudioGen-Omni uses an MMDiT backbone, a unified lyrics-transcription encoder, AdaLN-based joint attention, and Phase-Aligned Anisotropic Positional Infusion, and reports an inference time of 1.91 seconds for 8 seconds of audio (Wang et al., 1 Aug 2025).

Foley-Omni extends the generative agenda from isolated tasks to full soundtrack generation. Conditioned on structured text fields $[WORDS]$, $[AUDIO]$, and $[MUSIC]$, plus CLIP and Synchformer video features, it generates a single mixed soundtrack in latent space and evaluates on V2ST-Bench, a benchmark of 300 five-to-ten-second clips with coexisting speech, sound effects, and music (Tao et al., 2 Jun 2026).

5. Benchmarks, empirical performance, and what current systems actually improve

The empirical landscape shows substantial progress, but also considerable variation in what different systems improve. The table below organizes representative results that define the current Audio-Omni frontier.

System	Setting	Reported outcome
e5-omni-7B	AudioCaps text–audio retrieval	Recall@1 37.7
Omni-R1 (VGGS-GPT)	MMAU Test-mini	Avg 71.3
InteractiveOmni-4B	MMMB	52.47
Qwen3-Omni-30B-A3B-Instruct	FLEURS-avg over 19 languages ASR	WER 5.33
Qwen3.5-Omni-Plus	Librispeech ASR	1.11% clean, 2.23% other
AVI-Bench top model	AVSQA	16.50

Omni-R1 is notable because it shows that benchmark gains need not derive primarily from better raw-audio grounding. It fine-tunes Qwen2.5-Omni-7B with GRPO on audio QA data and reaches 71.3 on MMAU Test-mini and 71.2 on Test-full with VGGS-GPT, but its ablations show that much of the improvement comes from better text-based reasoning, and that text-only fine-tuning on ARC-Easy can substantially improve audio QA (Rouditchenko et al., 14 May 2025). This is one of the clearest demonstrations that audio QA benchmarks can reward general reasoning improvements as much as improved sensory grounding.

InteractiveOmni emphasizes multi-turn interaction and long-term conversational memory. On the Multi-modal Multi-turn Memory Benchmark, InteractiveOmni-4B reaches 52.47 and InteractiveOmni-8B reaches 58.17, compared with 25.48 for Qwen2.5-Omni-7B and 51.33 for GPT-4o-mini; on the Multi-turn Speech Interaction Benchmark, InteractiveOmni-4B and 8B score 3.95 and 4.03, respectively (Tong et al., 15 Oct 2025). The same paper reports MMAU averages of 72.00 for IO-4B and 67.39 for IO-8B, and OpenAudioBench averages of 69.10 and 72.74 (Tong et al., 15 Oct 2025).

At larger scale, Qwen3-Omni reports open-source SOTA on 32 of 36 audio and audio-visual benchmarks and overall SOTA on 22, with especially strong ASR, lyric ASR, music understanding, and real-time speech interaction (Xu et al., 22 Sep 2025). Qwen3.5-Omni-plus scales this agenda to 215 subtasks and benchmarks, reports 82.2 on MMAU and 82.8 on MMSU, surpasses Gemini-3.1 Pro on key audio tasks, and achieves 1.11% and 2.23% WER on Librispeech clean and other (Team, 17 Apr 2026).

Yet evaluation work remains cautionary. AVI-Bench reports that even Gemini-2.5-Pro reaches only 39.13 on Audio-Visual Localization, 35.08 on Audio-Visual Language Grounding, and 16.50 on AVSQA, while a pilot human score on AVSQA is 90.55 (Wang et al., 1 Jun 2026). The benchmark argues that most Omni-MLLMs remain visually stronger than they are acoustically grounded, and that domain adaptation to unfamiliar low-semantic stimuli remains weak.

6. Misconceptions, limitations, and open research directions

A common misconception is that stronger audio benchmark scores necessarily imply better direct audio understanding. Omni-R1 explicitly complicates that reading: text-only RL can improve audio QA performance, and much of the observed gain is attributed to better text-based reasoning rather than improved audio perception (Rouditchenko et al., 14 May 2025). A related finding appears in Baichuan-Omni-1.5, where transcripts consistently improve OmniBench accuracy relative to raw audio, indicating that the direct audio-understanding path is less robust than text-based reasoning (Li et al., 26 Jan 2025). This suggests that part of the current Audio-Omni literature measures the quality of multimodal reasoning under imperfect acoustic grounding rather than the latter alone.

A second misconception is that simply adding audio to a strong vision-LLM yields a stable omni-modal representation. e5-omni shows the opposite: audio introduces similarity-scale mismatch, hardness imbalance, and covariance mismatch, and these affect ranking stability and gradient balance unless explicitly corrected (Chen et al., 7 Jan 2026). Likewise, OmniEncoder argues that sparse video sampling combined with dense audio processing leaves cross-modal interaction impoverished during encoding, especially for motion-centric tasks (Bai et al., 2 May 2026).

A third issue is evaluation scope. AVI-Bench finds that current models often perform better on visual-dominant tasks than on audio-dominant ones, struggle with grounding and temporal reasoning, and remain brittle on unfamiliar, low-semantic stimuli (Wang et al., 1 Jun 2026). This suggests that progress in Audio-Omni still outpaces the field’s ability to certify human-like audio-visual intelligence.

On the generation side, present systems also have bounded scope. Audio-Omni notes that editing reliability can depend on source separation quality and CLAP filtering thresholds, and detailed failure cases are not fully reported (Tian et al., 12 Apr 2026). Foley-Omni directly outputs a single mixed track rather than explicit stems, which limits per-component control at inference time (Tao et al., 2 Jun 2026). LatentOmni improves audio-visual reasoning, but its fixed latent budget can miss very long or highly dispersed evidence (Dai et al., 21 May 2026). ContextGuard improves inference efficiency, but weak audio semantics, silent videos, music-only audio, off-screen sound sources, or noisy or misaligned audio reduce predictor reliability (Jung et al., 12 May 2026).

The broader direction of travel is nevertheless clear. Current work increasingly converges on several priorities: better temporal alignment mechanisms, stronger audio-native encoders, explicit treatment of mixed-modality optimization pathologies, more faithful long-horizon speech generation, and evaluation suites that separate raw acoustic grounding from language-side inference. A plausible implication is that future Audio-Omni systems will be judged less by whether they can merely “handle audio” and more by whether they can preserve audio’s temporal, semantic, and interactional structure without collapsing it into text too early.