Unified Audio Intelligence Without Regressing on Text Intelligence

Abstract: Audio intelligence involves understanding, reasoning about, and generating both audio and speech. In this work, we introduce Nemotron-Labs-Audex-30B-A3B (Audex), a unified audio-text LLM built on Nemotron-Cascade-2-30B-A3B, a strong text-only MoE LLM. Audex adopts a simple unified design with a single Transformer decoder: audio inputs are encoded and projected into the text embedding space, while text tokens and quantized audio output tokens are treated uniformly during generation. This architecture enables strong audio-text fusion, seamless multimodal generation, and compatibility with standard LLM training and inference infrastructure. For training, we meticulously curate audio-text datasets comprising 157.4B audio tokens and 320.5B text tokens. We apply multi-stage supervised training on these datasets, followed by text-only Cascade RL and multi-domain on-policy distillation. Audex delivers state-of-the-art audio understanding, speech recognition and translation, text-to-speech, audio generation, and speech-to-speech generation, while preserving very compelling reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only LLM backbone with marginal or no regression. We release the model checkpoints to facilitate open research.

• The paper introduces Audex, a unified audio-text LLM that achieves state-of-the-art performance across audio tasks without regressing on text intelligence.
• It employs a staged training curriculum with multi-stage supervised fine-tuning and text-only reinforcement learning to preserve core text reasoning.
• Empirical evaluations show improved performance on benchmarks including ASR, TTS, and text reasoning, demonstrating potential scalability for AGI.

Unified Audio Intelligence Without Regressing on Text Intelligence: Technical Essay

Motivation and Context

The development of LLMs has catalyzed advances in numerous modalities—text, vision, and audio. Multimodal LLMs have recently extended their reach to understanding and generating audio and speech, but this integration has historically caused regression in the model's text intelligence (reasoning, knowledge, alignment, agentic tasks) relative to text-only counterparts. This regression impedes the utility and generality required for artificial general intelligence (AGI). The paper "Unified Audio Intelligence Without Regressing on Text Intelligence" (2607.05196) addresses this challenge by proposing a unified model, Nemotron-Labs-Audex-30B-A3B (Audex), which delivers state-of-the-art performance in both audio and text tasks, preserving text reasoning and intelligence while supporting comprehensive audio capabilities.

Model Architecture and Modality Unification

Audex is constructed atop the Nemotron-Cascade-2-30B-A3B backbone, a MoE LLM with hybrid Mamba2-Transformer architecture. The architectural unification is realized via the following mechanisms:

• Audio Input Handling: Audio signals are encoded using a finetuned AF-Whisper—extending Whisper Large-v3's architecture for general audio understanding. These embeddings are projected into the text embedding space via MLP adapters.
• Token Unification: The model's vocabulary is extended to encode audio and speech tokens, quantized via specialized codecs (speech: X-Codec2, non-speech: X-Codec). These tokens are appended to the text vocabulary and treated identically in the autoregressive generation process.
• Autoregressive Generation: Both text and quantized audio tokens are produced directly, enabling seamless fusion and uniform next-token prediction across all modalities.
• Infrastructure Compatibility: Audex's architecture is compatible with Megatron-LM and vLLM stacks, supporting scalable training and inference with context lengths exceeding 1M tokens.

This design counters prior approaches relying on cascaded or separate thinker-talker models, instead ensuring modality fusion and minimizing the modality-induced regression on core text tasks.

Training Recipes and Curriculum

To optimize both text and audio capabilities, Audex implements a multi-stage supervised fine-tuning (SFT) curriculum followed by text-only reinforcement learning (RL) and multi-domain on-policy distillation (MOPD):

• Multi-stage SFT: Sequential training, starting with text SFT (borrowing Nemotron-Cascade-2's checkpoint), followed by audio warmup (only training MLP adapters and audio token embeddings while freezing text token embeddings), then audio generation SFT, and finally joint SFT on audio understanding, ASR, AST, and text. This strategy preserves long-context and reasoning abilities present in the backbone LLM.
• Single-stage SFT: Direct consolidation of all modalities and tasks leads to instability in long-context ability, indicating the superiority of staged curriculum.
• Text-Only RL: Cascade RL, as in Nemotron-Cascade, enhances text intelligence without affecting multimodal capabilities, circumventing catastrophic forgetting observed in prior multimodal RL.

Data curation spans 157.4B audio tokens and 320.5B text tokens, integrating ASR, AST, TTS, audio Q&A, and text-to-audio datasets. Classifier-free guidance is incorporated for effective controllable audio generation, using unconditional data blending.

Benchmark Evaluation and Numerical Performance

Audex exhibits strong empirical performance across benchmarks:

1. Text Intelligence: Baseline-superior accuracy is achieved on math (AIME, HMMT Feb25), code (LiveCodeBench), knowledge (MMLU-Redux, GPQA-Diamond), alignment (ArenaHard v2, IFBench), long-context (NIAH, AA-LCR), and agentic tasks (T2-Bench, SWE Verified). Notably, Audex achieves higher scores than Nemotron-Cascade-2 on several reasoning tasks and demonstrates only marginal regression on others.
2. Audio Understanding/Generation: Performance on MMAU, MMAR, and MMSU benchmarks matches or exceeds strong open audio LLMs (Step-Audio-R1.1-33B, Qwen3-Omni-30B-A3B-Thinking). Audex surpasses prior SOTA in ASR (LibriSpeech WER), noisy ASR, multilingual ASR (Fleurs), and AST (BLEU/COMET).
3. Text-to-Speech/Text-to-Audio: Audex achieves low WER on TTS and competitive Fréchet Distance (FDopenL3) on AudioCaps and SongDescriber for text-to-audio generation, outperforming diffusion and autoregressive task-specific baselines even with direct token-based generation, benefitting from classifier-free guidance and optimized inference hyperparameters.
4. Speech Interaction and Agentic Performance: On BigBenchAudio and VoiceBench, Audex demonstrates high instruction-following, reasoning, safety, and multi-choice QA performance, competitive with proprietary models and exceeding several open baselines.

Boldly, the paper claims that Audex is the only open model among strong baselines with general audio generation beyond speech, and that text intelligence is preserved—with negligible regression—through careful recipe and curriculum design.

Methodological Ablations

The paper provides analysis on curriculum, embedding management, and data blending:

• Audio Warmup Ablation: Freezing text embeddings during audio warmup is essential to maintain high text benchmark scores; unfreezing leads to substantial degradation.
• Text Blending Ratios: High text blending ratios during joint SFT phases are necessary to avoid text skill loss, particularly in consolidated single-stage training, but do not solve long-context breakage.
• Classifier-Free Guidance/Inference: CFG is pivotal for audio generation quality; optimal CFG scaling is explored for task-specific generation.

These insights contribute to protocol design for future multimodal LLMs targeting true modality unification.

Practical and Theoretical Implications

Audex establishes that unified audio-text LLMs capable of comprehensive understanding, reasoning, and generation across modalities can maintain core text intelligence, given careful architecture and curriculum design. This approach demonstrates:

• Compatibility: Seamless integration into standard LLM infrastructure, facilitating scalable practical deployment.
• Generalization: State-of-the-art results in diverse audio and text domains, including long-context, tool use, and multi-lingual scenarios.
• Open Model Availability: Release of model checkpoints promotes open research and broad adoption.

On a theoretical level, Audex validates the hypothesis that catastrophic forgetting and performance regression in multimodal models is not inherent but rather a function of blending and stage-wise curriculum design. The preservation of text abilities through text-only RL after multimodal SFT suggests resistance to interference, opening avenues for audio-text RL to further enhance audio capabilities without text regression.

Prospects for Future Developments

Future directions indicated by the findings include:

• Audio-Text RL: Extending RL to audio tasks, leveraging the paradigm's resistance to catastrophic forgetting.
• Enhanced Data Diversification: Addressing duration and content consistency in audio generation with variable-length audio and improved curation.
• Fine-Grained Control and Modality Reasoning: Leveraging unified token prediction and CFG to enable controllable, fine-grained generation and cross-modal reasoning.
• Unified Multimodal AGI: Applying the Audex recipe to broader modalities (vision, video), scaling to larger model sizes, and integrating agentic capabilities for generalist intelligence.

Conclusion

"Unified Audio Intelligence Without Regressing on Text Intelligence" (2607.05196) presents Audex, a unified audio-text LLM architecture that achieves strong empirical performance across both audio and text domains, without the longstanding regression in text intelligence seen in prior multimodal models. Key innovations include architectural unification, staged curriculum fine-tuning, careful data blending, classifier-free guidance, and infrastructure compatibility. Audex's results substantiate the feasibility of robust, general-purpose audio intelligence coexisting with comprehensive text intelligence in the same LLM, laying groundwork for future scalable, unified AGI models.