Audex: Unified Audio-Text 30B Model

• Audex is a unified audio-text model that integrates audio and text processing using a single Transformer decoder.
• It leverages a 30B MoE backbone featuring a 52-layer hybrid Mamba2–Transformer to ensure robust performance across diverse modalities.
• The model supports speech recognition, text-to-speech, audio generation, and multimodal reasoning while preserving strong text capabilities.

Nemotron-Labs-Audex-30B-A3B, usually abbreviated Audex, is a unified audio-text LLM that is built on Nemotron-Cascade-2-30B-A3B, a strong text-only MoE LLM, and is designed to understand, reason about, and generate both audio and speech while preserving the reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only backbone with marginal or no regression (Kong et al., 6 Jul 2026). Its defining architectural choice is a single Transformer decoder over a unified token space: audio inputs are encoded and projected into the text embedding space, while text tokens and quantized audio output tokens are treated uniformly during generation, enabling audio-text fusion, multimodal generation, and compatibility with standard LLM training and inference infrastructure (Kong et al., 6 Jul 2026).

1. Concept and scope

Audex is presented as a model for unified audio intelligence, where “audio intelligence” includes understanding, reasoning about, and generating both audio and speech (Kong et al., 6 Jul 2026). The stated target is not merely ASR or speech translation, but a broader set of capabilities: audio understanding, speech recognition, speech translation, text-to-speech, audio generation, and speech-to-speech generation. In parallel, the system is explicitly designed to avoid a common multimodal failure mode: adding multimodal I/O should not significantly degrade text reasoning, knowledge, alignment, long-context behavior, or agentic performance (Kong et al., 6 Jul 2026).

This positioning distinguishes Audex from narrowly specialized speech or audio systems. It is neither only an encoder-decoder speech model nor only a text LLM with an auxiliary audio front end. Instead, it is trained so that text tokens and codec tokens participate in a single autoregressive generative process. A central empirical claim of the work is therefore twofold: the model adds strong audio capabilities while preserving “very compelling” text capabilities of the backbone with only small and often symmetric benchmark deltas (Kong et al., 6 Jul 2026).

The model is released for open research, together with a smaller Audex-2B dense Transformer variant trained with the same recipe (Kong et al., 6 Jul 2026).

2. Backbone and unified architecture

Audex uses a 30B-parameter MoE backbone initialized from the Nemotron-Cascade-2 SFT checkpoint. The architecture retained in Audex is specified as a 52-layer hybrid Mamba2–Transformer with model dimension 2688, 128 routable experts, 6 activated per token, and an original vocabulary of 131,072 tokens (Kong et al., 6 Jul 2026).

Component	Specification	Role
Backbone	52-layer hybrid Mamba2–Transformer; model dimension 2688	Text-only core LLM
MoE	128 routable experts; 6 activated per token	Sparse conditional computation
Audio input path	AF-Whisper encoder + 2-layer MLP adapters	Maps waveform features into LLM embeddings
Audio output path	X-Codec2 for speech; X-Codec for non-speech	Produces discrete audio tokens
Vocabulary	205,312 padded tokens	Unified text + audio token space

The decisive architectural claim is that Audex keeps a single decoder-only LLM. Text tokens remain in the original subword/BPE vocabulary. Audio output is represented as discrete codec tokens appended to the vocabulary. Audio input, by contrast, is represented as continuous embeddings produced by an external encoder and then projected into the LLM hidden space (Kong et al., 6 Jul 2026).

For audio inputs, Audex uses the AF-Whisper encoder from Audio Flamingo 3. It takes 16 kHz waveform input and produces 1280-dim features at 25 Hz, yielding 750 frames for each 30 s. Audio longer than 30 s is chunked, and the final partial window is padded to 30 s. A 2-layer MLP maps these 1280-dim features into the model’s 2688-dim embedding space. In prompts, audio is represented by a <sound> token that is internally expanded to <so_start>, a sequence of <so_embedding> tokens, and <so_end>; these embeddings then participate in self-attention with text tokens, with no separate decoder (Kong et al., 6 Jul 2026).

For audio outputs, Audex uses two codec families. Speech is tokenized with X-Codec2, a speech-focused codec with 16 kHz input, 50 Hz frame rate, single-layer finite scalar quantization (FSQ), codebook size 65,536, and speech tokens of the form <speechcodec_{id}>. General audio uses X-Codec, again at 16 kHz and 50 Hz, with 8-layer residual vector quantization (RVQ) and 1,024 codes per layer; to reduce token rate, Audex uses only the first 4 RVQ layers (RVQ4) during training, giving an effective rate of 200 tokens/s, and reserves 8,192 non-speech codec indices as <audiocodec_{id}> (Kong et al., 6 Jul 2026).

The vocabulary extension is explicit: 131,072 text tokens + 65,536 speech codec tokens + 8,192 audio codec tokens + 5 special generation tokens = 204,805, padded to 205,312 for Megatron tensor parallelism. Newly added audio-token embeddings are initialized as $\mathcal{N}(0, 0.02^2)$ (Kong et al., 6 Jul 2026).

Audex retains the backbone’s MoE routing and applies it uniformly across modalities. The routing configuration is Top-K, with 128 experts and 6 activated per token, using auxiliary-loss-free load balancing plus a small standard load-balancing loss, with routing scale 2.5, update rate $10^{-3}$, and load-balancing loss coefficient $10^{-6}$ (Kong et al., 6 Jul 2026). This means that audio and text tokens pass through the same routed expert system rather than through modality-specific expert partitions.

3. Tokenization, sequence construction, and objective

Audex’s training objective is the standard next-token objective over the unified vocabulary, where the prediction space contains both text tokens and codec tokens (Kong et al., 6 Jul 2026). This is the formal mechanism by which the model becomes a single audio-text generator rather than a stitched multimodal pipeline.

Sequence construction follows the same principle of unification. Audio understanding tasks are formatted as conversational prompts in which the user instruction is followed by <sound>. TTS tasks place the transcription after a <|text to speech|> control string and require the assistant to produce a codec segment delimited by <speechgen_start> and <speechgen_end>. Text-to-audio tasks analogously use <|text to audio|> followed by <audiogen_start> and <audiogen_end> (Kong et al., 6 Jul 2026). Because all these outputs are generated by the same decoder, the model can in principle interleave text and audio tokens in a single trajectory.

In packed training sequences containing multiple samples, Audex uses square averaging over per-sample loss vectors $L_j$, described as average CE weighted by the square-root of sequence length for stability (Kong et al., 6 Jul 2026). This is a training-stability device rather than a modality-specific loss.

A further generative mechanism is classifier-free guidance (CFG) for audio generation. The paper defines combined logits as

$$p_\theta^{FG}(\cdot \mid x_{<i}) = p_\theta(\cdot \mid x_{<i}) + (\lambda - 1)\left[p_\theta(\cdot \mid x_{<i}) - p_\theta(\cdot \mid \varnothing)\right],$$

where $\lambda$ is the CFG scale (Kong et al., 6 Jul 2026). Rather than masking conditions on the fly, the training recipe pre-selects 10% of speech and audio generation samples, replaces text conditions with padding-token sequences of random length, and trains conditional and unconditional samples together. This lets the model learn both $p_\theta(\cdot \mid x_{<i})$ and $p_\theta(\cdot \mid \varnothing)$ within the same SFT regime (Kong et al., 6 Jul 2026).

4. Data curation and training pipeline

The dataset mixture is large and explicitly quantified: 157.4B audio tokens, 320.5B text tokens, and 477.9B total tokens across 394M samples and 1,092K audio hours (Kong et al., 6 Jul 2026). The task breakdown is:

• Text-to-Speech (TTS): 147M samples, 421K audio hours, 75.8B audio tokens, 5.2B text tokens.
• Text-to-Audio (TTA): 12M samples, 34K audio hours, 24.3B audio tokens, 0.4B text tokens.
• Audio Understanding: 49M samples, 308K audio hours, 27.7B audio tokens, 2.2B text tokens.
• Speech Recognition (ASR): 95M samples, 201K audio hours, 18.1B audio tokens, 4.4B text tokens.
• Speech Translation (AST): 58M samples, 128K audio hours, 11.5B audio tokens, 5.3B text tokens.
• Text SFT: 33M samples and 303.0B text tokens (Kong et al., 6 Jul 2026).

The Text SFT data are identical to Nemotron-Cascade-2 SFT and cover reasoning, math, code, alignment, long context, and agentic tasks, including multilingual tasks sourced from Nemotron-Cascade and Nemotron-3 Nano. Samples longer than 200K tokens are filtered out (Kong et al., 6 Jul 2026).

The supervised training curriculum is multi-stage. After starting from the Nemotron-Cascade-2 SFT checkpoint, the pipeline proceeds through Audio warmup, Audio generation SFT, and Audio generation + audio understanding SFT. In Audio warmup, only the MLP adapters and audio token embeddings are trainable; all LLM weights and text token embeddings are frozen. The paper emphasizes that if text embeddings are updated at this stage, text performance degrades heavily (Kong et al., 6 Jul 2026). In Audio generation SFT, the entire LLM is unfrozen and trained on TTS, TTA, and text-only tasks, but not yet ASR/AST. In the final Audio generation + audio understanding SFT stage, the trainable modules are the LLM + MLP adapters, while the audio encoder remains frozen; the data mixture combines audio understanding, audio generation, and text-only SFT, with text data weight increased to avoid text degradation (Kong et al., 6 Jul 2026).

For the 30B model, the paper reports stage-wise text weights of 44% for Audio warmup, 59% for Audio generation, and 69% for Audio generation + audio understanding, with global batch sizes 64 / 64 / 16 / 64 across the four stages and cosine LR schedules. The maximum LR values are $5\times10^{-5}$ for Text SFT, $2\times10^{-3}$ for Audio warmup, and $10^{-3}$0 for the later audio stages (Kong et al., 6 Jul 2026).

The paper also studies a single-stage SFT after warmup. Although most text and audio metrics remain close, long-context abilities collapse almost completely, with NIAH@256K | 1M at 6.0 | 0.0 for single-stage SFT versus 99.3 | 86.8 for multi-stage SFT. The authors therefore use multi-stage SFT in the final model (Kong et al., 6 Jul 2026).

After SFT, Audex undergoes text-only Cascade RL: instruction-following RL, multi-domain RL, multi-domain on-policy distillation (MOPD), RLHF, and long-context RL. A key reported outcome is that text benchmarks improve significantly through these RL phases while audio benchmarks show marginal or no regression; some audio metrics dip slightly in MOPD and recover in later RL stages (Kong et al., 6 Jul 2026).

5. Capabilities and empirical profile

The core empirical claim is preservation of text intelligence alongside strong audio performance. On selected text benchmarks, Audex and Nemotron-Cascade-2-30B-A3B differ only slightly. For AIME 2025 (no tools | with tools), the backbone scores 92.4 | 98.6, while Audex scores 91.2 | 98.3. For AIME 2026, the comparison is 90.9 | 95.0 versus 89.4 | 96.6. On MMLU-Redux, the scores are 86.3 versus 86.4; on MMLU-Pro, 79.8 versus 78.9; on AA-LCR, 39.1 versus 39.3; on LongBench v2, 40.3 versus 41.3. For NIAH@256K | 1M, the backbone scores 100 | 99.0, while Audex scores 99.4 | 83.4. On agentic benchmarks, τ-Bench is 58.9 versus 57.2, Terminal Bench 2.0 is 21.1 versus 19.1, and SWE Verified is 50.2 versus 48.2 (Kong et al., 6 Jul 2026). The pattern is therefore not one of wholesale text regression, but of small bidirectional movement depending on task.

On audio understanding, the reported MMAU scores for the 30B model are 76.5 / 81.5 / 77.5 / 67.9 / 75.6 for mini/sound/music/speech/average. The model also reports MMAR 63.2, MMSU 63.4, Audio Entailment 95.0, and CMM hallucination 90.3 (Kong et al., 6 Jul 2026).

For speech recognition, Audex reports OpenASR leaderboard (English) average WER 6.82. On Noisy LibriSpeech, the average WER is 2.92. On FLEURS multilingual ASR across DE/ES/FR/IT/PT/RU/KO, the average WER is 5.11 (Kong et al., 6 Jul 2026). For speech translation on FLEURS X→English, the average is 34.0 | 86.9 in BLEU | COMET (Kong et al., 6 Jul 2026).

For TTS, the model reports Seed-TTS-Eval fixed-voice WER 1.70, with prompting WER 2.07 and prompting speaker similarity 45.3. For text-to-audio generation on 10 s segments, Audex reports FD$10^{-3}$1 of 66.9 on AudioCaps and 62.7 on SongDescriber (Kong et al., 6 Jul 2026). The paper notes that the SFT stage is even better than the final RL checkpoint on AudioCaps for this metric, with 60.9 versus 66.9, which indicates that later text-centric RL does not uniformly improve every audio generation metric (Kong et al., 6 Jul 2026).

For speech interaction, Audex reports BigBenchAudio 90.0, close to Gemini 2.5 Flash Native Audio Dialog Thinking 90.7, and a strong VoiceBench profile including AlpacaEval 87.4, CommonEval 82.7, WildVoice 78.2, MMSU 82.6, OpenBookQA 91.0, BBH 88.0, IFEval 88.0, and AdvBench 99.6 (Kong et al., 6 Jul 2026). These results are used to support the paper’s broader claim that speech interaction quality is not bought at the cost of core reasoning and alignment.

6. Limitations, safety, and relation to adjacent Nemotron work

The paper identifies several limitations. Text-to-audio training uses fixed 10 s segments, and generation quality degrades for longer durations; consistency over longer sequences is described as unsolved. Speaker similarity in TTS remains below specialized TTS systems, reflecting objectives tuned more for intelligibility than identity. On MMAR and MMSU, Audex trails some state-of-the-art audio LLMs. At 1M tokens, Audex shows some regression relative to the text-only backbone on NIAH, though performance remains strong in absolute terms (Kong et al., 6 Jul 2026).

The release also foregrounds misuse risk. Audio generation and TTS can be misused for impersonation and deepfake audio, privacy-invasive recording summarization, and harassment and misinformation. For responsible release, the final model removes speech prompting abilities; only fixed-voice TTS remains exposed. The paper is explicit that this reduces but does not eliminate misuse risk (Kong et al., 6 Jul 2026).

Within the broader Nemotron line, Audex should not be conflated with Nemotron 3 Nano Omni, which is an omni-modal model built on Nemotron 3 Nano 30B-A3B and focused on native audio inputs alongside text, images, and video, together with multimodal token-reduction techniques for efficient long audio-video and document workloads (NVIDIA et al., 27 Apr 2026). Nor should it be reduced to a generic Nemotron 3 derivative: the direct Audex paper defines it specifically as a unified audio-text model built on Nemotron-Cascade-2-30B-A3B (Kong et al., 6 Jul 2026).

At the same time, adjacent Nemotron work provides important family context. Earlier Nemotron 3 Nano 30B-A3B work established a Mixture-of-Experts hybrid Mamba-Transformer model with 31.6B total parameters, 3.2B active parameters per forward pass, 25T pretraining tokens, and support up to 1M tokens, along with strong throughput claims relative to similarly sized open models (NVIDIA et al., 23 Dec 2025). A separate TwoTower line showed that a 30B A3B Nemotron backbone can be adapted into a block-wise autoregressive diffusion model with a frozen AR context tower and a trainable denoiser tower, retaining 98.7% of the autoregressive baseline’s quality while offering 2.42X higher wall-clock generation throughput (Reda et al., 25 Jun 2026). Because that work is framed around adapting “any pretrained autoregressive LLM,” a plausible implication is that analogous diffusion-style or hybrid decoding ideas could eventually be explored for Audex as well, although Audex itself is not named in the TwoTower paper (Reda et al., 25 Jun 2026).