---
title: 'Audex: Unified Audio-Text 30B Model'
url: https://www.emergentmind.com/topics/nemotron-labs-audex-30b-a3b-audex
type: topic
---

# Audex: Unified Audio-Text 30B Model

Nemotron-Labs-Audex-30B-A3B, usually abbreviated **Audex**, is a unified audio-text large language model that is built on **Nemotron-Cascade-2-30B-A3B**, a strong text-only MoE LLM, and is designed to understand, reason about, and generate both audio and speech while preserving the reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only backbone with *marginal or no regression* [2607.05196]. Its defining architectural choice is a **single Transformer decoder** over a unified token space: audio inputs are encoded and projected into the text embedding space, while text tokens and quantized audio output tokens are treated uniformly during generation, enabling audio-text fusion, multimodal generation, and compatibility with standard LLM training and inference infrastructure [2607.05196].

## 1. Concept and scope

Audex is presented as a model for **unified audio intelligence**, where “audio intelligence” includes understanding, reasoning about, and generating both audio and speech [2607.05196]. The stated target is not merely ASR or speech translation, but a broader set of capabilities: **audio understanding, speech recognition, speech translation, text-to-speech, audio generation, and speech-to-speech generation**. In parallel, the system is explicitly designed to avoid a common multimodal failure mode: adding multimodal I/O should not significantly degrade text reasoning, knowledge, alignment, long-context behavior, or agentic performance [2607.05196].

This positioning distinguishes Audex from narrowly specialized speech or audio systems. It is neither only an encoder-decoder speech model nor only a text LLM with an auxiliary audio front end. Instead, it is trained so that text tokens and codec tokens participate in a single autoregressive generative process. A central empirical claim of the work is therefore twofold: the model adds strong audio capabilities while preserving “very compelling” text capabilities of the backbone with only small and often symmetric benchmark deltas [2607.05196].

The model is released for open research, together with a smaller **Audex-2B** dense Transformer variant trained with the same recipe [2607.05196].

## 2. Backbone and unified architecture

Audex uses a 30B-parameter MoE backbone initialized from the **Nemotron-Cascade-2 SFT checkpoint**. The architecture retained in Audex is specified as a **52-layer hybrid Mamba2–Transformer** with **model dimension 2688**, **128 routable experts**, **6 activated per token**, and an original vocabulary of **131,072 tokens** [2607.05196].

| Component | Specification | Role |
|---|---|---|
| Backbone | 52-layer hybrid Mamba2–Transformer; model dimension 2688 | Text-only core LLM |
| MoE | 128 routable experts; 6 activated per token | Sparse conditional computation |
| Audio input path | AF-Whisper encoder + 2-layer MLP adapters | Maps waveform features into LLM embeddings |
| Audio output path | X-Codec2 for speech; X-Codec for non-speech | Produces discrete audio tokens |
| Vocabulary | 205,312 padded tokens | Unified text + audio token space |

The decisive architectural claim is that Audex keeps a **single decoder-only LLM**. Text tokens remain in the original subword/BPE vocabulary. Audio output is represented as discrete codec tokens appended to the vocabulary. Audio input, by contrast, is represented as continuous embeddings produced by an external encoder and then projected into the LLM hidden space [2607.05196].

For audio inputs, Audex uses the **AF-Whisper** encoder from Audio Flamingo 3. It takes **16 kHz waveform** input and produces **1280-dim features at 25 Hz**, yielding **750 frames for each 30 s**. Audio longer than **30 s** is chunked, and the final partial window is padded to **30 s**. A **2-layer MLP** maps these **1280-dim** features into the model’s **2688-dim** embedding space. In prompts, audio is represented by a `<sound>` token that is internally expanded to `<so_start>`, a sequence of `<so_embedding>` tokens, and `<so_end>`; these embeddings then participate in self-attention with text tokens, with no separate decoder [2607.05196].

For audio outputs, Audex uses two codec families. **Speech** is tokenized with **X-Codec2**, a speech-focused codec with **16 kHz** input, **50 Hz** frame rate, **single-layer finite scalar quantization (FSQ)**, **codebook size 65,536**, and speech tokens of the form `<speechcodec_{id}>`. **General audio** uses **X-Codec**, again at **16 kHz** and **50 Hz**, with **8-layer residual vector quantization (RVQ)** and **1,024 codes per layer**; to reduce token rate, Audex uses only the first **4 RVQ layers (RVQ4)** during training, giving an effective rate of **200 tokens/s**, and reserves **8,192** non-speech codec indices as `<audiocodec_{id}>` [2607.05196].

The vocabulary extension is explicit: **131,072 text tokens + 65,536 speech codec tokens + 8,192 audio codec tokens + 5 special generation tokens = 204,805**, padded to **205,312** for Megatron tensor parallelism. Newly added audio-token embeddings are initialized as $\mathcal{N}(0, 0.02^2)$ [2607.05196].

Audex retains the backbone’s MoE routing and applies it uniformly across modalities. The routing configuration is **Top-K**, with **128 experts** and **6 activated per token**, using **auxiliary-loss-free load balancing** plus a small standard load-balancing loss, with **routing scale 2.5**, **update rate $10^{-3}$**, and **load-balancing loss coefficient $10^{-6}$** [2607.05196]. This means that audio and text tokens pass through the same routed expert system rather than through modality-specific expert partitions.

## 3. Tokenization, sequence construction, and objective

Audex’s training objective is the standard **next-token objective over the unified vocabulary**, where the prediction space contains both text tokens and codec tokens [2607.05196]. This is the formal mechanism by which the model becomes a single audio-text generator rather than a stitched multimodal pipeline.

Sequence construction follows the same principle of unification. Audio understanding tasks are formatted as conversational prompts in which the user instruction is followed by `<sound>`. TTS tasks place the transcription after a `<|text to speech|>` control string and require the assistant to produce a codec segment delimited by `<speechgen_start>` and `<speechgen_end>`. Text-to-audio tasks analogously use `<|text to audio|>` followed by `<audiogen_start>` and `<audiogen_end>` [2607.05196]. Because all these outputs are generated by the same decoder, the model can in principle interleave text and audio tokens in a single trajectory.

In packed training sequences containing multiple samples, Audex uses **square averaging** over per-sample loss vectors $L_j$, described as average CE weighted by the square-root of sequence length for stability [2607.05196]. This is a training-stability device rather than a modality-specific loss.

A further generative mechanism is **classifier-free guidance (CFG)** for audio generation. The paper defines combined logits as
$$
p_\theta^{FG}(\cdot \mid x_{<i}) = p_\theta(\cdot \mid x_{<i}) + (\lambda - 1)\left[p_\theta(\cdot \mid x_{<i}) - p_\theta(\cdot \mid \varnothing)\right],
$$
where $\lambda$ is the CFG scale [2607.05196]. Rather than masking conditions on the fly, the training recipe pre-selects **10%** of speech and audio generation samples, replaces text conditions with padding-token sequences of random length, and trains conditional and unconditional samples together. This lets the model learn both $p_\theta(\cdot \mid x_{<i})$ and $p_\theta(\cdot \mid \varnothing)$ within the same SFT regime [2607.05196].

## 4. Data curation and training pipeline

The dataset mixture is large and explicitly quantified: **157.4B audio tokens**, **320.5B text tokens**, and **477.9B total tokens** across **394M** samples and **1,092K** audio hours [2607.05196]. The task breakdown is:

- **Text-to-Speech (TTS)**: **147M** samples, **421K** audio hours, **75.8B** audio tokens, **5.2B** text tokens.
- **Text-to-Audio (TTA)**: **12M** samples, **34K** audio hours, **24.3B** audio tokens, **0.4B** text tokens.
- **Audio Understanding**: **49M** samples, **308K** audio hours, **27.7B** audio tokens, **2.2B** text tokens.
- **Speech Recognition (ASR)**: **95M** samples, **201K** audio hours, **18.1B** audio tokens, **4.4B** text tokens.
- **Speech Translation (AST)**: **58M** samples, **128K** audio hours, **11.5B** audio tokens, **5.3B** text tokens.
- **Text SFT**: **33M** samples and **303.0B** text tokens [2607.05196].

The **Text SFT** data are identical to Nemotron-Cascade-2 SFT and cover reasoning, math, code, alignment, long context, and agentic tasks, including multilingual tasks sourced from Nemotron-Cascade and Nemotron-3 Nano. Samples longer than **200K tokens** are filtered out [2607.05196].

The supervised training curriculum is **multi-stage**. After starting from the Nemotron-Cascade-2 SFT checkpoint, the pipeline proceeds through **Audio warmup**, **Audio generation SFT**, and **Audio generation + audio understanding SFT**. In **Audio warmup**, only the **MLP adapters** and **audio token embeddings** are trainable; all LLM weights and text token embeddings are frozen. The paper emphasizes that if text embeddings are updated at this stage, text performance degrades heavily [2607.05196]. In **Audio generation SFT**, the entire LLM is unfrozen and trained on **TTS**, **TTA**, and **text-only tasks**, but not yet ASR/AST. In the final **Audio generation + audio understanding SFT** stage, the trainable modules are the **LLM + MLP adapters**, while the audio encoder remains frozen; the data mixture combines audio understanding, audio generation, and text-only SFT, with text data weight increased to avoid text degradation [2607.05196].

For the **30B** model, the paper reports stage-wise text weights of **44%** for Audio warmup, **59%** for Audio generation, and **69%** for Audio generation + audio understanding, with global batch sizes **64 / 64 / 16 / 64** across the four stages and cosine LR schedules. The maximum LR values are **$5\times10^{-5}$** for Text SFT, **$2\times10^{-3}$** for Audio warmup, and **$2\times10^{-5}$** for the later audio stages [2607.05196].

The paper also studies a **single-stage SFT** after warmup. Although most text and audio metrics remain close, **long-context abilities collapse almost completely**, with **NIAH@256K | 1M** at **6.0 | 0.0** for single-stage SFT versus **99.3 | 86.8** for multi-stage SFT. The authors therefore use multi-stage SFT in the final model [2607.05196].

After SFT, Audex undergoes **text-only Cascade RL**: **instruction-following RL**, **multi-domain RL**, **multi-domain on-policy distillation (MOPD)**, **RLHF**, and **long-context RL**. A key reported outcome is that text benchmarks improve significantly through these RL phases while audio benchmarks show **marginal or no regression**; some audio metrics dip slightly in MOPD and recover in later RL stages [2607.05196].

## 5. Capabilities and empirical profile

The core empirical claim is preservation of text intelligence alongside strong audio performance. On selected text benchmarks, Audex and **Nemotron-Cascade-2-30B-A3B** differ only slightly. For **AIME 2025 (no tools | with tools)**, the backbone scores **92.4 | 98.6**, while Audex scores **91.2 | 98.3**. For **AIME 2026**, the comparison is **90.9 | 95.0** versus **89.4 | 96.6**. On **MMLU-Redux**, the scores are **86.3** versus **86.4**; on **MMLU-Pro**, **79.8** versus **78.9**; on **AA-LCR**, **39.1** versus **39.3**; on **LongBench v2**, **40.3** versus **41.3**. For **NIAH@256K | 1M**, the backbone scores **100 | 99.0**, while Audex scores **99.4 | 83.4**. On agentic benchmarks, **τ-Bench** is **58.9** versus **57.2**, **Terminal Bench 2.0** is **21.1** versus **19.1**, and **SWE Verified** is **50.2** versus **48.2** [2607.05196]. The pattern is therefore not one of wholesale text regression, but of small bidirectional movement depending on task.

On audio understanding, the reported **MMAU** scores for the 30B model are **76.5 / 81.5 / 77.5 / 67.9 / 75.6** for mini/sound/music/speech/average. The model also reports **MMAR 63.2**, **MMSU 63.4**, **Audio Entailment 95.0**, and **CMM hallucination 90.3** [2607.05196].

For speech recognition, Audex reports **OpenASR leaderboard (English)** average **WER 6.82**. On **Noisy LibriSpeech**, the average WER is **2.92**. On **FLEURS multilingual ASR** across **DE/ES/FR/IT/PT/RU/KO**, the average WER is **5.11** [2607.05196]. For speech translation on **FLEURS X→English**, the average is **34.0 | 86.9** in **BLEU | COMET** [2607.05196].

For TTS, the model reports **Seed-TTS-Eval** fixed-voice **WER 1.70**, with prompting WER **2.07** and prompting speaker similarity **45.3**. For text-to-audio generation on **10 s** segments, Audex reports **FD\(_{\text{openl3}}\)** of **66.9** on **AudioCaps** and **62.7** on **SongDescriber** [2607.05196]. The paper notes that the **SFT stage** is even better than the final RL checkpoint on AudioCaps for this metric, with **60.9** versus **66.9**, which indicates that later text-centric RL does not uniformly improve every audio generation metric [2607.05196].

For speech interaction, Audex reports **BigBenchAudio 90.0**, close to **Gemini 2.5 Flash Native Audio Dialog Thinking 90.7**, and a strong **VoiceBench** profile including **AlpacaEval 87.4**, **CommonEval 82.7**, **WildVoice 78.2**, **MMSU 82.6**, **OpenBookQA 91.0**, **BBH 88.0**, **IFEval 88.0**, and **AdvBench 99.6** [2607.05196]. These results are used to support the paper’s broader claim that speech interaction quality is not bought at the cost of core reasoning and alignment.

## 6. Limitations, safety, and relation to adjacent Nemotron work

The paper identifies several limitations. **Text-to-audio** training uses fixed **10 s** segments, and generation quality degrades for longer durations; consistency over longer sequences is described as unsolved. **Speaker similarity** in TTS remains below specialized TTS systems, reflecting objectives tuned more for intelligibility than identity. On **MMAR** and **MMSU**, Audex trails some state-of-the-art audio LLMs. At **1M** tokens, Audex shows some regression relative to the text-only backbone on **NIAH**, though performance remains strong in absolute terms [2607.05196].

The release also foregrounds misuse risk. Audio generation and TTS can be misused for **impersonation and deepfake audio**, **privacy-invasive recording summarization**, and **harassment and misinformation**. For responsible release, the final model removes **speech prompting** abilities; only **fixed-voice TTS** remains exposed. The paper is explicit that this reduces but does not eliminate misuse risk [2607.05196].

Within the broader Nemotron line, Audex should not be conflated with **Nemotron 3 Nano Omni**, which is an omni-modal model built on **Nemotron 3 Nano 30B-A3B** and focused on native **audio inputs alongside text, images, and video**, together with multimodal token-reduction techniques for efficient long audio-video and document workloads [2604.24954]. Nor should it be reduced to a generic Nemotron 3 derivative: the direct Audex paper defines it specifically as a unified audio-text model built on **Nemotron-Cascade-2-30B-A3B** [2607.05196].

At the same time, adjacent Nemotron work provides important family context. Earlier **Nemotron 3 Nano 30B-A3B** work established a **Mixture-of-Experts hybrid Mamba-Transformer** model with **31.6B total parameters**, **3.2B active parameters per forward pass**, **25T** pretraining tokens, and support up to **1M tokens**, along with strong throughput claims relative to similarly sized open models [2512.20848]. A separate **TwoTower** line showed that a **30B A3B** Nemotron backbone can be adapted into a **block-wise autoregressive diffusion model** with a frozen AR context tower and a trainable denoiser tower, retaining **98.7%** of the autoregressive baseline’s quality while offering **2.42X** higher wall-clock generation throughput [2606.26493]. Because that work is framed around adapting “any pretrained autoregressive language model,” a plausible implication is that analogous diffusion-style or hybrid decoding ideas could eventually be explored for Audex as well, although Audex itself is not named in the TwoTower paper [2606.26493].

Source: https://www.emergentmind.com/topics/nemotron-labs-audex-30b-a3b-audex