---
title: 'UniTok-Audio: Unified Audio Generation'
url: https://www.emergentmind.com/topics/unitok-audio
type: topic
---

# UniTok-Audio: Unified Audio Generation

UniTok-Audio is a unified audio generation framework that casts multiple time-aligned audio tasks as conditional autoregressive prediction of discrete codec tokens. It is centered on the premise that many audio problems share a common structure in which a condition input, often audio and sometimes text, is temporally aligned with the target audio to be generated. The framework extracts continuous feature of conditions to generates discrete tokens of target audio in an autoregressive manner, uses a special task identifier token to unify different learning patterns of multiple tasks in a single framework, and employs a dual-stream audio codec involving acoustic and semantic branch for high-fidelity waveform reconstruction [2510.26372].

## 1. Problem setting and task scope

UniTok-Audio is defined around **time-aligned (TA) audio generation tasks**, where the input and output have frame-level temporal correspondence. In the formulation adopted by the framework, apparently different applications are treated as variants of a single conditional generation problem over codec-token sequences. The supported condition and output types are **audio input \(\rightarrow\) audio output** for speech restoration, target speaker extraction, speech separation, and voice conversion, and **text + audio input \(\rightarrow\) audio output** for language-queried audio source separation [2510.26372].

The operational modes are separated by task identifier tokens. This is necessary because several tasks share similar condition types while requiring distinct mappings. Both target speaker extraction and voice conversion, for example, use reference speech plus another speech input, but the intended output distributions are different. The task token therefore functions as a compact control code specifying which conditional distribution is to be learned.

The framework covers five headline tasks. **Speech restoration (SR)** maps degraded speech to clean speech. **Target speaker extraction (TSE)** extracts from a mixture the speech belonging to the same speaker as the reference speech. **Speech separation (SS)** is handled indirectly, because the model outputs one track at a time. The paper therefore introduces an auxiliary **rTSE** mode that extracts the timbre-mismatched speech component relative to the reference. For 2-speaker separation, inference first uses SR mode to obtain the main or high-energy speaker, then uses that output as reference and runs rTSE mode to recover the remaining speaker. **Voice conversion (VC)** changes source-speech timbre to match reference speech while preserving linguistic content. **Language-queried audio source separation (LASS)** extracts the audio component in a mixture that matches a text query [2510.26372].

This task design gives UniTok-Audio a narrower but technically coherent scope than universal audio foundation models aimed simultaneously at understanding and generation. Its unification target is not arbitrary audio-language behavior, but a family of temporally aligned conditional generation problems.

## 2. Unified autoregressive formulation

The central modeling decision is asymmetric: **conditions remain continuous**, while **targets are discrete**. Audio conditions are encoded as continuous features by HuBERT; text conditions are encoded as continuous embeddings by T5-base; target waveforms are converted into discrete H-Codec tokens and then generated autoregressively. The paper argues that continuous condition embeddings preserve richer semantic and acoustic detail than discretizing the inputs too early, while discrete targets are a natural fit for language-model-style sequence generation [2510.26372].

The task-specific objectives are standard teacher-forced negative log-likelihood losses over multi-layer codec tokens. For speech restoration, the loss is

\[
\mathcal{L}_{\rm SR} = - \sum_{t=1}^{L} \sum_{i=1}^{4} \log P\left(o^i_t \mid {\rm T_{SR}, I, E_i, S}, o_{<t}; \theta \right).
\]

For target speaker extraction, the conditioning prefix additionally contains reference-speech features:

\[
\mathcal{L}_{\rm TSE} = - \sum_{t=1}^{L} \sum_{i=1}^{4} \log P\left(o^i_t \mid {\rm T_{TSE}, R, E_r, I, E_i, S}, o_{<t}; \theta \right).
\]

For language-queried separation, the text embedding enters through a caption-start token and caption representation:

\[
\mathcal{L}_{\rm LASS} = - \sum_{t=1}^{L} \sum_{i=1}^{4} \log P\left(o^i_t \mid {\rm T_{LASS}, C, E_t, I, E_i, S}, o_{<t}; \theta \right).
\]

The VC objective is stated to have the same form as TSE, with different semantics for source and reference speech. A plausible summary of the shared training view is that all modes minimize the same autoregressive next-token loss under different task tokens and different condition-prefix layouts; this suggests that the unification is accomplished more by sequence serialization and task control than by task-specific architectural branching.

Token scheduling uses the **delay pattern** from MusicGen rather than full flattening of every codec layer into a single 1D stream. H-Codec produces 4-layer acoustic tokens and 4-layer semantic tokens. These are interleaved across time steps into a representation

\[
{\rm E}_c \in \mathbb{Z}^{T \times 4},
\]

with an effective frame rate of **50 Hz**. The language model uses **4 embedding layers** for the 4 token layers, sums the embeddings as transformer input, and uses **4 output heads** to predict the 4-layer logits at the next step. The delay pattern keeps sequence length manageable while allowing higher-layer token prediction to condition on lower-layer tokens [2510.26372].

## 3. Architecture and the dual-stream H-Codec

UniTok-Audio contains four principal modules: condition encoders, modality adapters, a decoder-only autoregressive generator, and the H-Codec tokenizer/decoder. The condition encoders are a HuBERT-based audio encoder and a T5-base text encoder. Two linear layers project HuBERT and T5 features into the LLaMA-space used by the generator. The autoregressive backbone is a decoder-only **LLaMA-style transformer** with **16 layers**, **16 attention heads**, **hidden size 1024**, and **481M trainable parameters** [2510.26372].

A defining component is **H-Codec**, which factorizes audio representation into an **acoustic branch** and a **semantic branch**. The acoustic branch takes waveform input \(\boldsymbol{x} \in \mathbb{R}^{n}\), uses an Encodec-style acoustic encoder, and quantizes acoustic features with a **4-layer RVQ** at **25 Hz**. The semantic branch extracts SSL features from a pretrained HuBERT by averaging outputs from all transformer layers, processes them with a semantic encoder, and quantizes the resulting semantic representation with another **4-layer RVQ**. Rather than fusing acoustic and semantic features before quantization, H-Codec quantizes them separately with distinct codebooks [2510.26372].

Waveform reconstruction concatenates quantized acoustic and semantic features along the hidden dimension, passes the combined representation to the acoustic decoder, and uses an **ISTFT head following Vocos** to reconstruct waveform. A semantic decoder reconstructs the SSL feature from the quantized semantic representation, enforcing semantic retention. Important codec parameters are a **downsampling ratio of 640**, **4 RVQ layers** per branch, **codebook size 1024** per layer, **codebook dimension 512**, and **total bitrate 2000 bps**. Because the acoustic and semantic token streams are interleaved in generation, the effective generation schedule operates at **50 Hz** [2510.26372].

The codec generator loss is

\[
\mathcal{L}_{gen} = \lambda_{commit} \mathcal{L}_{commit} + \lambda_{mel} \mathcal{L}_{mel} + \lambda_{adv} \mathcal{L}_{adv} + \lambda_{fm} \mathcal{L}_{fm} + \lambda_{aux} \mathcal{L}_{aux}.
\]

Here \(\mathcal{L}_{commit}\) is the VQ commitment loss, \(\mathcal{L}_{mel}\) is the mel-spectrogram reconstruction loss, \(\mathcal{L}_{adv}\) is the adversarial generator loss, \(\mathcal{L}_{fm}\) is the feature matching loss, and \(\mathcal{L}_{aux}\) is the MSE loss on reconstructed SSL features. The discriminator loss follows the hinge-loss setup from SoundStream, with a discriminator stack consisting of a multi-period discriminator, a multi-resolution discriminator, and a sub-band complex STFT discriminator. A perceptual loss is added during the final **100k** training steps [2510.26372].

This architecture places UniTok-Audio between classical acoustic codecs and semantic token pipelines. It does not rely on a single undifferentiated code stream, but it also does not split generation across separate semantic and acoustic generators. The codec preserves factorization, while the language model learns dependencies between the two branches through one autoregressive token-generation process.

## 4. Training methodology and data construction

Training proceeds in two conceptually separate phases: H-Codec training followed by UniTok-Audio training. H-Codec is trained on mixed-domain audio comprising **speech: VoxBox, about 100k hours; music: FMA-full + MUSDB18-HQ, about 8k hours; general audio: AudioSet + WavCaps, about 13k hours**. All audio is resampled to **16 kHz**, training uses random **5-second** crops, optimization uses **AdamW**, the initial learning rate is \(2 \times 10^{-4}\), cosine decay is applied, training runs for **600k** steps, and the perceptual loss is enabled in the final **100k** steps [2510.26372].

UniTok-Audio training uses separate data constructions for each task family. For speech-related tasks, clean speech comes from VoxBox subsets including **LibriSpeech, MLS English, Emilia_ZH**, totaling about **3.8k hours**. The noise and corruption pool includes **DNS, FSD50K, WHAM!, DESED, DEMAND, MUSAN, DISCO, MUSDB18-HQ, TUT Urban Acoustic Scenes**, a **460 hours** noise corpus, and **60k RIRs** from **SLR28**. For audio-caption tasks, the data include **WavCaps, CLAP_FreeSound, VGGSound, internal data**, totaling about **40k hours**. For voice conversion, an external voice conversion model is used to generate **6k hours** of fixed timbre-perturbed training data [2510.26372].

The data simulation pipeline is dynamic rather than fixed. For SR, degradations are applied probabilistically: additive noise with probability **0.5** and SNR uniform in \([-15, 20]\) dB, reverberation **0.4**, clipping **0.3**, bandwidth limitation **0.3** with cutoff in \(\{2,4\}\) kHz, packet loss **0.3** with rate uniform in \([0.05,0.25]\), and interfering speaker **0.2** with SIR uniform in \([15,25]\) dB. For TSE and rTSE, the interference-speaker probability is **1.0**, SIR is sampled in \([-5,5]\) dB, and the distortion chain is also applied. For LASS, target audio is mixed with another random audio at SIR sampled from \([-5,20]\) dB [2510.26372].

Each example is transformed into condition embeddings from the audio or text encoder and a target H-Codec token sequence for the desired waveform. The language model is then trained with **teacher forcing**: at each step it predicts the next delayed multi-layer token while conditioning on the ground-truth previous tokens. For the multi-task omni model, one of the five modes is **randomly selected per batch**, which is the paper’s explicit task-mixing strategy [2510.26372].

## 5. Reconstruction quality, downstream performance, and scaling

The paper evaluates H-Codec separately from the unified generator. On **LibriSpeech test-clean**, H-Codec at **25+25 FPS**, **4 layers**, and **2000 bps** reports **PESQ 2.99**, **STOI 0.94**, **UTMOS 4.06**, **SPK-SIM 0.84**, and **WER 3.18**. On cross-domain reconstruction it reports **speech mel/STFT loss 0.3394 / 0.1033**, **music 0.5158 / 0.1667**, and **audio 1.2512 / 0.4070**. The paper states that these are the best among the general-audio codecs listed, and uses them as evidence that H-Codec provides a stronger discrete representation than prior codecs for downstream generation [2510.26372].

The downstream results indicate that UniTok-Audio is generally competitive with strong task-specific and unified baselines across the five time-aligned tasks.

| Task | Dataset | UniTok-Audio result |
|---|---|---|
| Speech restoration | DNS 2020 | With Reverb: SIG 3.67, BAK 4.12, OVRL 3.42; No Reverb: SIG 3.66, BAK 4.15, OVRL 3.44 |
| PLC restoration | ICASSP 2022 PLC blind test | OVRL 3.35, PLCMOS 4.58 |
| Target speaker extraction | Libri2Mix clean test | SIG 3.62, BAK 4.05, OVRL 3.32, NISQA 4.00, SIM 0.95 |
| Speech separation | Libri2Mix noisy / WSJ0-2mix | OVRL 3.25 / 3.26 |
| Voice conversion | VCTK-based evaluation | WER 4.23, SIM 0.50, DNSMOS 3.51, NISQA 4.51 |
| LASS | DCASE 2024 LASS validation | Single-task: FAD 0.68, CLAPScore 28.85, CLAPScore\(_A\) 65.56; Omni: FAD 1.48, CLAPScore 26.21, CLAPScore\(_A\) 61.21 |

For SR, the omni model is competitive with the best discrete generative baselines on DNS 2020 and exceeds all listed baselines on the PLC benchmark, where it reaches **OVRL 3.35** and **PLCMOS 4.58**. For TSE, the omni model is roughly on par with the strongest generative baselines, though not clearly superior. For SS, the iterative SR + rTSE strategy yields **OVRL 3.25** on Libri2Mix and **3.26** on WSJ0-2mix, which the paper highlights because speech separation is usually difficult for unified token-generation systems with single-track output constraints. For VC, the omni model is strong, but the single-task WavLM-based variant is better, reaching **WER 3.02**, **SIM 0.51**, **DNSMOS 3.46**, and **NISQA 4.46**. For LASS, the single-task model is substantially better than the omni model, which the paper attributes to speech-vs-general-audio domain mismatch [2510.26372].

Ablative comparisons suggest that unified training works best when tasks share similar structure and domain. The paper reports that **SR** and **TSE** remain essentially as strong in the omni setting as in single-task training, whereas **VC** and **LASS** degrade. It also reports scaling behavior on VC: a **109M** model yields **WER 5.38**, **SIM 0.42**, **DNSMOS 3.41**, **NISQA 4.30**; a **481M** model yields **WER 3.02**, **SIM 0.51**, **DNSMOS 3.46**, **NISQA 4.46**; and a **1.3B** model yields **WER 2.10**, **SIM 0.61**, **DNSMOS 3.61**, **NISQA 4.54**. This suggests that the framework is scalable in the ordinary model-size sense, even though its task compatibility remains uneven across domains [2510.26372].

## 6. Position in the unified audio-token literature

UniTok-Audio belongs to a broader line of work seeking unified audio interfaces, but it occupies a distinct niche. **UniAudio** formulated many audio generation tasks as serialized next-token prediction over codec tokens, using a multi-scale Transformer to manage long RVQ sequences; its strongest contribution was unified sequence modeling rather than a fundamentally new tokenizer [2310.00704]. **UniCodec** and **AUV** pursued LM-friendly **single-layer, single-codebook** universal codecs, using partitioned or nested codebooks and domain-sensitive training to support speech, music, and sound with one token stream [2502.20067] [2509.21968]. **UniTTS** pushed toward a single-codebook regime in which the language model directly handles one stream of discrete audio tokens carrying comprehensive audio information, but it remained framed as an end-to-end TTS system rather than a broad audio understanding interface [2505.17426].

Other contemporaneous systems broadened the notion of unification beyond time-aligned generation. **Ming-UniAudio** introduced **MingTok-Audio**, a **continuous, VAE-based unified speech tokenizer** intended to support speech understanding, generation, and free-form editing without timestamps [2511.05516]. **UniAudio 2.0** proposed **ReasoningCodec**, which factorizes audio into reasoning tokens and reconstruction tokens, treating unified tokenization as a first-class design problem for an audio foundation model [2602.04683]. **UniAudio-Token**, from Yuhan Song, Linhao Zhang, Aiwei Liu, Chuhan Wu, Sijun Zhang, Wei Jia, Yuan Liu, Houfeng Wang, and Xiao Zhou at Peking University and Tencent WeChat AI, argued that a semantic speech tokenizer can be turned into a unified audio tokenizer without abandoning the compact single-codebook interface favored by Audio-LLMs [2605.31521]. **AVTok** extended unification across modalities by placing audio and video into a shared token space with a unified codebook for holistic audio-video generation [2606.30811].

Against that backdrop, UniTok-Audio is best understood as a **discrete codec-token autoregressive framework for aligned conditional generation**. It is not primarily a universal audio understanding interface, a single-codebook universal codec, or a multimodal shared-codebook tokenizer. Its characteristic combination is continuous condition features, task-token-controlled conditional generation, and a dual-stream acoustic/semantic codec.

The framework’s limitations are correspondingly specific. The model only supports **one-track output**, so speech separation requires iterative inference through SR and rTSE rather than native multi-track generation. **VC** and **LASS** degrade under omni training, indicating task interference or a speech-vs-general-audio domain gap. The paper also notes that **speaker similarity in generative methods can be limited by codec reconstruction fidelity**, which locates a remaining bottleneck in the codec rather than the autoregressive generator. More broadly, the results indicate that unified training is effective when tasks share structure, but that broader unification across heterogeneous audio domains remains unresolved [2510.26372].

Source: https://www.emergentmind.com/topics/unitok-audio