UniTok-Audio: Unified Audio Generation

• UniTok-Audio is a unified framework that generates audio by converting continuous conditions to discrete codec tokens, handling various time-aligned tasks.
• It employs a dual-stream H-Codec that separates acoustic and semantic features, enabling high-fidelity waveform reconstruction under a common model.
• The framework demonstrates competitive performance across tasks like speech restoration, target speaker extraction, voice conversion, and language-queried source separation.

UniTok-Audio is a unified audio generation framework that casts multiple time-aligned audio tasks as conditional autoregressive prediction of discrete codec tokens. It is centered on the premise that many audio problems share a common structure in which a condition input, often audio and sometimes text, is temporally aligned with the target audio to be generated. The framework extracts continuous feature of conditions to generates discrete tokens of target audio in an autoregressive manner, uses a special task identifier token to unify different learning patterns of multiple tasks in a single framework, and employs a dual-stream audio codec involving acoustic and semantic branch for high-fidelity waveform reconstruction (Liu et al., 30 Oct 2025).

1. Problem setting and task scope

UniTok-Audio is defined around time-aligned (TA) audio generation tasks, where the input and output have frame-level temporal correspondence. In the formulation adopted by the framework, apparently different applications are treated as variants of a single conditional generation problem over codec-token sequences. The supported condition and output types are audio input $\rightarrow$ audio output for speech restoration, target speaker extraction, speech separation, and voice conversion, and text + audio input $\rightarrow$ audio output for language-queried audio source separation (Liu et al., 30 Oct 2025).

The operational modes are separated by task identifier tokens. This is necessary because several tasks share similar condition types while requiring distinct mappings. Both target speaker extraction and voice conversion, for example, use reference speech plus another speech input, but the intended output distributions are different. The task token therefore functions as a compact control code specifying which conditional distribution is to be learned.

The framework covers five headline tasks. Speech restoration (SR) maps degraded speech to clean speech. Target speaker extraction (TSE) extracts from a mixture the speech belonging to the same speaker as the reference speech. Speech separation (SS) is handled indirectly, because the model outputs one track at a time. The paper therefore introduces an auxiliary rTSE mode that extracts the timbre-mismatched speech component relative to the reference. For 2-speaker separation, inference first uses SR mode to obtain the main or high-energy speaker, then uses that output as reference and runs rTSE mode to recover the remaining speaker. Voice conversion (VC) changes source-speech timbre to match reference speech while preserving linguistic content. Language-queried audio source separation (LASS) extracts the audio component in a mixture that matches a text query (Liu et al., 30 Oct 2025).

This task design gives UniTok-Audio a narrower but technically coherent scope than universal audio foundation models aimed simultaneously at understanding and generation. Its unification target is not arbitrary audio-language behavior, but a family of temporally aligned conditional generation problems.

2. Unified autoregressive formulation

The central modeling decision is asymmetric: conditions remain continuous, while targets are discrete. Audio conditions are encoded as continuous features by HuBERT; text conditions are encoded as continuous embeddings by T5-base; target waveforms are converted into discrete H-Codec tokens and then generated autoregressively. The paper argues that continuous condition embeddings preserve richer semantic and acoustic detail than discretizing the inputs too early, while discrete targets are a natural fit for language-model-style sequence generation (Liu et al., 30 Oct 2025).

The task-specific objectives are standard teacher-forced negative log-likelihood losses over multi-layer codec tokens. For speech restoration, the loss is

$$\mathcal{L}_{\rm SR} = - \sum_{t=1}^{L} \sum_{i=1}^{4} \log P\left(o^i_t \mid {\rm T_{SR}, I, E_i, S}, o_{<t}; \theta \right).$$

For target speaker extraction, the conditioning prefix additionally contains reference-speech features:

$$\mathcal{L}_{\rm TSE} = - \sum_{t=1}^{L} \sum_{i=1}^{4} \log P\left(o^i_t \mid {\rm T_{TSE}, R, E_r, I, E_i, S}, o_{<t}; \theta \right).$$

For language-queried separation, the text embedding enters through a caption-start token and caption representation:

$$\mathcal{L}_{\rm LASS} = - \sum_{t=1}^{L} \sum_{i=1}^{4} \log P\left(o^i_t \mid {\rm T_{LASS}, C, E_t, I, E_i, S}, o_{<t}; \theta \right).$$

The VC objective is stated to have the same form as TSE, with different semantics for source and reference speech. A plausible summary of the shared training view is that all modes minimize the same autoregressive next-token loss under different task tokens and different condition-prefix layouts; this suggests that the unification is accomplished more by sequence serialization and task control than by task-specific architectural branching.

Token scheduling uses the delay pattern from MusicGen rather than full flattening of every codec layer into a single 1D stream. H-Codec produces 4-layer acoustic tokens and 4-layer semantic tokens. These are interleaved across time steps into a representation

${\rm E}_c \in \mathbb{Z}^{T \times 4},$

with an effective frame rate of 50 Hz. The LLM uses 4 embedding layers for the 4 token layers, sums the embeddings as transformer input, and uses 4 output heads to predict the 4-layer logits at the next step. The delay pattern keeps sequence length manageable while allowing higher-layer token prediction to condition on lower-layer tokens (Liu et al., 30 Oct 2025).

3. Architecture and the dual-stream H-Codec

UniTok-Audio contains four principal modules: condition encoders, modality adapters, a decoder-only autoregressive generator, and the H-Codec tokenizer/decoder. The condition encoders are a HuBERT-based audio encoder and a T5-base text encoder. Two linear layers project HuBERT and T5 features into the LLaMA-space used by the generator. The autoregressive backbone is a decoder-only LLaMA-style transformer with 16 layers, 16 attention heads, hidden size 1024, and 481M trainable parameters (Liu et al., 30 Oct 2025).

A defining component is H-Codec, which factorizes audio representation into an acoustic branch and a semantic branch. The acoustic branch takes waveform input $\boldsymbol{x} \in \mathbb{R}^{n}$, uses an Encodec-style acoustic encoder, and quantizes acoustic features with a 4-layer RVQ at 25 Hz. The semantic branch extracts SSL features from a pretrained HuBERT by averaging outputs from all transformer layers, processes them with a semantic encoder, and quantizes the resulting semantic representation with another 4-layer RVQ. Rather than fusing acoustic and semantic features before quantization, H-Codec quantizes them separately with distinct codebooks (Liu et al., 30 Oct 2025).

Waveform reconstruction concatenates quantized acoustic and semantic features along the hidden dimension, passes the combined representation to the acoustic decoder, and uses an ISTFT head following Vocos to reconstruct waveform. A semantic decoder reconstructs the SSL feature from the quantized semantic representation, enforcing semantic retention. Important codec parameters are a downsampling ratio of 640, 4 RVQ layers per branch, codebook size 1024 per layer, codebook dimension 512, and total bitrate 2000 bps. Because the acoustic and semantic token streams are interleaved in generation, the effective generation schedule operates at 50 Hz (Liu et al., 30 Oct 2025).

The codec generator loss is

$$\mathcal{L}_{gen} = \lambda_{commit} \mathcal{L}_{commit} + \lambda_{mel} \mathcal{L}_{mel} + \lambda_{adv} \mathcal{L}_{adv} + \lambda_{fm} \mathcal{L}_{fm} + \lambda_{aux} \mathcal{L}_{aux}.$$

Here $\mathcal{L}_{commit}$ is the VQ commitment loss, $\mathcal{L}_{mel}$ is the mel-spectrogram reconstruction loss, $\rightarrow$0 is the adversarial generator loss, $\rightarrow$1 is the feature matching loss, and $\rightarrow$2 is the MSE loss on reconstructed SSL features. The discriminator loss follows the hinge-loss setup from SoundStream, with a discriminator stack consisting of a multi-period discriminator, a multi-resolution discriminator, and a sub-band complex STFT discriminator. A perceptual loss is added during the final 100k training steps (Liu et al., 30 Oct 2025).

This architecture places UniTok-Audio between classical acoustic codecs and semantic token pipelines. It does not rely on a single undifferentiated code stream, but it also does not split generation across separate semantic and acoustic generators. The codec preserves factorization, while the LLM learns dependencies between the two branches through one autoregressive token-generation process.

4. Training methodology and data construction

Training proceeds in two conceptually separate phases: H-Codec training followed by UniTok-Audio training. H-Codec is trained on mixed-domain audio comprising speech: VoxBox, about 100k hours; music: FMA-full + MUSDB18-HQ, about 8k hours; general audio: AudioSet + WavCaps, about 13k hours. All audio is resampled to 16 kHz, training uses random 5-second crops, optimization uses AdamW, the initial learning rate is $\rightarrow$3, cosine decay is applied, training runs for 600k steps, and the perceptual loss is enabled in the final 100k steps (Liu et al., 30 Oct 2025).

UniTok-Audio training uses separate data constructions for each task family. For speech-related tasks, clean speech comes from VoxBox subsets including LibriSpeech, MLS English, Emilia_ZH, totaling about 3.8k hours. The noise and corruption pool includes DNS, FSD50K, WHAM!, DESED, DEMAND, MUSAN, DISCO, MUSDB18-HQ, TUT Urban Acoustic Scenes, a 460 hours noise corpus, and 60k RIRs from SLR28. For audio-caption tasks, the data include WavCaps, CLAP_FreeSound, VGGSound, internal data, totaling about 40k hours. For voice conversion, an external voice conversion model is used to generate 6k hours of fixed timbre-perturbed training data (Liu et al., 30 Oct 2025).

The data simulation pipeline is dynamic rather than fixed. For SR, degradations are applied probabilistically: additive noise with probability 0.5 and SNR uniform in $\rightarrow$4 dB, reverberation 0.4, clipping 0.3, bandwidth limitation 0.3 with cutoff in $\rightarrow$5 kHz, packet loss 0.3 with rate uniform in $\rightarrow$6, and interfering speaker 0.2 with SIR uniform in $\rightarrow$7 dB. For TSE and rTSE, the interference-speaker probability is 1.0, SIR is sampled in $\rightarrow$8 dB, and the distortion chain is also applied. For LASS, target audio is mixed with another random audio at SIR sampled from $\rightarrow$9 dB (Liu et al., 30 Oct 2025).

Each example is transformed into condition embeddings from the audio or text encoder and a target H-Codec token sequence for the desired waveform. The LLM is then trained with teacher forcing: at each step it predicts the next delayed multi-layer token while conditioning on the ground-truth previous tokens. For the multi-task omni model, one of the five modes is randomly selected per batch, which is the paper’s explicit task-mixing strategy (Liu et al., 30 Oct 2025).

5. Reconstruction quality, downstream performance, and scaling

The paper evaluates H-Codec separately from the unified generator. On LibriSpeech test-clean, H-Codec at 25+25 FPS, 4 layers, and 2000 bps reports PESQ 2.99, STOI 0.94, UTMOS 4.06, SPK-SIM 0.84, and WER 3.18. On cross-domain reconstruction it reports speech mel/STFT loss 0.3394 / 0.1033, music 0.5158 / 0.1667, and audio 1.2512 / 0.4070. The paper states that these are the best among the general-audio codecs listed, and uses them as evidence that H-Codec provides a stronger discrete representation than prior codecs for downstream generation (Liu et al., 30 Oct 2025).

The downstream results indicate that UniTok-Audio is generally competitive with strong task-specific and unified baselines across the five time-aligned tasks.

Task	Dataset	UniTok-Audio result
Speech restoration	DNS 2020	With Reverb: SIG 3.67, BAK 4.12, OVRL 3.42; No Reverb: SIG 3.66, BAK 4.15, OVRL 3.44
PLC restoration	ICASSP 2022 PLC blind test	OVRL 3.35, PLCMOS 4.58
Target speaker extraction	Libri2Mix clean test	SIG 3.62, BAK 4.05, OVRL 3.32, NISQA 4.00, SIM 0.95
Speech separation	Libri2Mix noisy / WSJ0-2mix	OVRL 3.25 / 3.26
Voice conversion	VCTK-based evaluation	WER 4.23, SIM 0.50, DNSMOS 3.51, NISQA 4.51
LASS	DCASE 2024 LASS validation	Single-task: FAD 0.68, CLAPScore 28.85, CLAPScore$$\mathcal{L}_{\rm SR} = - \sum_{t=1}^{L} \sum_{i=1}^{4} \log P\left(o^i_t \mid {\rm T_{SR}, I, E_i, S}, o_{<t}; \theta \right).$$0 65.56; Omni: FAD 1.48, CLAPScore 26.21, CLAPScore$$\mathcal{L}_{\rm SR} = - \sum_{t=1}^{L} \sum_{i=1}^{4} \log P\left(o^i_t \mid {\rm T_{SR}, I, E_i, S}, o_{<t}; \theta \right).$$1 61.21

For SR, the omni model is competitive with the best discrete generative baselines on DNS 2020 and exceeds all listed baselines on the PLC benchmark, where it reaches OVRL 3.35 and PLCMOS 4.58. For TSE, the omni model is roughly on par with the strongest generative baselines, though not clearly superior. For SS, the iterative SR + rTSE strategy yields OVRL 3.25 on Libri2Mix and 3.26 on WSJ0-2mix, which the paper highlights because speech separation is usually difficult for unified token-generation systems with single-track output constraints. For VC, the omni model is strong, but the single-task WavLM-based variant is better, reaching WER 3.02, SIM 0.51, DNSMOS 3.46, and NISQA 4.46. For LASS, the single-task model is substantially better than the omni model, which the paper attributes to speech-vs-general-audio domain mismatch (Liu et al., 30 Oct 2025).

Ablative comparisons suggest that unified training works best when tasks share similar structure and domain. The paper reports that SR and TSE remain essentially as strong in the omni setting as in single-task training, whereas VC and LASS degrade. It also reports scaling behavior on VC: a 109M model yields WER 5.38, SIM 0.42, DNSMOS 3.41, NISQA 4.30; a 481M model yields WER 3.02, SIM 0.51, DNSMOS 3.46, NISQA 4.46; and a 1.3B model yields WER 2.10, SIM 0.61, DNSMOS 3.61, NISQA 4.54. This suggests that the framework is scalable in the ordinary model-size sense, even though its task compatibility remains uneven across domains (Liu et al., 30 Oct 2025).

6. Position in the unified audio-token literature

UniTok-Audio belongs to a broader line of work seeking unified audio interfaces, but it occupies a distinct niche. UniAudio formulated many audio generation tasks as serialized next-token prediction over codec tokens, using a multi-scale Transformer to manage long RVQ sequences; its strongest contribution was unified sequence modeling rather than a fundamentally new tokenizer (Yang et al., 2023). UniCodec and AUV pursued LM-friendly single-layer, single-codebook universal codecs, using partitioned or nested codebooks and domain-sensitive training to support speech, music, and sound with one token stream (Jiang et al., 27 Feb 2025, Chen et al., 26 Sep 2025). UniTTS pushed toward a single-codebook regime in which the LLM directly handles one stream of discrete audio tokens carrying comprehensive audio information, but it remained framed as an end-to-end TTS system rather than a broad audio understanding interface (Wang et al., 23 May 2025).

Other contemporaneous systems broadened the notion of unification beyond time-aligned generation. Ming-UniAudio introduced MingTok-Audio, a continuous, VAE-based unified speech tokenizer intended to support speech understanding, generation, and free-form editing without timestamps (Yan et al., 26 Oct 2025). UniAudio 2.0 proposed ReasoningCodec, which factorizes audio into reasoning tokens and reconstruction tokens, treating unified tokenization as a first-class design problem for an audio foundation model (Yang et al., 4 Feb 2026). UniAudio-Token, from Yuhan Song, Linhao Zhang, Aiwei Liu, Chuhan Wu, Sijun Zhang, Wei Jia, Yuan Liu, Houfeng Wang, and Xiao Zhou at Peking University and Tencent WeChat AI, argued that a semantic speech tokenizer can be turned into a unified audio tokenizer without abandoning the compact single-codebook interface favored by Audio-LLMs (Song et al., 29 May 2026). AVTok extended unification across modalities by placing audio and video into a shared token space with a unified codebook for holistic audio-video generation (Pham et al., 29 Jun 2026).

Against that backdrop, UniTok-Audio is best understood as a discrete codec-token autoregressive framework for aligned conditional generation. It is not primarily a universal audio understanding interface, a single-codebook universal codec, or a multimodal shared-codebook tokenizer. Its characteristic combination is continuous condition features, task-token-controlled conditional generation, and a dual-stream acoustic/semantic codec.

The framework’s limitations are correspondingly specific. The model only supports one-track output, so speech separation requires iterative inference through SR and rTSE rather than native multi-track generation. VC and LASS degrade under omni training, indicating task interference or a speech-vs-general-audio domain gap. The paper also notes that speaker similarity in generative methods can be limited by codec reconstruction fidelity, which locates a remaining bottleneck in the codec rather than the autoregressive generator. More broadly, the results indicate that unified training is effective when tasks share structure, but that broader unification across heterogeneous audio domains remains unresolved (Liu et al., 30 Oct 2025).