CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding

Abstract: Neural audio codecs are a key component of speech processing pipelines, compressing audio into discrete tokens for downstream modeling. However, existing codecs struggle to balance reconstruction quality with token efficiency, often encoding perceptually irrelevant information such as background noise and recording artifacts at the expense of linguistically and acoustically meaningful content. We reframe audio tokenization as a selective information bottleneck problem and propose CleanCodec, a denoising audio codec which learns to encode only perceptually important features and discard imperceptible information. At just 12.5 tokens per second, CleanCodec achieves state-of-the-art tokenization efficiency, substantially outperforming existing codecs in speaker similarity and speech intelligibility. Evaluations on downstream text-to-speech and voice conversion tasks further demonstrate improved performance and up to 17x faster inference, highlighting significant efficiency gains.

• The paper demonstrates a selective information bottleneck that discards noise while preserving perceptually significant speech features, achieving 12.5 tokens per second.
• It employs a dual-decoder architecture with explicit denoising and TitaNet-based speaker conditioning, resulting in near-perfect speaker verification and reduced WER.
• The codec outperforms prior methods in OOD robustness and efficiency, enabling faster training/inference for TTS, audio LLM pretraining, and voice conversion tasks.

CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding

Introduction

The proliferation of LLM-driven audio generation, latent diffusion TTS systems, and multimodal speech processing architectures necessitates highly efficient neural audio codecs. These codecs must minimize token rates while preserving semantically and acoustically relevant attributes for downstream modeling. The disparity in tokenization efficiency between text and audio domains results from the non-linguistic variance in speech waveforms, including speaker identity, environmental noise, and channel artifacts. Traditional approaches to speech tokenization—whether semantic tokenizers leveraging SSL models or acoustic codebooks employing VQ-VAE with RVQ—struggle to balance a low information bottleneck with robust perceptual fidelity and semantic preservation.

CleanCodec reframes the neural speech codec as a selective information bottleneck, retaining only perceptually significant information and aggressively discarding background noise and recording artifacts. Through joint training with explicit denoising objectives and robust conditioning on both semantic and speaker representations, CleanCodec achieves state-of-the-art tokenization efficiency at 12.5 tokens per second, with strong downstream and OOD generalization at substantially reduced computational cost.

Methodology

Architectural Overview

CleanCodec employs a disentangled VQ-VAE-style autoencoder operating on log-mel spectrograms, eschewing the fragility of SSL input features for enhanced robustness. The architecture consists of:

• Local Encoder: A stack of ConvNeXt-1D blocks with aggressive downsampling achieves low token rates. Finite Scalar Quantization (FSQ) is utilized for single-codebook, parameter-free discretization, supporting 32,768 code vectors.
• Global Encoder: Modified ConvNeXt blocks with attentive statistics pooling produce an utterance-level embedding, concatenated with the audio tokens during decoding (Figure 1).

  Figure 1: Architecture of CleanCodec, highlighting the local/global encoder-decoders, FSQ quantization, and denoising/conditioning loss terms.

• Dual Decoder: Parallel decoders reconstruct the mel spectrogram (acoustic path) and SSL features from WavLM-large (semantic path), driving the token representation to retain both linguistic and timbral information.
• Vocoder: A Vocos-based vocoder trained independently and subsequently fine-tuned end-to-end reconstructs waveforms from mel spectrograms with adversarial and feature matching supervision.

Training Framework

To maximize perceptual quality under extreme compression, CleanCodec introduces a joint denoising-reconstruction objective. The model is trained on audio corrupted via a stochastic augmentation pipeline (reverberation, background noise, filtering, resampling, codec artifacts) and supervised to reconstruct the clean reference. This forces the information bottleneck to prioritize semantically and perceptually important content.

Global conditioning is performed using a pretrained TitaNet speaker verification model. The global embedding is linearly projected and supervised to maximize cosine similarity with TitaNet representations, ensuring preservation of speaker attributes not captured by self-supervised semantic models. The autoencoder is optimized with combined L2, cosine, and speaker embeddings losses.

A two-stage curriculum prevents mode collapse in adversarially-trained components: the autoencoder and vocoder are pretrained separately with non-adversarial losses, then fine-tuned jointly end-to-end.

Experimental Results

Speech Reconstruction Performance

CleanCodec achieves first- or second-best results in most metrics on LibriTTS test-clean and test-other splits across WER, SIM, Mel L1, and UTMOS at all tested token rates. At 12.5 t/s, substantial improvements in speaker similarity (SIM 0.86 vs 0.64) and word error rate (WER 2.7 vs 4.0) are realized compared to prior work, even relative to baselines operating at much higher token rates.

The results confirm that CleanCodec's architectural decisions—FSQ-based bottleneck, explicit denoising, dual semantic/acoustic reconstruction, and TitaNet-based speaker conditioning—enable preservation of critical content while discarding superfluous information. Importantly, efficiency scaling is nearly monotonic: higher token rates further improve benchmarks, with [email protected] outperforming Qwen3-TTS-Tokenizer at a 3x lower token rate.

CleanCodec demonstrates strong OOD robustness, with minimal degradation in intelligibility and speaker preservation on expressive (Expresso), multilingual (AISHELL-3), and wild (CML-TTS) datasets, outperforming all low-rate baselines in WER and SIM.

Disentanglement Evaluation

Disentanglement is quantified via speaker verification (SV) and ASR tasks using representations from the global and local encoders, respectively. [email protected] achieves near-perfect SV accuracy (99.92%) and minimal EER (0.58%), with WER and CER far below prior disentangled models (BiCodec, Kanade). TitaNet-based conditioning is critical; ablation shows degradation in speaker similarity when replaced by WavLM-SV, validating the necessity of discriminative speaker guidance.

Downstream Task Transfer

On voice conversion (VC), [email protected] attains the highest speaker similarity (0.81) and competitive naturalness (UTMOS 4.09), outstripping BiCodec and Kanade. The model demonstrates effective separation and recombination of speaker and linguistic factors, though some residual entanglement remains at low bitrates.

For TTS, the main bottleneck is token rate. CleanCodec enables 10x faster training and 17x faster inference than Qwen3, with a simultaneous reduction in WER and improvement in speaker similarity. These results establish CleanCodec as well-suited for audio LLM pretraining and lightweight TTS systems requiring extremely compressed discrete representations.

Efficiency and Codebook Utilization

Despite a larger parameter count (396M post-training), [email protected] achieves strong encoding/decoding speed (RTF 0.0045, 221x real-time), second only to strictly lightweight codecs. Codebook usage, as measured by normalized entropy (0.985), indicates near-maximal utilization, confirming effective compression without codebook collapse.

Ablation Study

A comprehensive ablation reveals the necessity of each component:

• Removing denoising leads to increased error rates and diminished speaker similarity.
• Dropping semantic (SSL) conditioning produces a three-fold increase in WER/CER.
• Omitting global (TitaNet) conditioning or using WavLM-SV degrades speaker preservation.
• Single-stage end-to-end training causes global quality collapse, confirming the two-stage paradigms' criticality.

Theoretical and Practical Implications

CleanCodec's selective bottleneck formulation advances neural audio tokenization toward efficient, interpretable, and disentangled discrete representations matched to human perceptual criteria. These gains are directly applicable to LLM-based audio generation/scoring tasks, model compression for on-device applications, and robust audio manipulation for ASR/TTS. Its denoising-based framework highlights new directions for perceptual bottleneck learning, with possible extensions including dynamic token allocation and further scaling of compute budgets and training duration to investigate capacity limits.

Conclusion

CleanCodec introduces a denoising, disentangled neural audio codec achieving state-of-the-art efficiency at extreme bottlenecks by combining perceptually guided objectives, robust conditioning, and a stabilized two-stage training scheme. The model sets new standards for the tradeoff between token rate, reconstruction quality, semantic/identity preservation, OOD robustness, and computational efficiency. Limitations remain in disentanglement quality at high token rates and scaling behaviors across compute settings; future work should address these along with broader implications for LLM-driven audio understanding and synthesis.

---

References

• "CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding" (2606.04418)