---
title: 'CleanCodec: A Speech-Centric Neural Codec'
url: https://www.emergentmind.com/topics/cleancodec
type: topic
---

# CleanCodec: A Speech-Centric Neural Codec

CleanCodec is a neural speech codec and audio tokenizer that frames speech compression as a **selective information bottleneck**: instead of preserving the full waveform, it is trained to retain **linguistic content**, **speaker timbre**, and **usable prosody/acoustics** while discarding **background noise**, **reverberation**, **microphone/channel artifacts**, and other perceptually or task-irrelevant details. The system operates on **log-mel spectrograms**, uses a **single-codebook** discrete bottleneck based on **Finite Scalar Quantization (FSQ)**, separates **local time-varying tokens** from a **global utterance-level speaker embedding**, and reconstructs speech through a **Vocos-based vocoder**. Its flagship operating point is **12.5 tokens/s**, at which it is reported to achieve **state-of-the-art tokenization efficiency**, strong speech intelligibility and speaker similarity, and substantial downstream efficiency gains for text-to-speech and voice conversion [2606.04418].

## 1. Conceptual framing and problem definition

CleanCodec begins from the observation that conventional neural audio codecs often allocate bitrate to signal detail that is not central to speech understanding or downstream modeling. In this view, a speech tokenizer should not be optimized to reproduce every waveform irregularity. Rather, it should preserve the speech information that is most relevant to human perception and to applications such as TTS, voice conversion, and speech-language modeling, while compressing away nuisance variation [2606.04418].

The paper situates this position against three broad codec/tokenizer families. **Semantic tokenizers** are efficient, but tend to under-represent speaker identity and prosody, which harms reconstruction quality. **Acoustic codecs** can reconstruct audio well, but often preserve too much low-level detail and frequently rely on many **RVQ** codebooks, increasing token rate and downstream modeling complexity. **Disentangled codecs** reduce token rate by separating global and local factors, but may degrade reconstruction fidelity or robustness, particularly out of domain. CleanCodec’s proposal is to combine disentanglement, single-stream tokenization, and denoising supervision into a speech-centric codec whose bottleneck is explicitly pressured to keep only salient information [2606.04418].

This framing is consistent with a broader benchmarking concern in codec research: the ideal codec should preserve **content**, **speaker identity**, **paralinguistic information**, and **general audio information** under low bitrate, rather than optimizing only a narrow set of waveform metrics [2402.13071]. CleanCodec addresses that question specifically for speech by making denoising part of the tokenization objective itself.

A common misconception is that the best speech codec is necessarily the one that most faithfully reconstructs the observed input signal. CleanCodec explicitly rejects that assumption. Its central claim is that strict fidelity to the degraded input can be counterproductive when the objective is an efficient, robust, speech-oriented token stream. The codec is therefore designed to reconstruct **clean speech targets** from **corrupted speech inputs**, forcing the bottleneck to prioritize information that remains useful after nuisance variability is removed [2606.04418].

## 2. Representation design and model architecture

CleanCodec is a **disentangled VQ-VAE-style codec operating on log-mel spectrograms**. The processing pipeline is: waveform $\rightarrow$ log-mel spectrogram $\rightarrow$ local and global encoders $\rightarrow$ discrete local tokens + continuous global embedding $\rightarrow$ acoustic and semantic decoders $\rightarrow$ mel reconstruction $\rightarrow$ vocoder waveform synthesis [2606.04418].

The input waveform is sampled at **24 kHz**. Conversion to log-mel reduces temporal resolution to **62.5 Hz**. The local encoder then downsamples further by a factor of **5**, **2**, or **1**, producing three operating points:
\[
r=\frac{62.5}{d}\ \text{tokens/s},
\]
with **CleanCodec@12.5**, **CleanCodec@31.25**, and **CleanCodec@62.5** corresponding to $d=5,2,1$ respectively [2606.04418].

The **local encoder** is a fully convolutional architecture based on **1D ConvNeXt blocks**, following a Vocos-style design. Its appendix configuration specifies **16 layers**, **hidden dimension 768**, **depthwise kernel size 27**, and **MLP intermediate size 3072**. This branch produces the time-varying latent sequence that is quantized into discrete tokens [2606.04418].

Quantization is performed by **FSQ**, not by standard VQ or RVQ. The quantizer is **single-codebook** and **parameter-free**, with levels:
\[
[8,8,8,8,8].
\]
This yields
\[
8^5 = 32768
\]
possible codes. Each compressed frame therefore emits exactly one discrete symbol from a **32,768-way codebook**, producing a single token stream rather than multiple RVQ streams [2606.04418].

The **global encoder** models utterance-level speaker and timbre information. It is inspired by **Kanade**, uses modified ConvNeXt blocks from **NeXt-TDNN**, and ends with **attentive statistics pooling**. Its appendix configuration specifies **8 layers**, **hidden dimension 512**, **depthwise kernel size 7**, **MLP intermediate size 1536**, and a final global embedding dimension of **256**. During decoding, this global embedding is concatenated to the local token representation, so the local stream is not required to carry all speaker information [2606.04418].

Decoding is split into two branches. The **acoustic decoder** reconstructs the target log-mel spectrogram. It uses a 1D ConvNeXt backbone with the same size as the local encoder: **16 layers**, **768 dimension**, **kernel size 27**, and **MLP size 3072**. The **semantic decoder** reconstructs **WavLM-large** features and uses the same architecture except for **kernel size 53**. The semantic decoder exists only during training and is discarded afterward [2606.04418].

Waveform synthesis is delegated to a **Vocos-based vocoder** trained by the authors. Its appendix configuration specifies **16 layers**, **1536 hidden dimension**, **depthwise kernel size 5**, and **MLP intermediate size 4608**. Reported model sizes are **243M** parameters for the autoencoder, **228M** for the vocoder, and **471M** total; after discarding the semantic decoder at inference, the deployed model size is **396M** [2606.04418].

The reported system is effectively **non-causal** and **non-streaming**. The use of **attentive statistics pooling** and full-utterance global conditioning implies an utterance-level design rather than a low-latency streaming codec [2606.04418].

## 3. Objectives, supervision, and perceptually guided denoising

Although CleanCodec is motivated as a selective information bottleneck, the paper does **not** formulate an explicit mutual-information objective such as $I(X;Z)-\beta I(Z;Y)$. The bottleneck interpretation is conceptual. The actual training objective is expressed through supervised reconstruction and conditioning losses on mel acoustics, semantic features, and speaker embeddings [2606.04418].

The paper defines:
- $m$: true log-mel spectrogram,
- $s$: true SSL features from **WavLM-large**,
- $g$: true speaker embedding from **TitaNet-large**,
- $\hat{g}$: predicted global embedding,
- $\hat{m}$: predicted log-mel spectrogram,
- $\hat{s}$: predicted SSL features.

The explicit losses are:
\[
\mathcal{L}_\text{mel}=||m-\hat{m}||_2,
\]
\[
\mathcal{L}_\text{ssl}=1-\cos(s,\hat{s}),
\]
\[
\mathcal{L}_\text{emb}=1-\cos(g,p(\hat{g})),
\]
where $p$ is a learned linear projection into the pretrained speaker-embedding space [2606.04418].

The vocoder follows the Vocos training recipe. Its loss is composed of **multiscale mel spectrogram loss**, **adversarial loss**, and **feature matching loss**. The discriminator setup includes a **multi-period discriminator** in the style of HiFi-GAN and a **multi-band STFT discriminator at multiple timescales** in the style of UnivNet. The exact formulas are not printed in the excerpted paper [2606.04418].

What makes CleanCodec distinctive is that these objectives are applied under **degraded-input, clean-target supervision**. During training, the codec receives **corrupted speech** and is asked to reconstruct the **original clean speech**. The denoising effect is therefore not imposed by an explicit psychoacoustic mask or rule at inference, but is learned through supervision under a narrow bottleneck. Operationally, the model learns that noise, reverberation, low-pass effects, codec artifacts, and resampling artifacts are not worth encoding when the target is clean speech [2606.04418].

Each degradation is applied with **50% probability**. The training augmentations are:
- **Reverberation** with RT60 sampled from $\mathcal{U}(0.1,2.0)$ seconds, room dimensions from $\mathcal{U}(2,20)$ meters, simulated with **pyroomacoustics**, with RIRs of size $>50000$ samples filtered out.
- **Background noise** from **AudioSet**, **FSD50K**, and **WHAM!**, added at SNR $\mathcal{U}(15,30)$ dB.
- **Low-pass filtering** with cutoff between **2–8 kHz**.
- **Resampling** to $\{16,22.05\}$ kHz and then back to **24 kHz**.
- **Codec degradation** via MP3 compression with bitrate **32–245 kbps** [2606.04418].

A second common misconception is that denoising supervision necessarily removes useful speaker or semantic structure. CleanCodec’s design counteracts that risk in two ways. The **semantic decoder** forces local tokens to retain strong linguistic information, and the **TitaNet-supervised global encoder** forces the global branch to preserve speaker identity. The paper’s ablations show that both mechanisms are structurally important rather than auxiliary [2606.04418].

## 4. Tokenization efficiency, bitrate implications, and training regime

The flagship operating point is **12.5 tokens/s**. This is achieved by converting the waveform to log-mel at **62.5 frames/s**, downsampling the local encoder output by **5×**, and emitting one FSQ code per compressed frame. Because the codebook cardinality is $32768 = 2^{15}$, the local stream can be interpreted as carrying approximately
\[
12.5 \times 15 = 187.5\ \text{bits/s},
\]
excluding storage overhead and the continuous global embedding [2606.04418].

Within the paper’s baseline set, the reported token rates are **Qwen3: 200 / 100 / 50 t/s**, **Mimi: 100 t/s**, **BiCodec: 50 t/s**, **XCodec2: 50 t/s**, **WavTokenizer: 40 t/s**, **FocalCodec: 12.5 t/s**, and **Kanade: 12.5 t/s**. CleanCodec therefore matches the ultra-low-rate regime of FocalCodec and Kanade, but the paper argues that it uses this rate more effectively because it preserves cleaner speech structure rather than reproducing all observed detail [2606.04418].

The system also reports strong efficiency at inference. Appendix measurements give **Encoding RTF = 0.0011** (**928× realtime**), **Decoding RTF = 0.0034** (**291× realtime**), and **Total RTF = 0.0045** (**221× realtime**). In the downstream TTS setting, the lower token rate materially reduces autoregressive sequence length: compared with a **200 t/s** tokenizer, the paper reports **up to 17× faster inference**, using the ratio between **Qwen3 RTF = 2.930** and **CleanCodec RTF = 0.170** [2606.04418].

Codebook utilization is measured with normalized entropy:
\[
\text{Normalized Entropy}=-\frac1{\log N}\sum_{x=1}^N p(x)\log(p(x)).
\]
For **CleanCodec@12.5**, the reported value is **0.985**, indicating strong codebook usage [2606.04418].

Training uses approximately **2,400 hours** of speech: the full training split of **LibriTTS-R** (**585 hours**, **2,456 speakers**) and an **1,800-hour subset** of **Emilia-YODAS**. Audio is sampled at **24 kHz**; training segments have length **230,400 samples** (**9.6 seconds**); utterances shorter than that are repeated and trimmed. All audio is preprocessed with a **high-pass filter at 80 Hz** and loudness normalization to peak amplitude **0.75** [2606.04418].

Optimization follows a **two-stage** schedule. In **Stage 1**, the autoencoder and vocoder are trained independently for **1M steps each**. In **Stage 2**, the system is jointly fine-tuned end to end for **200K steps**. The optimizer is
\[
\text{AdamW } (\beta_1=0.8,\beta_2=0.9,\text{wd}=0.01),
\]
with learning rate **$1\mathrm{e}{-4}$** in stage 1 and **$2\mathrm{e}{-6}$** in stage 2. Training uses **1× A100 80GB** and takes about **one week** [2606.04418].

## 5. Empirical performance, robustness, and ablation evidence

On **LibriTTS test-clean**, the paper reports the following for **CleanCodec@12.5**: **WER 2.7**, **CER 1.4**, **SIM 0.86**, **UTMOS 4.32**, **SI-SDR 21.96**, and **Mel L1 0.44**. On **LibriTTS test-other**, it reports **WER 5.7**, **CER 3.2**, **SIM 0.84**, **UTMOS 4.17**, and **Mel L1 0.44**. At the same **12.5 t/s** rate, **Kanade** reports **WER 4.0**, **CER 2.1**, **SIM 0.65**, **UTMOS 4.22**, **SI-SDR 23.60**, and **Mel L1 0.57**, while **FocalCodec** reports **WER 8.3**, **CER 4.5**, **SIM 0.45**, **UTMOS 4.21**, **SI-SDR 25.19**, and **Mel L1 0.78**. The paper’s interpretation is that CleanCodec substantially improves intelligibility and speaker similarity at the same token rate, while remaining competitive with many higher-rate codecs [2606.04418].

Higher-rate variants extend the same design. On test-clean, **CleanCodec@62.5** reports **WER 1.3**, **CER 0.6**, **SIM 0.90**, and **Mel L1 0.29**, which the paper presents as close to **Qwen3@200**, also reported at **WER 1.3**, **CER 0.7**, **SIM 0.90**, and **Mel L1 0.29**. The paper treats this as one of its strongest efficiency results: comparable reconstruction behavior at about **3× fewer tokens** [2606.04418].

Out-of-domain robustness is a central empirical claim. At **12.5 t/s**, on **Expresso**, CleanCodec reports **WER 3.9**, **CER 1.9**, **SIM 0.82**, and **Mel L1 0.45**; on **AISHELL-3**, **CER 1.5**, **SIM 0.84**, and **Mel L1 0.39**; on **CML-TTS**, **WER 12.1**, **CER 5.1**, and **SIM 0.78**. These results are markedly stronger than the corresponding **Kanade** and **FocalCodec** numbers reported in the same tables, and the paper attributes this to denoising training that preserves robust speech structure instead of overfitting surface acoustics [2606.04418].

The paper also evaluates disentanglement directly. For **speaker verification from the global representation** on **VoxCeleb-1**, **CleanCodec@12.5** achieves **ACC 99.92** and **EER 0.58**. For **ASR from local tokens**, it reports **WER 5.6** and **CER 3.0**. This supports the claim that the global branch captures speaker information effectively while the local token stream preserves linguistic information [2606.04418].

In **voice conversion** on **VCTK**, using local tokens from a source utterance and the global embedding of a reference speaker, **CleanCodec@12.5** reports **SIM 0.81**, **UTMOS 4.09**, **WER 1.4**, and **CER 0.7**. The paper contrasts this with stronger speaker leakage in some competing codecs, noting that local-token speaker contamination can limit the effect of swapping global embeddings [2606.04418].

For autoregressive **text-to-speech** on **Seed-TTS-eval**, **CleanCodec@12.5** reports **Train time 01:00**, **RTF 0.170**, **SIM 0.56**, and **WER 3.9**. The reported **Qwen3@200** baseline gives **Train time 10:02**, **RTF 2.930**, **SIM 0.36**, and **WER 9.1**. These figures underpin the claim that lower token rate improves both modelability and inference speed in downstream generation [2606.04418].

The ablation study clarifies the role of the major components. Replacing **two-stage training** with **one-stage end-to-end adversarial training** degrades all reported metrics: **WER 2.7 → 4.1**, **CER 1.4 → 2.2**, **SIM 0.86 → 0.80**, **UTMOS 4.32 → 4.01**, **SI-SDR 21.96 → 18.58**, and **Mel L1 0.44 → 0.57**. Removing the **semantic SSL branch** causes a large linguistic collapse: **WER 2.7 → 8.1** and **CER 1.4 → 4.3**. Removing **TitaNet global conditioning** lowers **SIM 0.86 → 0.74**, while replacing it with **WavLM-based speaker conditioning** yields **SIM 0.77**. Removing **denoising** yields a more subtle but revealing pattern: **WER 2.7 → 3.4**, **CER 1.4 → 1.7**, **SIM 0.86 → 0.82**, **UTMOS 4.32 → 4.24**, **SI-SDR 21.96 → 20.55**, while **Mel L1** slightly improves from **0.44** to **0.41**. The paper uses this to argue that lower mel error does not necessarily imply better speech tokenization: the denoising bottleneck sacrifices a small amount of exact input fidelity in order to preserve more task-relevant information [2606.04418].

## 6. Relation to adjacent codec research, interpretation, and limitations

CleanCodec belongs to a broader movement toward low-rate, single-stream, downstream-friendly audio tokenization, but it occupies a distinct position within that literature. Earlier work on joint denoising and compression, such as **“Enhancing into the codec: Noise Robust Speech Coding with Vector-Quantized Autoencoders”** [2102.06610], already showed that a codec can be trained on **noisy input** and **clean output** through a discrete bottleneck. CleanCodec extends that general principle with a log-mel front end, explicit global/local disentanglement, FSQ single-stream tokenization, semantic supervision, and modern downstream evaluation [2606.04418].

Relative to other single-codebook speech codecs, CleanCodec is closest in spirit to designs that simplify token modeling by producing one discrete stream. **“Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation”** uses one **8192-entry** codebook, a continuous global embedding, and about **23 discrete tokens per second** at **304 bps**, with the explicit goal of LLM-friendly single-sequence generation [2406.07422]. **“LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec”** similarly combines low bitrate and speaker decoupling, reporting **0.45 kbps** at **50 Hz** with **$V=300$** and **0.25 kbps** at **25 Hz** with **$V=1024$**, using a multi-stage framework with speaker perturbation and a discrete token vocoder [2410.15764]. CleanCodec differs from both by centering the codec around **perceptually guided denoising** rather than primarily around speaker removal or phonetic abstraction alone.

Another adjacent direction addresses token simplicity not by denoising but by making the decoder more robust to quantization. **“Self-Guidance: Enhancing Neural Codecs via Decoder Manifold Alignment”** introduces a training-only decoder feature-matching loss that improves reconstruction in **XCodec2**, yields **no inference-time changes**, and supports a **4× codebook reduction** without fidelity loss in the reported setup [2606.12940]. This suggests an orthogonal design axis: CleanCodec reduces nuisance information before or during tokenization, whereas self-guidance improves decoder tolerance to quantization error after tokenization.

At the level of modality scope, CleanCodec is more specialized than recent unified audio codecs. **“UniSRCodec: Unified and Low-Bitrate Single Codebook Codec with Sub-Band Reconstruction”** targets **speech, music, and general audio**, operates at **44.1 kHz**, compresses in the **Mel-spectrogram** domain, and reports **40 tokens per second** and **0.52 kbps** at its main low-bitrate operating point [2601.02776]. CleanCodec, by contrast, is explicitly a **speech tokenizer** and does not claim equivalent unified audio coverage.

The main limitations reported for CleanCodec are several. **Disentanglement degrades at higher token rates**, as voice-conversion results suggest that more speaker information leaks into local tokens when capacity rises. The design is **non-streaming** and utterance-level. Training data, while large for a speech codec, remain moderate by frontier standards at approximately **2.4k hours**. The paper also notes that robustness to all languages, conversational telephony, pathological speech, or extreme acoustics is not fully established. Finally, the system’s improved speech tokenization can also aid **voice cloning** or related misuse [2606.04418].

Taken together, these features make CleanCodec best understood as a **speech-centric, denoising, disentangled, single-stream neural tokenizer**. Its central claim is not merely that bitrate can be reduced, but that the token budget can be spent more efficiently if the codec is trained to encode **clean, perceptually important speech structure** rather than the full degraded signal. In that sense, CleanCodec represents a specific answer to a broader codec-design question: not how to preserve everything, but how to preserve the right things.

Source: https://www.emergentmind.com/topics/cleancodec