CleanCodec: A Speech-Centric Neural Codec

• CleanCodec is a speech-centric neural codec that uses a selective information bottleneck to preserve linguistic content, speaker identity, and prosody while filtering out noise and irrelevant details.
• It integrates a single-codebook discrete bottleneck via Finite Scalar Quantization on log-mel spectrograms, combining local time-varying tokens with a global speaker embedding for effective disentangled representation.
• Achieving state-of-the-art efficiency at 12.5 tokens/s, CleanCodec delivers robust performance in tasks such as TTS and voice conversion while significantly reducing autoregressive sequence length and bitrate.

CleanCodec is a neural speech codec and audio tokenizer that frames speech compression as a selective information bottleneck: instead of preserving the full waveform, it is trained to retain linguistic content, speaker timbre, and usable prosody/acoustics while discarding background noise, reverberation, microphone/channel artifacts, and other perceptually or task-irrelevant details. The system operates on log-mel spectrograms, uses a single-codebook discrete bottleneck based on Finite Scalar Quantization (FSQ), separates local time-varying tokens from a global utterance-level speaker embedding, and reconstructs speech through a Vocos-based vocoder. Its flagship operating point is 12.5 tokens/s, at which it is reported to achieve state-of-the-art tokenization efficiency, strong speech intelligibility and speaker similarity, and substantial downstream efficiency gains for text-to-speech and voice conversion (Kwek et al., 3 Jun 2026).

1. Conceptual framing and problem definition

CleanCodec begins from the observation that conventional neural audio codecs often allocate bitrate to signal detail that is not central to speech understanding or downstream modeling. In this view, a speech tokenizer should not be optimized to reproduce every waveform irregularity. Rather, it should preserve the speech information that is most relevant to human perception and to applications such as TTS, voice conversion, and speech-language modeling, while compressing away nuisance variation (Kwek et al., 3 Jun 2026).

The paper situates this position against three broad codec/tokenizer families. Semantic tokenizers are efficient, but tend to under-represent speaker identity and prosody, which harms reconstruction quality. Acoustic codecs can reconstruct audio well, but often preserve too much low-level detail and frequently rely on many RVQ codebooks, increasing token rate and downstream modeling complexity. Disentangled codecs reduce token rate by separating global and local factors, but may degrade reconstruction fidelity or robustness, particularly out of domain. CleanCodec’s proposal is to combine disentanglement, single-stream tokenization, and denoising supervision into a speech-centric codec whose bottleneck is explicitly pressured to keep only salient information (Kwek et al., 3 Jun 2026).

This framing is consistent with a broader benchmarking concern in codec research: the ideal codec should preserve content, speaker identity, paralinguistic information, and general audio information under low bitrate, rather than optimizing only a narrow set of waveform metrics (Wu et al., 2024). CleanCodec addresses that question specifically for speech by making denoising part of the tokenization objective itself.

A common misconception is that the best speech codec is necessarily the one that most faithfully reconstructs the observed input signal. CleanCodec explicitly rejects that assumption. Its central claim is that strict fidelity to the degraded input can be counterproductive when the objective is an efficient, robust, speech-oriented token stream. The codec is therefore designed to reconstruct clean speech targets from corrupted speech inputs, forcing the bottleneck to prioritize information that remains useful after nuisance variability is removed (Kwek et al., 3 Jun 2026).

2. Representation design and model architecture

CleanCodec is a disentangled VQ-VAE-style codec operating on log-mel spectrograms. The processing pipeline is: waveform $\rightarrow$ log-mel spectrogram $\rightarrow$ local and global encoders $\rightarrow$ discrete local tokens + continuous global embedding $\rightarrow$ acoustic and semantic decoders $\rightarrow$ mel reconstruction $\rightarrow$ vocoder waveform synthesis (Kwek et al., 3 Jun 2026).

The input waveform is sampled at 24 kHz. Conversion to log-mel reduces temporal resolution to 62.5 Hz. The local encoder then downsamples further by a factor of 5, 2, or 1, producing three operating points: $r=\frac{62.5}{d}\ \text{tokens/s},$ with [email protected], [email protected], and [email protected] corresponding to $d=5,2,1$ respectively (Kwek et al., 3 Jun 2026).

The local encoder is a fully convolutional architecture based on 1D ConvNeXt blocks, following a Vocos-style design. Its appendix configuration specifies 16 layers, hidden dimension 768, depthwise kernel size 27, and MLP intermediate size 3072. This branch produces the time-varying latent sequence that is quantized into discrete tokens (Kwek et al., 3 Jun 2026).

Quantization is performed by FSQ, not by standard VQ or RVQ. The quantizer is single-codebook and parameter-free, with levels: $[8,8,8,8,8].$ This yields

$8^5 = 32768$

possible codes. Each compressed frame therefore emits exactly one discrete symbol from a 32,768-way codebook, producing a single token stream rather than multiple RVQ streams (Kwek et al., 3 Jun 2026).

The global encoder models utterance-level speaker and timbre information. It is inspired by Kanade, uses modified ConvNeXt blocks from NeXt-TDNN, and ends with attentive statistics pooling. Its appendix configuration specifies 8 layers, hidden dimension 512, depthwise kernel size 7, MLP intermediate size 1536, and a final global embedding dimension of 256. During decoding, this global embedding is concatenated to the local token representation, so the local stream is not required to carry all speaker information (Kwek et al., 3 Jun 2026).

Decoding is split into two branches. The acoustic decoder reconstructs the target log-mel spectrogram. It uses a 1D ConvNeXt backbone with the same size as the local encoder: 16 layers, 768 dimension, kernel size 27, and MLP size 3072. The semantic decoder reconstructs WavLM-large features and uses the same architecture except for kernel size 53. The semantic decoder exists only during training and is discarded afterward (Kwek et al., 3 Jun 2026).

Waveform synthesis is delegated to a Vocos-based vocoder trained by the authors. Its appendix configuration specifies 16 layers, 1536 hidden dimension, depthwise kernel size 5, and MLP intermediate size 4608. Reported model sizes are 243M parameters for the autoencoder, 228M for the vocoder, and 471M total; after discarding the semantic decoder at inference, the deployed model size is 396M (Kwek et al., 3 Jun 2026).

The reported system is effectively non-causal and non-streaming. The use of attentive statistics pooling and full-utterance global conditioning implies an utterance-level design rather than a low-latency streaming codec (Kwek et al., 3 Jun 2026).

3. Objectives, supervision, and perceptually guided denoising

Although CleanCodec is motivated as a selective information bottleneck, the paper does not formulate an explicit mutual-information objective such as $\rightarrow$0. The bottleneck interpretation is conceptual. The actual training objective is expressed through supervised reconstruction and conditioning losses on mel acoustics, semantic features, and speaker embeddings (Kwek et al., 3 Jun 2026).

The paper defines:

• $\rightarrow$1: true log-mel spectrogram,
• $\rightarrow$2: true SSL features from WavLM-large,
• $\rightarrow$3: true speaker embedding from TitaNet-large,
• $\rightarrow$4: predicted global embedding,
• $\rightarrow$5: predicted log-mel spectrogram,
• $\rightarrow$6: predicted SSL features.

The explicit losses are: $\rightarrow$7

$\rightarrow$8

$\rightarrow$9

where $\rightarrow$0 is a learned linear projection into the pretrained speaker-embedding space (Kwek et al., 3 Jun 2026).

The vocoder follows the Vocos training recipe. Its loss is composed of multiscale mel spectrogram loss, adversarial loss, and feature matching loss. The discriminator setup includes a multi-period discriminator in the style of HiFi-GAN and a multi-band STFT discriminator at multiple timescales in the style of UnivNet. The exact formulas are not printed in the excerpted paper (Kwek et al., 3 Jun 2026).

What makes CleanCodec distinctive is that these objectives are applied under degraded-input, clean-target supervision. During training, the codec receives corrupted speech and is asked to reconstruct the original clean speech. The denoising effect is therefore not imposed by an explicit psychoacoustic mask or rule at inference, but is learned through supervision under a narrow bottleneck. Operationally, the model learns that noise, reverberation, low-pass effects, codec artifacts, and resampling artifacts are not worth encoding when the target is clean speech (Kwek et al., 3 Jun 2026).

Each degradation is applied with 50% probability. The training augmentations are:

• Reverberation with RT60 sampled from $\rightarrow$1 seconds, room dimensions from $\rightarrow$2 meters, simulated with pyroomacoustics, with RIRs of size $\rightarrow$3 samples filtered out.
• Background noise from AudioSet, FSD50K, and WHAM!, added at SNR $\rightarrow$4 dB.
• Low-pass filtering with cutoff between 2–8 kHz.
• Resampling to $\rightarrow$5 kHz and then back to 24 kHz.
• Codec degradation via MP3 compression with bitrate 32–245 kbps (Kwek et al., 3 Jun 2026).

A second common misconception is that denoising supervision necessarily removes useful speaker or semantic structure. CleanCodec’s design counteracts that risk in two ways. The semantic decoder forces local tokens to retain strong linguistic information, and the TitaNet-supervised global encoder forces the global branch to preserve speaker identity. The paper’s ablations show that both mechanisms are structurally important rather than auxiliary (Kwek et al., 3 Jun 2026).

4. Tokenization efficiency, bitrate implications, and training regime

The flagship operating point is 12.5 tokens/s. This is achieved by converting the waveform to log-mel at 62.5 frames/s, downsampling the local encoder output by 5×, and emitting one FSQ code per compressed frame. Because the codebook cardinality is $\rightarrow$6, the local stream can be interpreted as carrying approximately

$\rightarrow$7

excluding storage overhead and the continuous global embedding (Kwek et al., 3 Jun 2026).

Within the paper’s baseline set, the reported token rates are Qwen3: 200 / 100 / 50 t/s, Mimi: 100 t/s, BiCodec: 50 t/s, XCodec2: 50 t/s, WavTokenizer: 40 t/s, FocalCodec: 12.5 t/s, and Kanade: 12.5 t/s. CleanCodec therefore matches the ultra-low-rate regime of FocalCodec and Kanade, but the paper argues that it uses this rate more effectively because it preserves cleaner speech structure rather than reproducing all observed detail (Kwek et al., 3 Jun 2026).

The system also reports strong efficiency at inference. Appendix measurements give Encoding RTF = 0.0011 (928× realtime), Decoding RTF = 0.0034 (291× realtime), and Total RTF = 0.0045 (221× realtime). In the downstream TTS setting, the lower token rate materially reduces autoregressive sequence length: compared with a 200 t/s tokenizer, the paper reports up to 17× faster inference, using the ratio between Qwen3 RTF = 2.930 and CleanCodec RTF = 0.170 (Kwek et al., 3 Jun 2026).

Codebook utilization is measured with normalized entropy: $\rightarrow$8 For [email protected], the reported value is 0.985, indicating strong codebook usage (Kwek et al., 3 Jun 2026).

Training uses approximately 2,400 hours of speech: the full training split of LibriTTS-R (585 hours, 2,456 speakers) and an 1,800-hour subset of Emilia-YODAS. Audio is sampled at 24 kHz; training segments have length 230,400 samples (9.6 seconds); utterances shorter than that are repeated and trimmed. All audio is preprocessed with a high-pass filter at 80 Hz and loudness normalization to peak amplitude 0.75 (Kwek et al., 3 Jun 2026).

Optimization follows a two-stage schedule. In Stage 1, the autoencoder and vocoder are trained independently for 1M steps each. In Stage 2, the system is jointly fine-tuned end to end for 200K steps. The optimizer is

$\rightarrow$9

with learning rate $\rightarrow$0 in stage 1 and $\rightarrow$1 in stage 2. Training uses 1× A100 80GB and takes about one week (Kwek et al., 3 Jun 2026).

5. Empirical performance, robustness, and ablation evidence

On LibriTTS test-clean, the paper reports the following for [email protected]: WER 2.7, CER 1.4, SIM 0.86, UTMOS 4.32, SI-SDR 21.96, and Mel L1 0.44. On LibriTTS test-other, it reports WER 5.7, CER 3.2, SIM 0.84, UTMOS 4.17, and Mel L1 0.44. At the same 12.5 t/s rate, Kanade reports WER 4.0, CER 2.1, SIM 0.65, UTMOS 4.22, SI-SDR 23.60, and Mel L1 0.57, while FocalCodec reports WER 8.3, CER 4.5, SIM 0.45, UTMOS 4.21, SI-SDR 25.19, and Mel L1 0.78. The paper’s interpretation is that CleanCodec substantially improves intelligibility and speaker similarity at the same token rate, while remaining competitive with many higher-rate codecs (Kwek et al., 3 Jun 2026).

Higher-rate variants extend the same design. On test-clean, [email protected] reports WER 1.3, CER 0.6, SIM 0.90, and Mel L1 0.29, which the paper presents as close to Qwen3@200, also reported at WER 1.3, CER 0.7, SIM 0.90, and Mel L1 0.29. The paper treats this as one of its strongest efficiency results: comparable reconstruction behavior at about 3× fewer tokens (Kwek et al., 3 Jun 2026).

Out-of-domain robustness is a central empirical claim. At 12.5 t/s, on Expresso, CleanCodec reports WER 3.9, CER 1.9, SIM 0.82, and Mel L1 0.45; on AISHELL-3, CER 1.5, SIM 0.84, and Mel L1 0.39; on CML-TTS, WER 12.1, CER 5.1, and SIM 0.78. These results are markedly stronger than the corresponding Kanade and FocalCodec numbers reported in the same tables, and the paper attributes this to denoising training that preserves robust speech structure instead of overfitting surface acoustics (Kwek et al., 3 Jun 2026).

The paper also evaluates disentanglement directly. For speaker verification from the global representation on VoxCeleb-1, [email protected] achieves ACC 99.92 and EER 0.58. For ASR from local tokens, it reports WER 5.6 and CER 3.0. This supports the claim that the global branch captures speaker information effectively while the local token stream preserves linguistic information (Kwek et al., 3 Jun 2026).

In voice conversion on VCTK, using local tokens from a source utterance and the global embedding of a reference speaker, [email protected] reports SIM 0.81, UTMOS 4.09, WER 1.4, and CER 0.7. The paper contrasts this with stronger speaker leakage in some competing codecs, noting that local-token speaker contamination can limit the effect of swapping global embeddings (Kwek et al., 3 Jun 2026).

For autoregressive text-to-speech on Seed-TTS-eval, [email protected] reports Train time 01:00, RTF 0.170, SIM 0.56, and WER 3.9. The reported Qwen3@200 baseline gives Train time 10:02, RTF 2.930, SIM 0.36, and WER 9.1. These figures underpin the claim that lower token rate improves both modelability and inference speed in downstream generation (Kwek et al., 3 Jun 2026).

The ablation study clarifies the role of the major components. Replacing two-stage training with one-stage end-to-end adversarial training degrades all reported metrics: WER 2.7 → 4.1, CER 1.4 → 2.2, SIM 0.86 → 0.80, UTMOS 4.32 → 4.01, SI-SDR 21.96 → 18.58, and Mel L1 0.44 → 0.57. Removing the semantic SSL branch causes a large linguistic collapse: WER 2.7 → 8.1 and CER 1.4 → 4.3. Removing TitaNet global conditioning lowers SIM 0.86 → 0.74, while replacing it with WavLM-based speaker conditioning yields SIM 0.77. Removing denoising yields a more subtle but revealing pattern: WER 2.7 → 3.4, CER 1.4 → 1.7, SIM 0.86 → 0.82, UTMOS 4.32 → 4.24, SI-SDR 21.96 → 20.55, while Mel L1 slightly improves from 0.44 to 0.41. The paper uses this to argue that lower mel error does not necessarily imply better speech tokenization: the denoising bottleneck sacrifices a small amount of exact input fidelity in order to preserve more task-relevant information (Kwek et al., 3 Jun 2026).

6. Relation to adjacent codec research, interpretation, and limitations

CleanCodec belongs to a broader movement toward low-rate, single-stream, downstream-friendly audio tokenization, but it occupies a distinct position within that literature. Earlier work on joint denoising and compression, such as “Enhancing into the codec: Noise Robust Speech Coding with Vector-Quantized Autoencoders” (Casebeer et al., 2021), already showed that a codec can be trained on noisy input and clean output through a discrete bottleneck. CleanCodec extends that general principle with a log-mel front end, explicit global/local disentanglement, FSQ single-stream tokenization, semantic supervision, and modern downstream evaluation (Kwek et al., 3 Jun 2026).

Relative to other single-codebook speech codecs, CleanCodec is closest in spirit to designs that simplify token modeling by producing one discrete stream. “Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation” uses one 8192-entry codebook, a continuous global embedding, and about 23 discrete tokens per second at 304 bps, with the explicit goal of LLM-friendly single-sequence generation (Li et al., 2024). “LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec” similarly combines low bitrate and speaker decoupling, reporting 0.45 kbps at 50 Hz with $\rightarrow$2 and 0.25 kbps at 25 Hz with $\rightarrow$3, using a multi-stage framework with speaker perturbation and a discrete token vocoder (Guo et al., 2024). CleanCodec differs from both by centering the codec around perceptually guided denoising rather than primarily around speaker removal or phonetic abstraction alone.

Another adjacent direction addresses token simplicity not by denoising but by making the decoder more robust to quantization. “Self-Guidance: Enhancing Neural Codecs via Decoder Manifold Alignment” introduces a training-only decoder feature-matching loss that improves reconstruction in XCodec2, yields no inference-time changes, and supports a 4× codebook reduction without fidelity loss in the reported setup (Li et al., 11 Jun 2026). This suggests an orthogonal design axis: CleanCodec reduces nuisance information before or during tokenization, whereas self-guidance improves decoder tolerance to quantization error after tokenization.

At the level of modality scope, CleanCodec is more specialized than recent unified audio codecs. “UniSRCodec: Unified and Low-Bitrate Single Codebook Codec with Sub-Band Reconstruction” targets speech, music, and general audio, operates at 44.1 kHz, compresses in the Mel-spectrogram domain, and reports 40 tokens per second and 0.52 kbps at its main low-bitrate operating point (Zhang et al., 6 Jan 2026). CleanCodec, by contrast, is explicitly a speech tokenizer and does not claim equivalent unified audio coverage.

The main limitations reported for CleanCodec are several. Disentanglement degrades at higher token rates, as voice-conversion results suggest that more speaker information leaks into local tokens when capacity rises. The design is non-streaming and utterance-level. Training data, while large for a speech codec, remain moderate by frontier standards at approximately 2.4k hours. The paper also notes that robustness to all languages, conversational telephony, pathological speech, or extreme acoustics is not fully established. Finally, the system’s improved speech tokenization can also aid voice cloning or related misuse (Kwek et al., 3 Jun 2026).

Taken together, these features make CleanCodec best understood as a speech-centric, denoising, disentangled, single-stream neural tokenizer. Its central claim is not merely that bitrate can be reduced, but that the token budget can be spent more efficiently if the codec is trained to encode clean, perceptually important speech structure rather than the full degraded signal. In that sense, CleanCodec represents a specific answer to a broader codec-design question: not how to preserve everything, but how to preserve the right things.