SiTok: Speech Diffusion Tokenizer

• The paper introduces SiTok, which unifies extreme compression, high-fidelity reconstruction, and semantic richness by jointly optimizing a diffusion decoder with vector quantization and CTC-based semantic supervision.
• SiTok operates on stacked mel-spectrograms at 12.5 Hz (200 bits/s), demonstrating a balanced trade-off between low token rate and high-quality acoustic reconstruction.
• Scaling experiments reveal that an optimal configuration at 1.12B parameters achieves competitive error rates and effective downstream generation, confirming the value of integrated diffusion with semantic regularization.

Speech Diffusion Tokenizer (SiTok) is a speech tokenizer formulated as a diffusion autoencoder with vector quantization and semantic regularization, proposed to address a central constraint in speech language modeling: obtaining a discrete representation that is simultaneously compact enough for scalable language modeling, semantically rich enough for understanding, and expressive enough for high-fidelity reconstruction. In its default single-codebook setting, SiTok operates at 12.5 Hz and 200 bits per second, while targeting speech understanding, reconstruction, and downstream generation within one tokenizer interface (Wang et al., 6 Feb 2026).

1. Problem formulation and design objective

SiTok is motivated by a three-way tension in speech tokenization: extreme compression, high-fidelity reconstruction, and semantic richness. The paper argues that prior tokenizers typically realize only two of these three properties at once, especially in the low-token-rate regime. One challenge is the trade-off between encoding semantics for understanding and acoustics for reconstruction; the other is achieving both low token rate and low bit rate without collapsing performance (Wang et al., 6 Feb 2026).

The paper’s diagnosis is that this limitation is structural rather than merely a matter of scale. Deterministic reconstruction under aggressive quantization forces the latent space to collapse uncertainty: one code must stand for many acoustically plausible realizations, and deterministic decoders then either overfit low-level detail or lose linguistic structure. SiTok’s central response is to move the generative burden into a diffusion decoder while explicitly supervising the quantized latent space with text transcripts through CTC. In this formulation, the discrete codes are encouraged to preserve compressed, semantically useful structure, while the decoder models the conditional distribution of plausible speech given those codes (Wang et al., 6 Feb 2026).

This positioning distinguishes SiTok from earlier unified discrete tokenizers such as "SpeechTokenizer" (Zhang et al., 2023), which hierarchically disentangled semantics and acoustics across RVQ layers, and from two-stage systems that first tokenize speech and then separately train a diffusion model for reconstruction. SiTok instead treats quantization and diffusion reconstruction as one jointly optimized autoencoding problem (Wang et al., 6 Feb 2026).

2. Architecture and learning formulation

SiTok operates on 50 Hz, 128-bin mel-spectrograms rather than waveforms. Every four consecutive frames are stacked, reducing the effective tokenizer frame rate to 12.5 Hz. The core system contains five components: an input/downsampling stage, an encoder, a vector quantizer, an auxiliary CTC decoder, and a diffusion decoder. A separate Vocos-based vocoder converts reconstructed mels to waveform at 24 kHz (Wang et al., 6 Feb 2026).

Component	Role	Configuration
Encoder	Maps stacked mel input to latent features	16 causal Llama-style Transformer decoder blocks, hidden size 1536
Quantizer	Discretizes encoder output	Codebook dim 32, codebook size 65,536, EMA updates
CTC decoder	Applies semantic supervision on quantized latents	4 layers
Diffusion decoder	Reconstructs mel from quantized embeddings	16 non-causal Transformer layers
Vocoder	Converts mel to waveform	Vocos-based, 24 kHz

The encoder maps downsampled mel input $x$ to latent features $z = E_\theta(x)$. These are projected into the VQ space, quantized against a learned codebook $E = \{e_1, e_2, \dots, e_k\}$, and then mapped back to decoder conditioning space as quantized embeddings $z_q$. The quantizer uses EMA updates and the default operating point uses a single codebook, although the paper also studies RVQ by increasing the number of codebooks $CN$ (Wang et al., 6 Feb 2026).

A key design choice is that semantic supervision is applied after vector quantization. The paper adds an auxiliary decoder $D_{\mathrm{ctc}}$ on top of $z_q$ and predicts transcripts $y$ with CTC loss, so the quantized latent space itself is pushed toward linguistic structure. This contrasts with approaches that align pre-quantized representations to SSL teacher features by MSE or cosine loss (Wang et al., 6 Feb 2026).

The diffusion decoder is trained with a flow-matching objective. The noisy interpolation is defined by

$$x_t = t x + (1 - t)\epsilon, \qquad \epsilon \sim \mathcal{N}(0, I), \quad t \sim U(0,1),$$

and the decoder predicts a velocity field

$$v_\theta(x_t, t, z_q) = D_\phi(x_t, t, z_q) \rightarrow x - \epsilon.$$

The total objective is written as

$z = E_\theta(x)$0

In the appendix training pseudocode, the diffusion term is implemented as an L1 loss on the target velocity $z = E_\theta(x)$1 (Wang et al., 6 Feb 2026).

Two clarifications are central. First, SiTok is not a waveform codec; it reconstructs mel-spectrograms and delegates waveform synthesis to a vocoder. Second, diffusion is used only in decoding/reconstruction, not in encoding or quantization (Wang et al., 6 Feb 2026).

3. Tokenization regime, compression, and temporal granularity

SiTok’s signature operating point is 12.5 FPS / 12.5 TPS with 0.20 kbps in the single-codebook setting. Since the codebook size is $z = E_\theta(x)$2, each token carries 16 bits, yielding

$z = E_\theta(x)$3

With multiple codebooks, the token rate scales linearly with the number of codebooks: CN = 1 gives 12.5 TPS, CN = 2 gives 25 TPS, and CN = 4 gives 50 TPS (Wang et al., 6 Feb 2026).

This tokenization regime is unusually aggressive. The paper explicitly compares it with prior systems such as SpeechTokenizer, BigCodec, Mimi, X-codec 2, StableCodec, FireRedTTS Tokenizer, and Cosy Voice Tokenizer, emphasizing that SiTok drives both token rate and bitrate below those systems while remaining competitive in reconstruction and often superior in understanding (Wang et al., 6 Feb 2026).

Temporal granularity is not treated as a minor implementation detail. SiTok’s own frame-rate ablation reports a major collapse at 6.25 Hz, with WER 23.0, whereas 12.5 Hz is the balanced default and 25 Hz improves quality at the cost of doubled sequence length, reaching WER 3.05, SIM 0.688, and UTMOS 3.72 (Wang et al., 6 Feb 2026). A plausible implication is that SiTok’s default 12.5 Hz point occupies a practical middle ground between LM efficiency and semantic recoverability.

Independent work on frame-rate sensitivity in speech tokenization supports the importance of this design variable. A study on Mandarin and English reported that reducing frame rate from 12.5 Hz toward 5.0 Hz monotonically reduced codebook usage and hurt ASR, with much sharper degradation for Mandarin; at 5.0 Hz, Mandarin WER rose to 27.20 while English reached 8.48 (Zhang et al., 20 May 2025). This suggests that SiTok’s frame-rate choice should be read not only as a compression decision but also as a bias on what linguistic structure survives discretization.

4. Scaling, acceleration, and decoder refinements

SiTok is explicitly a scaling study. The paper trains on 2 million hours of in-house speech, described as multilingual but with English accounting for the vast majority, and scales the architecture up to 1.61B parameters (Wang et al., 6 Feb 2026).

Model	Encoder/Decoder layers	Parameters	Default-setting WER	Default-setting ASR
S	8 / 8	0.63B	4.18	5.24
B	12 / 12	0.88B	4.01	5.19
L	16 / 16	1.12B	4.06	4.95
XL	24 / 24	1.61B	3.84	5.07

Scaling improves reconstruction fairly consistently, but understanding peaks around the L model at 1.12B. The authors therefore identify L as the best overall balance. This is notable because the XL model gives the best reconstruction metrics but not the best downstream understanding, which the paper interprets as evidence that very large capacity may overemphasize acoustic detail relative to abstract features useful for understanding (Wang et al., 6 Feb 2026).

Because diffusion decoding is expensive, SiTok introduces two acceleration mechanisms. The first is Shortcut Fine-tuning, which freezes encoder and VQ and fine-tunes only the decoder so it can take larger denoising steps directly. The second is a Light-weight Diffusion Head, where the first 12 layers of the 16-layer decoder are executed once as a main body and the last 4 layers form a small iterative diffusion head (Wang et al., 6 Feb 2026).

These modifications materially affect runtime. The paper reports RTF 0.041 for 16 steps, 0.024 for 8 steps, and 0.013 for 4 steps. The lightweight head yields nearly unchanged reconstruction metrics: Base gives WER 4.06, SIM 0.641, UTMOS 3.44, while With light head gives WER 3.97, SIM 0.610, UTMOS 3.46 (Wang et al., 6 Feb 2026).

The decoder also benefits from two refinement procedures. Decoder finetuning freezes encoder and VQ and continues training only the decoder. Token classifier-free guidance (Token CFG) randomly drops all input tokens with 10% probability during training so the decoder learns both conditional and unconditional generation. At the single-codebook 12.5 Hz / 0.2 kbps setting, the progression is:

• Base SiTok: WER 4.06, SIM 0.641, UTMOS 3.44
• + decoder finetuning: WER 3.79, SIM 0.682, UTMOS 3.48
• + Token CFG: WER 3.34, SIM 0.635, UTMOS 3.60 (Wang et al., 6 Feb 2026)

5. Empirical behavior on reconstruction, understanding, and generation

SiTok is evaluated on three fronts: reconstruction, understanding, and generation. Reconstruction is measured on the SeedTTS test-en set with WER from whisper-large-v3, SIM as cosine similarity between WavLM-TDNN speaker embeddings, and UTMOS for speech quality (Wang et al., 6 Feb 2026).

At the default single-codebook 12.5 Hz / 0.2 kbps setting, SiTok reports:

• WER 4.06
• SIM 0.641
• UTMOS 3.44

With more codebooks, the expected rate–quality trade-off appears:

• CN = 2, 0.35 kbps: WER 3.17, SIM 0.658, UTMOS 3.44
• CN = 4, 0.70 kbps: WER 2.80, SIM 0.660, UTMOS 3.46 (Wang et al., 6 Feb 2026)

Against low-bitrate baselines, the paper emphasizes that SiTok operates at lower bitrate than StableCodec, FireRedTTS Tokenizer, and Cosy Voice Tokenizer while remaining comparable or better in WER and speaker similarity, though some baselines retain competitive UTMOS (Wang et al., 6 Feb 2026).

The semantic regularization ablation is one of the paper’s strongest results. Without CTC regularization at 12.5/12.5, performance collapses:

• WER 33.0
• SIM 0.495
• UTMOS 2.68
• CTC ASR 29.4
• LLM ASR 57.9
• ER 18.9
• KS 86.1

With CTC regularization at the same rate:

• WER 4.06
• SIM 0.641
• UTMOS 3.44
• CTC ASR 9.50
• LLM ASR 4.95
• ER 63.5
• SV 13.8
• KS 96.9 (Wang et al., 6 Feb 2026)

The paper identifies $z = E_\theta(x)$4 as the best balance. Too little CTC weakens semantics; too much hurts acoustic fidelity. This is presented as evidence that transcript supervision "anchors the quantized latent space to linguistic meaning" (Wang et al., 6 Feb 2026).

SiTok is also evaluated as the discrete interface for a 0.5B autoregressive TTS system, SiTok-AR-TTS, initialized from Qwen2.5-0.5B and trained on 100K hours of Emilia. On SeedTTS test-en, it reports:

• WER 2.46
• SIM 0.64
• RTF 0.234

Compared baselines include:

• Cosy Voice 2: WER 2.89, SIM 0.66, RTF 0.455
• SparkTTS: WER 2.50, SIM 0.57, RTF 0.601
• Llasa: WER 3.94, SIM 0.58, RTF 0.422 (Wang et al., 6 Feb 2026)

The paper’s claim is therefore not only that SiTok reconstructs well, but that its low-rate tokens remain usable for understanding and efficient downstream generation.

6. Relation to adjacent diffusion tokenizers, limitations, and open questions

SiTok occupies a specific point in the speech-tokenizer design space. Relative to hybrid systems such as "DiffSoundStream" (Yang et al., 27 Jun 2025), which combine WavLM semantic tokens and SoundStream acoustic tokens and use a latent diffusion decoder to reduce acoustic token depth, SiTok is more unified: quantization and diffusion reconstruction are optimized inside a single diffusion autoencoder rather than through separately defined semantic and acoustic streams. Relative to "TaDiCodec" (Wang et al., 22 Aug 2025), which also uses end-to-end diffusion training but adds text guidance and pushes compression to 6.25 Hz and 0.0875 kbps, SiTok relies instead on CTC supervision on quantized latents and adopts 12.5 Hz as its default balance point.

This suggests that recent diffusion-tokenizer research spans at least three distinct strategies: joint diffusion autoencoders such as SiTok (Wang et al., 6 Feb 2026), hybrid semantic-conditioned codecs with diffusion decoding (Yang et al., 27 Jun 2025), and text-aware end-to-end diffusion tokenizers (Wang et al., 22 Aug 2025). The shared theme is that diffusion or flow-matching decoders are used to make very low-rate tokenization viable, but the information allocated to the tokens themselves differs substantially across systems.

Several common misconceptions are addressed by the SiTok paper itself. SiTok is not a purely semantic tokenizer; its tokens are jointly shaped by reconstruction, quantization, and transcript supervision. It is also not a standard diffusion model over waveforms; diffusion operates on mel-spectrogram reconstruction conditioned on quantized speech embeddings (Wang et al., 6 Feb 2026).

The paper is explicit about limitations. First, despite strong results, SiTok "still falls short of continuous feature representations." Second, diffusion decoding remains a deployment cost even after shortcut fine-tuning and lightweight heads. Third, streaming generation is difficult, and the authors note ongoing investigation of chunk-wise autoregressive diffusion for low-latency or streaming output. Fourth, the reported gains rely on very large-scale training—up to 1.6B parameters and 2 million hours of data. Fifth, although the dataset is multilingual, English accounts for the vast majority, so multilingual robustness is not deeply characterized. Finally, model scaling reveals an unresolved tension: larger models improve reconstruction more consistently than understanding, implying that acoustic precision and abstract semantic structure are not yet perfectly aligned in the architecture (Wang et al., 6 Feb 2026).

Within the broader history of speech tokenization, SiTok’s main contribution is to recast temporal compression, semantic supervision, and reconstruction uncertainty as one optimization problem. Its defining thesis is that under ultra-low-rate discrete speech representation, the decoder should not be deterministic. Instead, the quantized codes should preserve the most useful compressed structure—especially linguistic structure—while a diffusion decoder reconstructs high-fidelity speech by modeling what quantization necessarily discards (Wang et al., 6 Feb 2026).