---
title: 'CLaM-TTS: Zero-Shot Text-to-Speech'
url: https://www.emergentmind.com/topics/klaam-tts-system
type: topic
---

# CLaM-TTS: Zero-Shot Text-to-Speech

CLaM-TTS (Codec Language Model for Text-to-Speech) is a zero-shot TTS system designed to address scalability challenges associated with neural audio codec-based generation. It leverages a combination of probabilistic residual vector quantization within a Mel-spectrogram variational autoencoder (Mel-VAE) and a latent Transformer language model to achieve high compression efficiency and enable simultaneous multi-token generation. CLaM-TTS delivers competitive or superior results in naturalness, intelligibility, and speaker similarity compared to state-of-the-art neural codec TTS models [2404.02781].

## 1. System Architecture

CLaM-TTS integrates two core components: a probabilistic residual vector quantization-enabled Mel-VAE (Mel-VAE w/ PRVQ) and a latent Transformer-based language model (LM). The architecture operates as follows:

- **Stage 1: Mel-VAE (w/ Probabilistic Residual VQ)**
  - The encoder $\varphi$ maps an input mel-spectrogram $y$ into a sequence of continuous latent vectors $z_1,\dots,z_T$ at a reduced temporal rate (e.g., 10 Hz).
  - Each $z_t$ is quantized by a depth-$D$ probabilistic residual vector quantizer $\psi$ into a tuple of discrete code indices $c_{t,1:D}$, producing latent code tensors.
  - The Mel-VAE decoder $\omega$ reconstructs the output spectrogram $\hat{y}$ from the quantized embeddings $\hat{z}_{1:T}$.

- **Stage 2: Latent Language Model**
  - Input text $x$ is embedded using a frozen text encoder (ByT5-large).
  - An autoregressive Transformer decoder predicts a mixture-of-Gaussians (MoG) distribution over $z_t$ at each time step, conditioned on $x$ and previously generated codes $c_{<t}$.
  - At generation, the latent $z_t$ is sampled, quantized by the learned PRVQ, and all code indices for that timestep are emitted in parallel.
  - An EOS head signals termination via a thresholded probability.

The stack of discrete codes $c_{1:T}$ is decoded by the Mel-VAE and synthesized as audio using a BigVGAN vocoder.

**Key technical parameters** include a codeword rate of 10 Hz, RVQ depth $D=32$ with vocabulary size $V=1024$ per depth, and a 12-layer Transformer decoder with 16 attention heads and 1,536 hidden dimensions.

**End-to-end inference pseudocode:**
```python
def CLaM_TTS_Generate(text x):
    C = []
    for t in 1..T_max:
        h_t = TransformerDecoder(x, C)
        p_k = Softmax(W_k @ h_t)
        k_t = sample(p_k)
        mu_t = MLP_mu(k_t, h_t)
        z_t = mu_t + sigma_psi * epsilon    # epsilon~N(0,I)
        c_t = RQ_psi(z_t)
        C.append(c_t)
        if sigmoid(W_eos @ h_t) > 0.5:
            break
    y_hat = MelVAE_Decoder({sum_{d=1}^D e_psi(c_{t,d}; d)}_{t=1:|C|})
    waveform = BigVGAN(y_hat)
    return waveform
```
[2404.02781]

## 2. Probabilistic Residual Vector Quantization (PRVQ)

PRVQ in CLaM-TTS extends standard residual vector quantization (RVQ) by learning the codebooks within a variational-inference framework, preventing codebook collapse and improving the utilization of codeword space.

- **Standard RVQ** iteratively assigns code indices to minimize squared reconstruction error:
  \[
  r_{t,0} = z_t \\
  c_{t,d} = \arg\min_{c'\in[1,V]} \| r_{t,d-1} - e_\psi(c';d) \|^2 \\
  r_{t,d} = r_{t,d-1} - e_\psi(c_{t,d};d)
  \]
  The quantized vector $\hat{z}_t$ is reconstructed as $\sum_{d=1}^D e_\psi(c_{t,d};d)$.

- **PRVQ** instead adopts a mean-field variational posterior:
  \[
  q(c_{t,1:D}|z_t) = \prod_{d=1}^D q(c_{t,d}|z_t)
  \]
  The optimal code posterior for component $d$ is derived from the evidence lower bound (ELBO):
  \[
  q^*(c_{t,d}|z_t) \propto \exp(\mathbb{E}_{q(c_{t,-d}|z_t)}[\log p_\psi(z_t|c_{t,d},c_{t,-d})])
  \]
  In practice, the expectation is approximated by conditioning on the greedy codes from standard RVQ.

- **Empirical Effects:** PRVQ yields more uniform usage of codewords and higher reconstruction quality at fixed bit rates. For speech reconstruction, PRVQ achieves PESQ 2.63 (vs. 2.54 for baseline RVQ) and ViSQOL 4.48 (vs. 4.44); when compared to EnCodec (6 kbps), CLaM-TTS demonstrates notable improvements (PESQ 2.95 vs. 2.59, ViSQOL 4.66 vs. 4.26), combining high compression (5 s audio $\to$ 1,600 tokens at 10 Hz, $D=32$) with efficiency: all 32 codes per frame are emitted in parallel, drastically reducing autoregressive steps.

## 3. Multi-Token Generation Strategy

A principal advantage over prior neural codec LMs is the elimination of cascaded, framewise autoregression over multiple audio token streams. In CLaM-TTS:

- The language model predicts the entire continuous latent $z_t$ (as a $K$-component MoG) directly at each timestep.
- The shared PRVQ quantizes $z_t$ to the complete tuple $(c_{t,1},\ldots,c_{t,D})$ in a single step.

As a result, the LM loop proceeds over $T$ steps, not $T \times D$ or $T \times S$ for $S$ audio token streams.

**Training objective:**
\[
\mathcal{L}_{VB}(\theta) = -\sum_{t=1}^T \left\{ \min_{q(k)} \left[ 
\mathbb{E}_{p_\psi(z_t|c_t)}[ \mathrm{KL}(p_\psi(z_t|c_t) \| p_\theta(z_t|x,c_{<t},k)) ] 
- \mathbb{E}_{q(k)}[ \log p_\theta(k|x,c_{<t}) ] \right]
\right\}
\]
with an additional binary EOS cross-entropy loss.

During inference, the LM emits three parallel heads for each step: (1) softmax over mixture index $k$, (2) mixture means $\mu_\theta(k,\cdot)$, and (3) EOS probability. The process involves sampling $k$ and $z_t$, quantizing, appending codes, and repeating until EOS.

## 4. Text Encoder Choice and Pretraining Analysis

CLaM-TTS employs a frozen text encoder and systematically benchmarks several T5-derived models:

- T5 (subword)
- mT5 (multilingual)
- ByT5 (byte-level)
- Flan-T5 (instruction-tuned)
- T5-lm-adapt (prefix LM adapted)
- XPhoneBERT (phoneme-based)

**Ablation on English continuation (LibriSpeech test-clean):**

| Model         | WER (%) | CER (%) | SIM-o    |
|---------------|---------|---------|----------|
| XPhoneBERT    | 2.62    | 0.96    | 0.3943   |
| ByT5          | 2.79    | 1.00    | 0.3879   |
| T5            | 9.48    | 7.04    | -        |

Best results are obtained with phoneme-based models. Among subword/byte models, ByT5 yields the lowest WER and CER. This indicates that large-scale pretraining on byte-level data enhances LM-to-speech mapping, yet a slight advantage remains for phoneme supervision where available.

## 5. Experimental Protocol and Quantitative Performance

CLaM-TTS systems are trained in monolingual (CLaM-en; 55K hours English) and multilingual (CLaM-multi; 100K hours, 11 languages) settings. Baselines include YourTTS (VITS-based), Vall-E, SPEAR-TTS, and VoiceBox (non-autoregressive phoneme/duration model).

**Evaluation Tasks:**
- **Continuation:** Extend a 3 s speech prompt with matching transcript.
- **Cross-sentence:** Synthesize new text in the same voice after a 3 s speech prompt.

**Metrics:**
- Intelligibility: WER, CER (ASR via HuBERT-large, Whisper)
- Speaker Similarity: SIM-o (output–target), SIM-r (reconstruction–target via WavLM)
- Subjective: QMOS, SMOS, CMOS
- Inference speed: Time to produce 10 s on 1 × A100 GPU

**Key Results (English continuation):**

| Model      | WER (%) | CER (%) | SIM-o | SIM-r | Inference (s) |
|------------|---------|---------|-------|-------|---------------|
| GT         | 2.2     | 0.61    | 0.754 | 0.754 | -             |
| YourTTS    | 7.57    | 3.06    | 0.393 | -     | -             |
| Vall-E     | 3.8     | -       | 0.452 | 0.508 | 6.2           |
| VoiceBox   | 2.0     | -       | 0.593 | 0.616 | 6.4           |
| CLaM-en    | 2.36    | 0.79    | 0.477 | 0.513 | 4.15          |

On cross-sentence tasks, CLaM-en achieves WER 5.11%, CER 2.87%, and SIM-o 0.495 (all second-best among tested models). Subjective evaluations for quality and similarity also reflect gains: QMOS for CLaM-en is 3.87 ± 0.12 (vs. 2.39 ± 0.19 for YourTTS, 4.45 for GT), and SMOS is 3.49 ± 0.14.

In multilingual continuation over 11 languages, WER ranges from 4–20% with SIM-o between 0.42–0.60, showing broad applicability.

## 6. Trade-offs, Limitations, and Prospects

**Trade-offs:**
- Lower codeword rates (e.g., 10 Hz) reduce AR steps but can degrade perceptual quality (PESQ), while higher rates boost quality but increase LM computation. Empirically, 8× downsampling is effective.
- PRVQ combats codebook collapse, facilitating denser code usage, though excessive compression increases latent variance and challenges the LM.

**Speed vs. Expressiveness:**
- Single-pass LM with PRVQ supports ≈4 s/10 s utterance generation versus ≥6 s for cascaded approaches. Non-AR models with phoneme supervision may offer faster inference with fewer diffusion steps but require additional labeling.

**Limitations:**
- Autoregressive LMs can still exhibit issues such as word omission or repetition; non-AR paradigms (diffusion, flow-matching) may ameliorate these.
- Expressiveness is constrained by predominantly audiobook-style training data; increasing the diversity (conversational, singing) is likely to enhance style control.
- Conditioning on speaker metadata for age, accent, and emotion is promising.
- Further scaling in model size and dataset duration (>100K hours) is a projected avenue for progress toward ground-truth parity.

In summary, CLaM-TTS demonstrates that combining probabilistic RVQ-driven Mel-VAE compression with an efficient, multi-token latent Transformer LM establishes strong zero-shot TTS capabilities in terms of naturalness, intelligibility, speaker similarity, and inference speed, all without reliance on phoneme or duration supervision [2404.02781].

Source: https://www.emergentmind.com/topics/klaam-tts-system