---
title: 'XTTS: Multilingual Zero-Shot TTS'
url: https://www.emergentmind.com/topics/xtts-model
type: topic
---

# XTTS: Multilingual Zero-Shot TTS

XTTS is an extensible, massively multilingual zero-shot text-to-speech (ZS-TTS) model designed to synthesize high-fidelity speech in a large variety of speaker identities and languages without explicit speaker- or language-specific fine-tuning. Built on the Tortoise TTS framework and integrating advances from YourTTS, VALL-E X, Mega-TTS 2, and Voicebox, XTTS introduces architectural modifications and training optimizations enabling robust voice-cloning, rapid inference, and broad language support—critically extending coverage to low- and medium-resource languages such as Bangla and Arabic dialects [2406.04904][2502.05729][2406.16751].

## 1. Core Architecture

XTTS utilizes a five-component pipeline inherited from Tortoise, but leverages three principal modules for synthesis:

- **VQ-VAE Audio Encoder/Decoder**: Converts raw waveform inputs (or mel-spectrograms $\mathbf{M}\in\mathbb{R}^{T\times F}$) to continuous latent vectors, discretizes via a learned codebook, and reconstructs $\hat{\mathbf{x}} = D(z)$.
- **GPT-2-Based Autoregressive Prior**: Acts as a joint “acoustic language model,” autoregressively predicting both text tokens ($t_1\ldots t_n$) and discrete audio tokens ($a_1\ldots a_l$) using self-attention over concatenated linguistic and acoustic streams. Speaker identity is injected via a Perceiver Resampler producing a fixed-length speaker embedding $\mathbf{h}$.
- **HiFi-GAN Vocoder**: Decodes latent representations or predicted audio tokens into high-fidelity waveforms, further conditioned on speaker embeddings.

The input sequence structure is:

```
[BOS_SPK], s_1…s_k, [BOS_TTS], [lang], ([dialect]), t_1…t_n, [EOS_TTS], [BOS_AUD], a_1…a_l, [EOS_AUD]
```

A detailed schematic is summarized in the table below:

| Module                      | Function                                   | Key Innovations             |
|-----------------------------|--------------------------------------------|-----------------------------|
| VQ-VAE                      | Quantizes waveform/mel into codebook tokens| Aggressive codebook pruning |
| GPT-2 Autoregressive Prior  | Models text+audio code sequences           | Perceiver multi-slice prompt|
| HiFi-GAN Vocoder            | Generates waveform from tokens+embedding   | Speaker embedding addition  |

XTTS parameterization includes 13M for VQ-VAE, 443M for GPT-2 prior, and 26M for HiFi-GAN [2406.04904].

## 2. Multilingual and Dialectal Adaptation

XTTS was pre-trained on 27.3k hours of speech across 16 languages, leveraging a single shared BPE vocabulary of 6.6k tokens and romanizing CJK scripts to prevent script drift. No explicit language embedding is used; instead, the model learns pronunciation and prosody via joint text/audio conditioning. Language batch balancing prevents English over-representation [2406.04904].

To extend capabilities for Bangla (in BnTTS) and Arabic dialects:

- **Bangla Adaptation**:
  - Augments phoneme inventory to include aspirated stops, retroflex consonants, and inherent vowels [2502.05729].
  - Two-stage pretraining—five epochs of partial-audio prompting, one epoch of full prompt—enhances modeling for short and long utterances.
  - No architectural changes for language addition; the phoneme set $\mathcal{P}_{bn}$ is merged into the token codebook.

- **Arabic Dialects Adaptation**:
  - Vocabulary expanded with 22 new dialect tokens (21 regional, 1 MSA), growing from 6,681 to 6,703 entries [2406.16751].
  - Dialect information injected as token embeddings initialized from $\mathcal{N}(0,1)$.
  - Pseudo-labeling via ensemble voting from eight dialect ID models assigns dialect conditioning.

## 3. Training Objectives and Optimization

XTTS optimization involves several loss terms, often in parallel channels:

- **VQ-VAE Loss**:
  $$
  \mathcal{L}_{VQ\text{-}VAE} = \mathbb{E}_x \left[ \|x - D(\mathrm{quantize}(E(x)))\|^2 \right] + \beta \| \mathrm{sg}(E(x)) - z \|^2 + \| E(x) - \mathrm{sg}(z) \|^2
  $$
- **Autoregressive Cross-Entropy Loss** (GPT-2):
  $$
  \mathcal{L}_{CE} = -\sum_{t=1}^T \log P_\theta(y_t | y_{<t}, s, \mathrm{lang}, [\mathrm{dialect}])
  $$
- **HiFi-GAN Generator Loss** (adversarial, feature matching, and mel-reconstruction):
  $$
  \mathcal{L}_G = \mathcal{L}_{\text{adv}} + \lambda_{\mathrm{FM}} \mathcal{L}_{\mathrm{FM}} + \lambda_{\mathrm{mel}} \|\mathrm{STFT}(x) - \mathrm{STFT}(\hat{x})\|_1
  $$
- **Speaker Consistency Loss** (SCL, from YourTTS):
  $$
  L_{\mathrm{SCL}} = 1 - \cos \left( S(x_{\mathrm{ref}}), S(\hat{x}) \right )
  $$

In few-shot speaker adaptation (BnTTS), all but the audio and speaker encoders are updated on multi-speaker studio data; no explicit speaker classification loss is required [2502.05729].

## 4. Evaluation Metrics and Empirical Performance

XTTS and its fine-tuned derivatives are assessed by:

- **Word/Character Error Rate (WER/CER)**: $\mathrm{CER} = \frac{S+D+I}{N}$; WER computed via Whisper-Large and Jiwer.
- **Speaker Embedding Cosine Similarity (SECS)**:
  $$
  \mathrm{SECS}(x,y) = \frac{ \langle s(x), s(y) \rangle }{ \| s(x) \| \| s(y) \| }
  $$
- **Mean Opinion Score (MOS/SMOS)**: 1–5, 0.5 increments, scored by native annotators.
- **SpeechBERTScore**: Maximum cosine similarity over SSL embeddings between generated and reference speech.
- **Duration Equality**: $1 / \max(a/b, b/a)$ for durations $a, b$.

Reported results [2406.16751][2502.05729][2406.04904]:

| Task / Dataset        | Model                | WER/CER      | SECS        | MOS/SMOS           |
|----------------------|--------------------- |------------- |------------ |--------------------|
| QASR (Arabic, zero-shot) | Baseline XTTS       | 6.42%        | 0.755       | —                  |
| QASR (fine-tuned, w/ dialect) | Fine-tuned      | 17.96%       | 0.766       | —                  |
| In-house dialects    | Baseline XTTS        | 47.16%       | 0.790       | 3.61               |
| In-house dialects    | Fine-tuned, w/ dialect | 62.74%      | 0.825       | 3.19               |
| FLORES+ (multi-lang) | XTTS (avg 16 lang.)  | CER=2.06     | 0.50        | —                  |
| Bangla (BnTTS-n, few-shot)| Ground truth     | —            | 0.548       | 4.809 (SMOS)       |
| Bangla (BnTTS-n)     | Fine-tuned           | —            | 0.586       | 4.601 (SMOS)       |

MOS in Bangla attains 4.601 for few-shot adaptation, approaching ground truth (4.809) [2502.05729]. Speaker similarity (SECS) is enhanced by dialect conditioning ($\Delta$SECS ≈ 0.035), notably for clean, in-house dialect data [2406.16751].

## 5. Zero-Shot and Few-Shot Speaker Adaptation

XTTS operates zero-shot by encoding a short reference utterance through the Perceiver Conditioning Encoder, synthesizing target text in the given speaker's voice with no model updates [2406.04904][2502.05729]. The adoption of multi-slice audio conditioning and Speaker Consistency Loss allows accurate speaker mimicry, including cross-lingual and cross-dialect transfer.

In few-shot settings (BnTTS), the pipeline is briefly fine-tuned on specific speaker data (20 min, four speakers, 10 epochs), yielding improved MOS, naturalness, and clarity over baseline zero-shot results [2502.05729].

Strengths include:

- High-fidelity cross-lingual voice cloning without explicit fine-tuning per speaker.
- Robust handling of unseen languages/dialects via vocabulary/phoneme augmentation and dialect token injection.

Limitations highlighted:

- Increased WER after fine-tuning, suggesting degraded phoneme-to-waveform alignment [2406.16751].
- Gender imbalance in source datasets resulting in reduced voice quality for under-represented groups.
- Pseudo-labeling for dialect annotation may introduce noise; properly annotated dialect data could further enhance performance.

## 6. Training, Deployment, and Model Optimizations

XTTS training leverages mixed GPU parallelism and large-batch gradient accumulation stabilizing large-model optimization, typically on clusters of NVIDIA A100 GPUs. Hyperparameters (e.g., AdamW optimizer, learning rate schedules) are inherited from base XTTS defaults; Bangla-specific models tune hyperparameters to mitigate overrunning on short inputs [2406.16751][2502.05729]. Inferences are tuned for output diversity and stability via top-$k$, top-$p$, temperature, length- and repetition-penalties.

Key optimizations:

- **Perceiver Resampler**: Replaces the single style token paradigm with 32 embeddings for richer speaker/style representation without sequence-length inflation.
- **Low Frame-Rate Tokenization**: At 21.53 Hz (vs. 75 Hz for VALL-E), drastically reduces autoregressive sequence length.
- **Shared BPE Vocabulary**: Enables efficient multilingual pretraining; romanization of non-Latin scripts normalizes input representations.

## 7. Comparative Analysis and Future Directions

XTTS establishes state-of-the-art (SOTA) performance for zero-shot multilingual TTS, surpassing Mega-TTS 2, StyleTTS 2, and YourTTS in most supported languages on CER and SECS. Subjective MOS and speaker similarity remain competitive but modestly lower than heavily fine-tuned monolingual systems [2406.04904].

Proposed future work includes:

- **End-to-End VQ-VAE Vocoding**: Improving the vector-quantized decoder to bypass the external HiFi-GAN stage and enhance synthesis quality.
- **Disentangled Prosody/Speaker Embeddings**: Factoring out prosody from speaker identity for flexible, cross-speaker prosody transfer.

XTTS's extensible architecture, demonstrated by adaptations for Bangla and Arabic dialects, positions the model as a foundation for continued expansion to additional languages and dialects with minimal architectural modifications. The public availability of code and models supports reproducibility and community-driven enhancements [2406.04904][2502.05729][2406.16751].

Source: https://www.emergentmind.com/topics/xtts-model