VQTTS System: Discrete TTS Architecture

• VQTTS System is a neural TTS architecture that uses self-supervised, vector-quantized acoustic features replacing mel-spectrograms to simplify and robustify acoustic modeling.
• Its two-stage pipeline integrates a classification-based text-to-vector module and a GAN-based vocoder, achieving high MOS and low WER at low bitrates.
• Advanced features include multi-speaker, cross-lingual synthesis and explicit prosody control, enabling versatile, high-fidelity speech synthesis.

The term "VQTTS System" refers to a class of neural text-to-speech (TTS) architectures built around self-supervised, vector-quantized (VQ) acoustic features rather than conventional continuous features such as mel-spectrograms. This paradigm shift aims to improve the mapping between text and acoustic representations by discretizing the target feature space, reducing regression complexity, and enabling a robust, classification-based approach to acoustic modeling. Below, all major aspects of VQTTS are detailed on the basis of representative papers from the field, specifically (Du et al., 2022, Du et al., 2023, Guo et al., 9 Apr 2024), and related high-fidelity discrete unit TTS systems.

1. Definition and Overview

VQTTS ("Vector-Quantized Text-to-Speech System"—Editor’s term) replaces the mel-spectrogram target in neural TTS with self-supervised, vector-quantized acoustic features. It integrates a classification-based acoustic model ("txt2vec") that predicts VQ code sequences, and a vocoder ("vec2wav") that synthesizes waveforms from these discrete codes, optionally with auxiliary prosodic features.

Distinct from prior continuous-feature TTS approaches, VQTTS leverages discrete representations obtained from models such as vq-wav2vec or FunCodec. These codebooks capture rich phonetic and speaker-invariant structure, which are easier for the acoustic model to predict and more robust to modeling and alignment errors.

2. Architectural Principles

VQTTS adopts a two-stage cascade pipeline:

1. Acoustic Model ("txt2vec"):
   • Input: Text (e.g., phonemes, graphemes)
   • Encoder: Deep conformer or transformer blocks
   • Prosody Controller: Sequence model (LSTM or transformer) predicts phoneme-level prosody labels (e.g., quantized pitch, energy)
   • Duration Prediction: Follows the standard TTS protocol (FastSpeech2-style)
   • Decoder: Predicts sequence of VQ code indices; outputs are discrete
   • Loss: Cross-entropy for VQ prediction, L2 loss for duration, L1/L2 for prosody
2. Vocoder ("vec2wav"):
   • Feature Encoder: Smoothing network (conformer or convolution) to reduce discontinuities from quantized inputs
   • Input Augmentation: Auxiliary features (prosody, pitch, energy) are concatenated with VQ code embedding
   • Generator: HiFi-GAN or comparable GAN-based waveform synthesizer
   • Losses: Adversarial, mel L1, and feature matching losses

Below is a simplified schematic (based on (Du et al., 2022)):

Text (phonemes) → [Conformer encoder] → [Prosody/Duration predictor] → [Conformer decoder] → [VQ code indices] + [Prosody features] → [Conformer feature encoder] → [HiFi-GAN vocoder] → waveform

3. Discrete Acoustic Feature Preparation

The VQ features utilized in VQTTS are prepared through large-scale, self-supervised speech representation learning—typically via vq-wav2vec, FunCodec, or similar. These models convert raw speech into sequences of discrete code indices:

• vq-wav2vec: Outputs code indices from a large codebook (e.g., 21.5k entries; 10 ms frame shift)
• FunCodec: Low-bitrate codebooks (1st codebook only, e.g., 1024 entries; 40 ms frame shift)
• Auxiliary Features: 3D prosody vector (log pitch, energy, Probability of Voicing/POV) appended

The key property is that VQ features are less correlated across time/frequency than mel-spectrograms, transforming the regression-to-discrete classification and making prediction more tractable. Bitrate is often a constraint: FunCodec tokens, for instance, yield only 250 bps (Guo et al., 9 Apr 2024).

4. Training Strategies and Transfer Learning

VQTTS models are trained to minimize the cross-entropy between predicted and ground-truth code indices, combined with auxiliary losses:

• Acoustic model loss:

$$\mathcal{L}_{txt2vec} = \mathcal{L}_{PL\:lab} + \mathcal{L}_{dur} + \mathcal{L}_{VQ} + \mathcal{L}_{pros}$$

where $\mathcal{L}_{VQ}$ is cross-entropy over codebook entries, $\mathcal{L}_{pros}$ is L1/L2 for prosodic features, and $\mathcal{L}_{dur}$ is L2 on durations.

• Vocoder training (during warmup):

$$\mathcal{L}_{vec2wav} = \mathcal{L}_{HifiGAN} + \alpha \mathcal{L}_{mel}$$

$\alpha$ is annealed.

• Prosody control: Phoneme-level clustering (e.g., k-means with $K$ classes) allows explicit, diverse and interpretable prosody modeling.
• Multi-speaker, multi-lingual adaptation: Speaker and language embeddings (learned, or extracted via pretrained x-vector models) are added to the encoder output. (Du et al., 2023) applies language and speaker embeddings for cross-lingual synthesis, with articulation/timbre decoupling at model/vocoder stages.

5. Performance Metrics and Experimental Results

Performance is measured via subjective and objective metrics:

System	Feature	Bitrate (bps)	MOS ↑	WER ↓
Recording	—	—	4.86 ± 0.04	—
VQTTS (vec2wav+VQ+pros)	VQ+pros	250–729	4.71–4.79	2.11–2.77
Mel + HifiGAN	Mel	—	4.68 ± 0.04	—
VITS	—	—	4.62 ± 0.04	24.8–8.87
Tacotron2+HifiGAN	Mel	—	3.67 ± 0.05	—
FastSpeech2+HifiGAN	Mel	—	3.79 ± 0.05	—

VQTTS achieves state-of-the-art MOS and WER, outperforming mel-spectrogram baselines and non-autoregressive end-to-end models, often at much lower bitrates (Du et al., 2022, Guo et al., 9 Apr 2024).

6. Advanced Features: Cross-Lingual, Multi-Speaker, and Prosody Control

VQTTS supports advanced synthesis tasks:

• Multi-speaker, multi-lingual synthesis (Du et al., 2023):
  • Dedicated speaker/language embeddings
  • Cross-lingual synthesis via decoupling: txt2vec receives native speaker embedding for correct articulation; vec2wav receives target speaker embedding for timbre control.
• Prosody and expressiveness:
  • Discrete prosody labels enable explicit pitch and energy variation, with beam search decoding for diverse outputs (Du et al., 2022).
• Low-resource regimes:
  • VQTTS is robust to small training datasets and maintains high naturalness/intelligibility (Guo et al., 9 Apr 2024). Select tokenization strategies (e.g., FunCodec) further enhance stability.

7. Significance and Limitations

VQTTS systems offer significant advances in TTS by alleviating the regression bottleneck of mel-spectrogram prediction, improving naturalness, robustness, and generalization—especially for rare and out-of-vocabulary input, multilingual transfer, and resource-constrained deployment. Discrete token approaches outperform traditional continuous representations for both prediction tractability and quality of synthesized speech.

Potential limitations include sensitivity of speaker similarity in cross-lingual settings (Du et al., 2023), dependency on effective VQ feature extractors, and residual drops in objective quality versus ground-truth at extreme bitrates (see ablation studies, (Guo et al., 9 Apr 2024)). These issues are generally offset by tailored vocoder architectures and auxiliary embedding control.

Summary Table: Core VQTTS Components

Component	Input	Output	Key Features
txt2vec	Text (+speaker/lang embeddings)	VQ codes + prosody/duration	Classification model, discrete
vec2wav	VQ codes, prosody, speaker	Waveform	Feature encoder, GAN-based

• VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature (Du et al., 2022)
• Multi-Speaker Multi-Lingual VQTTS System for LIMMITS 2023 Challenge (Du et al., 2023)
• The X-LANCE Technical Report for Interspeech 2024 (Guo et al., 9 Apr 2024)
• vq-wav2vec: Self-supervised learning of discrete speech representations
• FunCodec: [Du et al., 2023]
• Related: VQTalker (Liu et al., 13 Dec 2024) for facial motion quantization in multimodal TTS

VQTTS represents a convergent direction in speech synthesis research—harnessing discrete, self-supervised, and low-bitrate speech representations for scalable, natural, and robust neural TTS synthesis.