---
title: 'Qwen3-TTS Series: Advanced Multilingual TTS'
url: https://www.emergentmind.com/topics/qwen3-tts-series
type: topic
---

# Qwen3-TTS Series: Advanced Multilingual TTS

Qwen3-TTS Series is a family of advanced multilingual, controllable, robust, and streaming text-to-speech (TTS) models designed to unify zero-shot voice cloning, description-based control, and ultra-low-latency synthesis within a single autoregressive framework. The Qwen3-TTS models leverage a dual-track language model (LM) architecture with two specialized speech tokenizers to achieve state-of-the-art performance across objective and subjective benchmarks. The series supports precise style conditioning, rapid voice cloning from mere seconds of reference speech, and high-fidelity streaming generation, all backed by large-scale multilingual training data. All primary variants, model weights, and tokenizers are released under the Apache 2.0 license, fostering broad community adoption and research [2601.15621].

## 1. Dual-Track Autoregressive LM Architecture

Qwen3-TTS employs a decoder-only Transformer backbone that interleaves two distinct tracks:
- **Text Track**: Standard Qwen token embeddings representing input text, with a special “no-text” token signifying text sequence completion.
- **Acoustic Track**: Speech tokens discretized by one of the Qwen-TTS-Tokenizers.

At each LM step $t$, the model ingests the current text token $x_t$ (or the “no-text” token) together with previously generated acoustic tokens $\{c_1, \ldots, c_{t-1}\}$, and immediately predicts the next group of acoustic tokens $c_t$. For Qwen3-TTS-25Hz, $c_t$ is a single code; for Qwen3-TTS-12Hz, $c_t$ comprises $N_Q$ residual codebook entries. In the 12Hz variant, code prediction is two-staged: a lightweight linear head predicts the first (semantic) codebook, then a Multi-Token Prediction (MTP) block, conditioned on the semantic code, predicts the $N_Q-1$ residual codebooks in parallel.

Joint training is performed with the negative log-likelihood of the true acoustic sequence given the text and any control tokens:

$$
\mathcal{L} = - \sum_{t=1}^T \log P\bigl(c_t \mid x_{1:L},\,c_{<t}\bigr) = \sum_{t=1}^T \mathrm{CE}\bigl(y_t,\;P(\cdot\mid x,c)\bigr)
$$

where $\mathrm{CE}$ is standard cross-entropy and $y_t$ is the one-hot ground-truth code(s).

## 2. Speech Tokenization: Qwen-TTS-Tokenizer-25Hz & 12Hz

Qwen3-TTS is distinguished by its use of two discrete speech tokenizers:

- **Qwen-TTS-Tokenizer-25Hz**: A single-codebook vector quantization (VQ) codec with $K=32,768$ entries (15 bits per code) at 25 frames per second (FPS). The resulting bitrate is:
  $$
  R_{25} = 15 \, \mathrm{bits/code} \times 25 \, \mathrm{codes/sec} = 375 \, \mathrm{bit/sec}
  $$
  Streaming waveform reconstruction is achieved via block-wise Flow Matching diffusion (DiT) and BigVGAN, with codes grouped in blocks of 8 (320 ms window), a 4-block receptive field in DiT (3 past, 1 lookahead), and BigVGAN providing a fixed 130 ms right-context lookahead.

- **Qwen-TTS-Tokenizer-12Hz**: Two-stream residual vector quantization (RVQ) with 1 semantic codebook (size 2,048, 11 bits) and 15 residual codebooks (also size 2,048). Total codes per frame $N_Q=16$; per-frame bits $= 16 \times 11 = 176$. At 12.5 FPS:
  $$
  R_{12} = 176 \, \mathrm{bits/frame} \times 12.5 \, \mathrm{frames/sec} = 2200 \, \mathrm{bit/sec} \, (2.2\, \mathrm{kbit/sec})
  $$
  The encoder and ConvNet decoder are purely causal; each 80 ms frame is decoded immediately, with inference amortized via emission of 4 tokens per packet (320 ms total latency).

| Variant           | Bitrate     | Tokenizer Type                     | Latency (first packet) |
|-------------------|-------------|------------------------------------|------------------------|
| 25Hz              | 375 bit/sec | Single-codebook VQ                 | Block-wise DiT, ~320ms |
| 12Hz              | 2.2 kbit/sec| 16 codebooks RVQ + causal ConvNet  | 97ms LM + 4ms tokenizer|

## 3. Voice Cloning and Description-Based Control

Qwen3-TTS provides two leading mechanisms for speaker and style control:
- **3-Second Voice Cloning**: A lightweight speaker encoder, trained jointly with the backbone LM, extracts a fixed hidden speaker embedding $s$ from a 3-second reference utterance. During fine-tuning (Stage 3), the base LM can be adapted to a target speaker by minimizing:
  $$
  \mathcal{L}_{\text{clone}} = -\sum_t \log P\bigl(c_t \mid x, s, c_{<t} \bigr) + \lambda \|s - s_{\text{ref}}\|^2
  $$
- **Description-Based Control**: Natural language prompts, using the ChatML format (e.g., "a warm elderly male voice with slow tempo"), are prepended to the text track as learned control token embeddings (e.g., 〈style:warm〉, 〈pitch:+2〉). The LM is conditioned on these tokens for fine-grained manipulation of output prosody, timbre, and style.

A plausible implication is that Qwen3-TTS’s design supports both the synthesis of entirely novel voices and highly specific speaker, style, and prosody edits based on description or small reference samples.

## 4. Multilingual Training and Data Stratification

Pretraining comprises over 5 million hours of speech data spanning 10 languages (including CommonVoice, Librispeech, and additional audiobook/news corpora). Training is organized into three stages:
- **S1—General Pretraining**: Large multilingual dataset pooled without adapters; model learns parameter sharing across languages.
- **S2—Quality Stratification**: Data scored using ASR models; high-quality utterances are upsampled, noisy data downsampled for robustness.
- **S3—Long-Context Training**: Maximum text+code input length increased from 8,192 to 32,768 tokens; long speech segments are upsampled to improve handling of extended utterances.

Balanced sampling ensures comprehensive non-English language representation without dedicated adapters.

## 5. Objective and Subjective Performance Benchmarks

Qwen3-TTS demonstrates state-of-the-art results in diverse evaluation scenarios:

- **Streaming Efficiency**: On vLLM + CUDA Graph, Qwen3-TTS-12Hz-0.6B achieves first-packet latency of 97 ms (93 ms LM + 4 ms tokenizer), and the 1.7B variant achieves 101 ms.
- **Zero-Shot Cloning** (Seed-TTS test, Word Error Rate ↓):
  - Seed-TTS baseline: 1.12 (ZH) | 2.25 (EN)
  - 25Hz-1.7B: 1.10 | 1.49
  - 12Hz-1.7B: 0.77 | 1.24
- **Multilingual Test (10 languages)**: Qwen3-TTS outperforms MiniMax-Speech and ElevenLabs on 6/10 languages for intelligibility (WER↓) and all languages for speaker similarity (SIM↑).
- **Controllable Generation (InstructTTSEval)**, Zh (12Hz-1.7B-VD):
  - APS↑ = 85.2%
  - DSD↑ = 81.1%
  - RP↑ = 65.1%
  - Target-Speaker Editing: +28% APS over GPT-4o-mini-tts (ZH)
- **Long Speech Synthesis (≥10 min), WER↓**:
  - VibeVoice: 22.6 (ZH) | 1.78 (EN)
  - 25Hz-1.7B: 1.517 | 1.225
- **Cross-Lingual Cloning (zh→ko, MixER↓)**:
  - CosyVoice3: 14.4%
  - Qwen3-TTS: 4.82%

This suggests that Qwen3-TTS excels not only in low-latency generation but also in high-fidelity voice preservation, multilingual accuracy, style transfer, and robustness for extended-duration synthesis.

## 6. Deployment, Streaming Pipeline, and Licensing

Deployment utilizes a streaming inference pipeline comprising:
- LM decoding with vLLM V0 (torch.compile + CUDA Graph)
- Token-to-waveform conversion using block-wise DiT + BigVGAN (25Hz) or a purely causal ConvNet decoder (12Hz)
- Service SLA for API: first-packet emission ≈ 100 ms, steady-state real-time factor (RTF) ≤ 0.5 at six-way concurrency.

Released artifacts (Apache 2.0 license) include:
- Qwen3-TTS models (0.6B, 1.7B sizes; both 12 Hz and 25 Hz tokenization variants)
- Both tokenizers (25Hz & 12Hz)
- Seamless integration with Qwen-Audio via a shared audio encoder

A plausible implication is that the public release of all components under a permissive license allows rapid experimentation, benchmarking, and integration in commercial and community contexts.

## 7. Significance and Synthesis

Qwen3-TTS Series constitutes a unified framework addressing multiple long-standing challenges in TTS: zero-shot voice cloning with minimal reference, fine-grained control via natural language or symbolic style tokens, robust and balanced multilingual performance, and streaming generation with extremely low latency. The dual-track LM and hybrid discrete tokenization enable a design that balances expressive capacity with practical efficiency, yielding leading results across intelligibility, similarity, and style controllability benchmarks [2601.15621].

Source: https://www.emergentmind.com/topics/qwen3-tts-series