---
title: 'ZipVoice: Flow-Matching Zero-Shot TTS'
url: https://www.emergentmind.com/topics/zipvoice
type: topic
---

# ZipVoice: Flow-Matching Zero-Shot TTS

ZipVoice is a flow-matching-based non-autoregressive zero-shot text-to-speech system whose design emphasizes compact model size, stable speech-text alignment, and fast inference, and the name also denotes a broader family that extends the same backbone to zero-shot spoken dialogue generation and stereo dialogue synthesis [2506.13053] [2507.09318]. In the source papers, the base model is defined by a Zipformer-based conditional flow-matching decoder, a Zipformer-based text encoder, average upsampling for initial alignment, and a pre-trained Vocos vocoder; the dialogue extension, ZipVoice-Dialog, adds speaker-turn embeddings, curriculum learning from monologue pre-training to dialogue fine-tuning, and specialized stereo training strategies [2506.13053] [2507.09318].

## 1. Core definition and system organization

ZipVoice’s core decoder is a conditional flow-matching model built on the “Zipformer” encoder originally proposed for ASR [2506.13053]. Key architectural features of Zipformer include U-Net-like downsampling/upsampling stacks for multi-resolution feature processing, convolutional modules interleaved with self-attention to capture local fine-grained patterns alongside long-range dependencies, and attention-weight reuse across two self-attention blocks and one non-linear attention block per layer, reducing parameter count and compute.

In ZipVoice’s decoder, the architecture uses five downsampling stages at rates $[1\times, 2\times, 4\times, 2\times, 1\times]$ with $[2,2,4,4,4]$ Zipformer layers respectively; each layer’s hidden dimension is $512$, the feedforward dimension is $1536$, and the total decoder parameters together with the text encoder are approximately $123$ million [2506.13053]. The text encoder is a lightweight Zipformer-style encoder with $4$ layers, encoder dimension $192$, and feedforward dimension $512$, producing per-token embeddings $\hat y \in \mathbb{R}^{F \times N}$.

The family-level formulation given for ZipVoice-Dialog preserves this backbone structure. The text encoder maps the token sequence $y=(y_1,\ldots,y_N)$ to features $\hat y\in\mathbb{R}^{F\times N}$, the average upsampler expands token features to frame-level $z\in\mathbb{R}^{F\times T}$, the vector-field estimator $v_t(\cdot;\theta)$ is another Zipformer, and a pre-trained Vocos vocoder converts predicted speech features into waveforms [2507.09318]. This suggests that the dialogue model is not a separate architecture so much as a structured extension of the same non-autoregressive TTS backbone.

## 2. Conditional flow matching and alignment

ZipVoice uses Conditional Flow Matching to learn a time-dependent vector field $v_t(x_t;\theta)$ that transports a simple prior $p_0$ to the data distribution $p_1$ [2506.13053]. Under the linear interpolation schedule,
$$
x_t = (1-t)\cdot x_0 + t\cdot x_1,\qquad t\in[0,1],
$$
the underlying ODE is
$$
\frac{dx_t}{dt} = v_t(x_t;\theta).
$$

For TTS with masking and text condition $Z$, the training loss is
$$
L_{CFM\text{--}TTS}
= E_{t,x_0,x_1}\,\|[v_t(x_t,\,Z,\,(1-m)\odot x_1;\theta) - (x_1-x_0)]\odot m\|^2,
$$
where $m\in\{0,1\}^{D\times T}$ masks out the $(1-m)$ regions provided as speech context [2506.13053]. The dialogue paper presents the same objective in equivalent form, with $x_1\sim q(x_1)$ from real speech features, $x_0\sim p_0(x_0)=\mathcal{N}(0,I)$, and masked prediction restricted to the masked positions [2507.09318].

A central design choice is average upsampling-based initial speech-text alignment. Given $T$ frames of target speech features and $N$ text tokens, ZipVoice assumes uniform token duration $d=\lfloor T/N\rfloor$, repeats each text embedding $d$ times, and pads with filler embeddings if needed, yielding $Z\in\mathbb{R}^{F\times T}$ [2506.13053]. The dialogue model adopts the same principle and explicitly states that instead of explicitly providing token durations, durations are learned implicitly through average upsampling [2507.09318]. The ablation evidence is direct: removing the text encoder raises WER from $1.69$ to $2.04$, and replacing average upsampling with no alignment raises WER to $20.19$; adding ConvNeXt improves that to $15.49$ but remains far worse than $1.69$ [2506.13053].

During inference, the field is integrated by an ODE solver from $t=1\rightarrow 0$, and classifier-free guidance can be applied by mixing conditioned and unconditioned vector fields [2507.09318]. In the base ZipVoice paper, classifier-free guidance is written as
$$
\tilde v_t(x_t,c,\omega)
= (1+\omega)\cdot v_t(x_t,c) - \omega\cdot v_t(x_t,\emptyset),
$$
where $\omega\ge 0$ trades off fidelity vs. diversity [2506.13053].

## 3. Efficiency, compactness, and flow distillation

ZipVoice is framed against large-scale zero-shot TTS systems whose parameters often exceed $300$M to $1$B, with examples including E2-TTS at approximately $333$M and MaskGCT at approximately $1.05$B [2506.13053]. ZipVoice’s parameter count is approximately $123$M. The paper states that despite being approximately $3\times$ smaller, ZipVoice matches or surpasses these systems in WER, UTMOS, and achieves comparable speaker similarity [2506.13053].

A further speed-oriented component is flow distillation. A pretrained teacher model $\theta^T$ with CFG is used to build a high-quality vector field over two ODE steps:
$$
x_{t_{\text{mid}}} = \Phi(x_t,\,t,\,t_{\text{mid}};\,c,\omega;\theta^T),
$$
$$
x_{t_{\text{est}}} = \Phi(x_{t_{\text{mid}}},\,t_{\text{mid}},\,t_{\text{est}};\,c,\omega;\theta^T),
$$
where
$$
\Phi(x_s,\,s,\,t;\,\ldots)=x_s+(t-s)\cdot \tilde v_s(x_s,\,c,\omega;\theta^T).
$$
The effective teacher vector field is
$$
v^T \coloneqq \frac{x_{t_{\text{est}}}-x_t}{t_{\text{est}}-t}.
$$
A student model $\theta^S$, initialized from $\theta^T$, is then trained with
$$
L_{FD} = E_{t,x_0,x_1}\,\|[v_t(x_t,\,Z,\,(1-m)\odot x_1;\theta^S)-v^T]\odot m\|^2.
$$
The student is explicitly conditioned on $\omega$ through a Fourier embedding, so that after distillation a single model evaluation per step suffices to realize the CFG effect [2506.13053].

The reported speed differences are substantial. For a $3$ s prompt to $10$ s output, GPU-RTF and CPU-RTF are reported as follows [2506.13053]:

| Model | Params | GPU-RTF / CPU-RTF |
|---|---:|---:|
| F5-TTS (32 NFE) | 336 M | 0.2958 / 37.28 |
| ZipVoice (16 NFE) | 123 M | 0.0557 / 9.55 |
| ZipVoice-Distill (8 NFE) | 123 M | 0.0233 / 2.42 |
| ZipVoice-Distill (4 NFE) | 123 M | 0.0125 / 1.22 |

The paper states that ZipVoice-Distill with $4$ NFE is approximately $23.7\times$ faster on GPU and approximately $32.6\times$ faster on CPU versus F5-TTS [2506.13053]. In the distillation-method comparison at $NFE=4$, no distillation yields WER approximately $2.11$ and UTMOS approximately $3.84$; consistency distillation yields WER approximately $1.97$ and UTMOS approximately $3.30$; ReFlow yields WER approximately $2.56$ and UTMOS approximately $3.92$; and the paper’s flow distillation yields WER approximately $1.68$ and UTMOS approximately $4.22$ [2506.13053].

## 4. Extension to spoken dialogue: ZipVoice-Dialog

ZipVoice-Dialog is described as a non-autoregressive zero-shot spoken dialogue generation model built upon flow matching [2507.09318]. The task is more demanding than monologue TTS because realistic turn-taking and distinct speaker timbres must be maintained, while existing spoken dialogue generation models are described as auto-regressive and therefore slow and unstable at inference.

The input is a single interleaved token sequence of two speakers. Utterances are sorted by start time, merged if the same speaker speaks consecutively, and prefixed by a turn symbol `[S1]` or `[S2]` [2507.09318]. The principal architectural modification is the addition of two learnable speaker-turn embeddings $e_1,e_2\in\mathbb{R}^F$, trained from scratch. For each token $y_i$ belonging to speaker $s(i)\in\{1,2\}$,
$$
\hat y_i = \text{text-encoder}(y_i),\qquad
\tilde y_i = \hat y_i + e_{s(i)}.
$$
The paper states that this simple additive bias strongly disambiguates which voice to use [2507.09318].

For zero-shot prompting, all speech except a variable-length prefix across $0$ to $N$ turns is randomly masked out, allowing the model to continue in either voice [2507.09318]. This suggests that the prompting interface is designed to generalize beyond continuation of a single speaker and instead preserve dialogue state across alternating turns.

Training from scratch on dialogue data is reported to fail because two voices disrupt the speech-text pairing, with WER much greater than $80\%$ [2507.09318]. The solution is a two-stage curriculum. Stage 1 initializes all weights $\theta$ from a ZipVoice model pre-trained on $100$ k h of monologue data (Emilia), which is said to instill robust acoustic generation and speech-text alignment. Stage 2 fine-tunes $\theta$ on single-channel dialogue data (OpenDialog) for approximately $60$ k updates with batch size approximately $4$ k s of speech [2507.09318]. Without Stage 1, the model “sounds like speech” but is completely unintelligible, with WER approximately $84\%\rightarrow 116\%$; with the curriculum, WER drops to approximately $4\%$ and cpSIM rises above $0.56$ [2507.09318].

## 5. Stereo dialogue generation

ZipVoice-Dialog-Stereo extends the dialogue model so that each speaker is placed on a separate channel [2507.09318]. To do this, the model’s input and output feature dimensions double from $F\rightarrow 2F$.

The stereo design contains three stated techniques. The first is weight initialization: all core Zipformer weights are copied from the single-channel model, and for input/output projections, whose sizes are mismatched, the single-channel weights are duplicated into both channel-halves [2507.09318]. The second is single-channel regularization: both single-channel and stereo projection layers are kept in parallel, and training batches alternate between single-channel dialogues and two-channel data, which the paper states prevents catastrophic forgetting of the single-channel capabilities [2507.09318]. The third is a speaker-exclusive loss. At each ODE step $t$, the clean feature is predicted as
$$
\hat x_1 = x_t + (1-t)\cdot v_t(x_t).
$$
Splitting $\hat x_1$ into two channels $f^0,f^1\in\mathbb{R}^{D\times T}$, frame energies are computed as
$$
E_i^c = \frac{1}{D}\sum_{j=1}^{D} f_{i,j}^c,\qquad c\in\{0,1\}.
$$
Let $\tau$ be the median of all ground-truth frame energies across both channels:
$$
\tau = \mathrm{Quantile}(\{E_i^c\},0.5).
$$
The overlap penalty is
$$
L_{SE}
= \frac{1}{T}\sum_{i=1}^{T}\mathbf{1}[E_i^0>\tau \wedge E_i^1>\tau]\cdot(E_i^0-\tau)(E_i^1-\tau),
$$
and the final loss is
$$
L = L_{CFM\text{--}TTS} + \lambda\cdot L_{SE},
$$
with $\lambda=1$ in the paper [2507.09318].

The stereo ablation results are summarized in the source as follows: the full model yields WER $2.91\%$ on Chinese and $4.67\%$ on English, with cpSIM $0.474/0.321$; removing the speaker-exclusive loss raises WER to $3.66\%$ and lowers cpSIM to $0.468/0.314$; removing single-channel regularization raises WER to $3.03\%$ and lowers cpSIM to $0.461/0.317$; removing single-channel initialization raises WER to $3.89\%$ and lowers cpSIM to $0.457/0.319$ [2507.09318]. The paper concludes that each component distinctly contributes to intelligibility, turn-taking, and similarity in the stereo model.

## 6. OpenDialog dataset and benchmark methodology

OpenDialog is a spoken dialogue dataset curated because of what the paper describes as the lack of open-source large-scale spoken dialogue datasets [2507.09318]. Its total size is $6\,800$ hours of single-channel spoken dialogues, comprising $1\,759$ h Chinese and $5\,074$ h English.

The mining pipeline is given as a six-step process: voice activity detection to filter silence and noise; speaker diarization for preliminary speaker labels per segment; ASR transcription and speaker attribution via WhisperD, with English using open WhisperD and Chinese using fine-tuned Whisper; an LLM classifier to select “interactive dialogue” files; rule-based filtering for abnormal turns, weird tokens, and repetition; and DNSMOS P.835 scoring, dropping segments with $\mathrm{OVRL}<2.8$ [2507.09318]. The reported statistics indicate that DNSMOS is mostly between $2.8$ and $4.0$, while dialogue length and number of turns vary widely across both English and Chinese.

The benchmark defines three test sets: `test-dialog-zh`, containing $357$ real Chinese dialogues ($2.23$ h); `test-dialog-en`, containing $280$ English dialogues ($1.84$ h); and `test-dialog-en (short)`, a subset shorter than $30$ s for cpWER [2507.09318]. Objective metrics are intelligibility (WER), speaker turn-taking accuracy (cpWER), speaker similarity (cpSIM), UTMOS, and inference speed (RTF). WER is ASR transcription versus input text while ignoring `[S1]` and `[S2]`; cpWER is concatenated minimum-permutation WER across speaker assignments, and the gap cpWER minus WER indicates misassignments; cpSIM uses ECAPA-TDNN embeddings with minimum-permutation cosine; UTMOS is a neural MOS predictor of overall quality; and RTF is measured on NVIDIA H800 GPU [2507.09318]. Subjective metrics on Chinese only are CMOS, ranging from $-3$ to $+3$, and SMOS, ranging from $0$ to $5$.

## 7. Reported results, ablations, and stated limitations

The base ZipVoice paper reports, on LibriSpeech-PC test-clean at $16$ NFE, that ZipVoice with $123$M parameters achieves SIM-o $=0.668$, WER $=1.64\%$, UTMOS $=3.98$, CMOS $=+0.17$, and SMOS $=3.94$ [2506.13053]. The baseline F5-TTS at $336$M parameters and $32$ NFE is reported at SIM-o $=0.655$, WER $=1.89\%$, UTMOS $=3.89$, CMOS approximately $-0.03$, and SMOS approximately $3.76$. ZipVoice-Distill at $4$ NFE still obtains SIM-o approximately $0.657$, WER approximately $1.51\%$, and UTMOS approximately $4.05$ [2506.13053].

For dialogue generation, the benchmark compares Dia, MoonCast, and ZipVoice-Dialog [2507.09318]:

| Setting | Model | Reported metrics |
|---|---|---|
| Model size & RTF | Dia | 1.6 B params, RTF 1.66 |
| Model size & RTF | MoonCast | 2.7 B params, RTF 0.95 |
| Model size & RTF | ZipVoice-Dialog | 123 M params, RTF 0.063 (≈15× faster) |
| test-dialog-zh | MoonCast | cpSIM 0.463, WER 15.85 %, UTMOS 1.78 |
| test-dialog-zh | ZipVoice-Dialog | cpSIM 0.556, WER 3.17 %, UTMOS 2.25; CMOS 0.00; SMOS 3.86 |
| test-dialog-en | Dia | cpSIM 0.333, WER 11.80 %, UTMOS 1.87 |
| test-dialog-en | MoonCast | cpSIM 0.356, WER 23.62 %, UTMOS 2.37 |
| test-dialog-en | ZipVoice-Dialog | cpSIM 0.437, WER 3.25 %, UTMOS 3.07 |
| Short English (cpWER) | MoonCast | WER 8.41 %, cpWER 16.53 % |
| Short English (cpWER) | ZipVoice-Dialog | WER 2.79 %, cpWER 3.27 % |

The paper states that ZipVoice-Dialog uniformly outperforms both autoregressive (Dia) and hybrid AR/NAR (MoonCast) baselines in intelligibility, turn-taking, similarity, overall quality, and speed [2507.09318].

The ablation findings isolate several mechanisms. For speaker-turn control, a single `"|"` separator yields WER $5.34\%$ and cpWER $37.82\%$, two tokens `"[S1] [S2]"` yield WER $5.57\%$ and cpWER $31.34\%$, and additive speaker-turn embedding yields WER $5.07\%$ and cpWER $5.82\%$ [2507.09318]. For curriculum learning, monologue pre-training yields WER approximately $5\%$ on English and approximately $4\%$ on Chinese, whereas without it WER is approximately $116\%$ on English and $84\%$ on Chinese, and cpSIM plunges. For training-data size, OpenDialog only at $6.8$ k h yields WER $3.34\%$ on English, cpSIM $0.428$, and WER $2.86\%$ on Chinese; in-house only at $0.8$ k h yields higher WER $5.47\%$ on English and cpSIM $0.444$; combined data at $7.6$ k h yields WER $3.25\%$ on English, cpSIM $0.437$, and WER $3.17\%$ on Chinese with cpSIM $0.556$ [2507.09318].

The stated limitations of the base ZipVoice model are that the uniform-duration assumption is simplistic; further NFE reduction, such as $1$ to $2$ steps, remains challenging, especially without explicit durations; more powerful text encoders or cross-modal pretraining may further boost intelligibility; and robustness to noisy or low-resource prompts, multi-speaker mixing, and prosody control remains to be investigated [2506.13053]. A plausible implication is that the dialogue extension inherits both the efficiency advantages and some of the alignment-related constraints of the underlying average-upsampling flow-matching formulation.

Source: https://www.emergentmind.com/topics/zipvoice