ZipVoice: Flow-Matching Zero-Shot TTS

• ZipVoice is a flow-matching, non-autoregressive zero-shot text-to-speech system that emphasizes a compact model size and efficient speech-text alignment.
• It employs a Zipformer-based conditional flow-matching decoder and a lightweight text encoder with average upsampling for robust initial alignment.
• Extensions such as ZipVoice-Dialog and Stereo synthesis incorporate speaker-turn embeddings and flow distillation to enable rapid, high-quality dialogue generation.

ZipVoice is a flow-matching-based non-autoregressive zero-shot text-to-speech system whose design emphasizes compact model size, stable speech-text alignment, and fast inference, and the name also denotes a broader family that extends the same backbone to zero-shot spoken dialogue generation and stereo dialogue synthesis (Zhu et al., 16 Jun 2025, Zhu et al., 12 Jul 2025). In the source papers, the base model is defined by a Zipformer-based conditional flow-matching decoder, a Zipformer-based text encoder, average upsampling for initial alignment, and a pre-trained Vocos vocoder; the dialogue extension, ZipVoice-Dialog, adds speaker-turn embeddings, curriculum learning from monologue pre-training to dialogue fine-tuning, and specialized stereo training strategies (Zhu et al., 16 Jun 2025, Zhu et al., 12 Jul 2025).

1. Core definition and system organization

ZipVoice’s core decoder is a conditional flow-matching model built on the “Zipformer” encoder originally proposed for ASR (Zhu et al., 16 Jun 2025). Key architectural features of Zipformer include U-Net-like downsampling/upsampling stacks for multi-resolution feature processing, convolutional modules interleaved with self-attention to capture local fine-grained patterns alongside long-range dependencies, and attention-weight reuse across two self-attention blocks and one non-linear attention block per layer, reducing parameter count and compute.

In ZipVoice’s decoder, the architecture uses five downsampling stages at rates $[1\times, 2\times, 4\times, 2\times, 1\times]$ with $[2,2,4,4,4]$ Zipformer layers respectively; each layer’s hidden dimension is $512$, the feedforward dimension is $1536$, and the total decoder parameters together with the text encoder are approximately $123$ million (Zhu et al., 16 Jun 2025). The text encoder is a lightweight Zipformer-style encoder with $4$ layers, encoder dimension $192$, and feedforward dimension $512$, producing per-token embeddings $\hat y \in \mathbb{R}^{F \times N}$.

The family-level formulation given for ZipVoice-Dialog preserves this backbone structure. The text encoder maps the token sequence $y=(y_1,\ldots,y_N)$ to features $[2,2,4,4,4]$0, the average upsampler expands token features to frame-level $[2,2,4,4,4]$1, the vector-field estimator $[2,2,4,4,4]$2 is another Zipformer, and a pre-trained Vocos vocoder converts predicted speech features into waveforms (Zhu et al., 12 Jul 2025). This suggests that the dialogue model is not a separate architecture so much as a structured extension of the same non-autoregressive TTS backbone.

2. Conditional flow matching and alignment

ZipVoice uses Conditional Flow Matching to learn a time-dependent vector field $[2,2,4,4,4]$3 that transports a simple prior $[2,2,4,4,4]$4 to the data distribution $[2,2,4,4,4]$5 (Zhu et al., 16 Jun 2025). Under the linear interpolation schedule,

$[2,2,4,4,4]$6

the underlying ODE is

$[2,2,4,4,4]$7

For TTS with masking and text condition $[2,2,4,4,4]$8, the training loss is

$[2,2,4,4,4]$9

where $512$0 masks out the $512$1 regions provided as speech context (Zhu et al., 16 Jun 2025). The dialogue paper presents the same objective in equivalent form, with $512$2 from real speech features, $512$3, and masked prediction restricted to the masked positions (Zhu et al., 12 Jul 2025).

A central design choice is average upsampling-based initial speech-text alignment. Given $512$4 frames of target speech features and $512$5 text tokens, ZipVoice assumes uniform token duration $512$6, repeats each text embedding $512$7 times, and pads with filler embeddings if needed, yielding $512$8 (Zhu et al., 16 Jun 2025). The dialogue model adopts the same principle and explicitly states that instead of explicitly providing token durations, durations are learned implicitly through average upsampling (Zhu et al., 12 Jul 2025). The ablation evidence is direct: removing the text encoder raises WER from $512$9 to $1536$0, and replacing average upsampling with no alignment raises WER to $1536$1; adding ConvNeXt improves that to $1536$2 but remains far worse than $1536$3 (Zhu et al., 16 Jun 2025).

During inference, the field is integrated by an ODE solver from $1536$4, and classifier-free guidance can be applied by mixing conditioned and unconditioned vector fields (Zhu et al., 12 Jul 2025). In the base ZipVoice paper, classifier-free guidance is written as

$1536$5

where $1536$6 trades off fidelity vs. diversity (Zhu et al., 16 Jun 2025).

3. Efficiency, compactness, and flow distillation

ZipVoice is framed against large-scale zero-shot TTS systems whose parameters often exceed $1536$7M to $1536$8B, with examples including E2-TTS at approximately $1536$9M and MaskGCT at approximately $123$0B (Zhu et al., 16 Jun 2025). ZipVoice’s parameter count is approximately $123$1M. The paper states that despite being approximately $123$2 smaller, ZipVoice matches or surpasses these systems in WER, UTMOS, and achieves comparable speaker similarity (Zhu et al., 16 Jun 2025).

A further speed-oriented component is flow distillation. A pretrained teacher model $123$3 with CFG is used to build a high-quality vector field over two ODE steps:

$123$4

$123$5

where

$123$6

The effective teacher vector field is

$123$7

A student model $123$8, initialized from $123$9, is then trained with

$4$0

The student is explicitly conditioned on $4$1 through a Fourier embedding, so that after distillation a single model evaluation per step suffices to realize the CFG effect (Zhu et al., 16 Jun 2025).

The reported speed differences are substantial. For a $4$2 s prompt to $4$3 s output, GPU-RTF and CPU-RTF are reported as follows (Zhu et al., 16 Jun 2025):

Model	Params	GPU-RTF / CPU-RTF
F5-TTS (32 NFE)	336 M	0.2958 / 37.28
ZipVoice (16 NFE)	123 M	0.0557 / 9.55
ZipVoice-Distill (8 NFE)	123 M	0.0233 / 2.42
ZipVoice-Distill (4 NFE)	123 M	0.0125 / 1.22

The paper states that ZipVoice-Distill with $4$4 NFE is approximately $4$5 faster on GPU and approximately $4$6 faster on CPU versus F5-TTS (Zhu et al., 16 Jun 2025). In the distillation-method comparison at $4$7, no distillation yields WER approximately $4$8 and UTMOS approximately $4$9; consistency distillation yields WER approximately $192$0 and UTMOS approximately $192$1; ReFlow yields WER approximately $192$2 and UTMOS approximately $192$3; and the paper’s flow distillation yields WER approximately $192$4 and UTMOS approximately $192$5 (Zhu et al., 16 Jun 2025).

4. Extension to spoken dialogue: ZipVoice-Dialog

ZipVoice-Dialog is described as a non-autoregressive zero-shot spoken dialogue generation model built upon flow matching (Zhu et al., 12 Jul 2025). The task is more demanding than monologue TTS because realistic turn-taking and distinct speaker timbres must be maintained, while existing spoken dialogue generation models are described as auto-regressive and therefore slow and unstable at inference.

The input is a single interleaved token sequence of two speakers. Utterances are sorted by start time, merged if the same speaker speaks consecutively, and prefixed by a turn symbol [S1] or [S2] (Zhu et al., 12 Jul 2025). The principal architectural modification is the addition of two learnable speaker-turn embeddings $192$6, trained from scratch. For each token $192$7 belonging to speaker $192$8,

$192$9

The paper states that this simple additive bias strongly disambiguates which voice to use (Zhu et al., 12 Jul 2025).

For zero-shot prompting, all speech except a variable-length prefix across $512$0 to $512$1 turns is randomly masked out, allowing the model to continue in either voice (Zhu et al., 12 Jul 2025). This suggests that the prompting interface is designed to generalize beyond continuation of a single speaker and instead preserve dialogue state across alternating turns.

Training from scratch on dialogue data is reported to fail because two voices disrupt the speech-text pairing, with WER much greater than $512$2 (Zhu et al., 12 Jul 2025). The solution is a two-stage curriculum. Stage 1 initializes all weights $512$3 from a ZipVoice model pre-trained on $512$4 k h of monologue data (Emilia), which is said to instill robust acoustic generation and speech-text alignment. Stage 2 fine-tunes $512$5 on single-channel dialogue data (OpenDialog) for approximately $512$6 k updates with batch size approximately $512$7 k s of speech (Zhu et al., 12 Jul 2025). Without Stage 1, the model “sounds like speech” but is completely unintelligible, with WER approximately $512$8; with the curriculum, WER drops to approximately $512$9 and cpSIM rises above $\hat y \in \mathbb{R}^{F \times N}$0 (Zhu et al., 12 Jul 2025).

5. Stereo dialogue generation

ZipVoice-Dialog-Stereo extends the dialogue model so that each speaker is placed on a separate channel (Zhu et al., 12 Jul 2025). To do this, the model’s input and output feature dimensions double from $\hat y \in \mathbb{R}^{F \times N}$1.

The stereo design contains three stated techniques. The first is weight initialization: all core Zipformer weights are copied from the single-channel model, and for input/output projections, whose sizes are mismatched, the single-channel weights are duplicated into both channel-halves (Zhu et al., 12 Jul 2025). The second is single-channel regularization: both single-channel and stereo projection layers are kept in parallel, and training batches alternate between single-channel dialogues and two-channel data, which the paper states prevents catastrophic forgetting of the single-channel capabilities (Zhu et al., 12 Jul 2025). The third is a speaker-exclusive loss. At each ODE step $\hat y \in \mathbb{R}^{F \times N}$2, the clean feature is predicted as

$\hat y \in \mathbb{R}^{F \times N}$3

Splitting $\hat y \in \mathbb{R}^{F \times N}$4 into two channels $\hat y \in \mathbb{R}^{F \times N}$5, frame energies are computed as

$\hat y \in \mathbb{R}^{F \times N}$6

Let $\hat y \in \mathbb{R}^{F \times N}$7 be the median of all ground-truth frame energies across both channels:

$\hat y \in \mathbb{R}^{F \times N}$8

The overlap penalty is

$\hat y \in \mathbb{R}^{F \times N}$9

and the final loss is

$y=(y_1,\ldots,y_N)$0

with $y=(y_1,\ldots,y_N)$1 in the paper (Zhu et al., 12 Jul 2025).

The stereo ablation results are summarized in the source as follows: the full model yields WER $y=(y_1,\ldots,y_N)$2 on Chinese and $y=(y_1,\ldots,y_N)$3 on English, with cpSIM $y=(y_1,\ldots,y_N)$4; removing the speaker-exclusive loss raises WER to $y=(y_1,\ldots,y_N)$5 and lowers cpSIM to $y=(y_1,\ldots,y_N)$6; removing single-channel regularization raises WER to $y=(y_1,\ldots,y_N)$7 and lowers cpSIM to $y=(y_1,\ldots,y_N)$8; removing single-channel initialization raises WER to $y=(y_1,\ldots,y_N)$9 and lowers cpSIM to $[2,2,4,4,4]$00 (Zhu et al., 12 Jul 2025). The paper concludes that each component distinctly contributes to intelligibility, turn-taking, and similarity in the stereo model.

6. OpenDialog dataset and benchmark methodology

OpenDialog is a spoken dialogue dataset curated because of what the paper describes as the lack of open-source large-scale spoken dialogue datasets (Zhu et al., 12 Jul 2025). Its total size is $[2,2,4,4,4]$01 hours of single-channel spoken dialogues, comprising $[2,2,4,4,4]$02 h Chinese and $[2,2,4,4,4]$03 h English.

The mining pipeline is given as a six-step process: voice activity detection to filter silence and noise; speaker diarization for preliminary speaker labels per segment; ASR transcription and speaker attribution via WhisperD, with English using open WhisperD and Chinese using fine-tuned Whisper; an LLM classifier to select “interactive dialogue” files; rule-based filtering for abnormal turns, weird tokens, and repetition; and DNSMOS P.835 scoring, dropping segments with $[2,2,4,4,4]$04 (Zhu et al., 12 Jul 2025). The reported statistics indicate that DNSMOS is mostly between $[2,2,4,4,4]$05 and $[2,2,4,4,4]$06, while dialogue length and number of turns vary widely across both English and Chinese.

The benchmark defines three test sets: test-dialog-zh, containing $[2,2,4,4,4]$07 real Chinese dialogues ($[2,2,4,4,4]$08 h); test-dialog-en, containing $[2,2,4,4,4]$09 English dialogues ($[2,2,4,4,4]$10 h); and test-dialog-en (short), a subset shorter than $[2,2,4,4,4]$11 s for cpWER (Zhu et al., 12 Jul 2025). Objective metrics are intelligibility (WER), speaker turn-taking accuracy (cpWER), speaker similarity (cpSIM), UTMOS, and inference speed (RTF). WER is ASR transcription versus input text while ignoring [S1] and [S2]; cpWER is concatenated minimum-permutation WER across speaker assignments, and the gap cpWER minus WER indicates misassignments; cpSIM uses ECAPA-TDNN embeddings with minimum-permutation cosine; UTMOS is a neural MOS predictor of overall quality; and RTF is measured on NVIDIA H800 GPU (Zhu et al., 12 Jul 2025). Subjective metrics on Chinese only are CMOS, ranging from $[2,2,4,4,4]$12 to $[2,2,4,4,4]$13, and SMOS, ranging from $[2,2,4,4,4]$14 to $[2,2,4,4,4]$15.

7. Reported results, ablations, and stated limitations

The base ZipVoice paper reports, on LibriSpeech-PC test-clean at $[2,2,4,4,4]$16 NFE, that ZipVoice with $[2,2,4,4,4]$17M parameters achieves SIM-o $[2,2,4,4,4]$18, WER $[2,2,4,4,4]$19, UTMOS $[2,2,4,4,4]$20, CMOS $[2,2,4,4,4]$21, and SMOS $[2,2,4,4,4]$22 (Zhu et al., 16 Jun 2025). The baseline F5-TTS at $[2,2,4,4,4]$23M parameters and $[2,2,4,4,4]$24 NFE is reported at SIM-o $[2,2,4,4,4]$25, WER $[2,2,4,4,4]$26, UTMOS $[2,2,4,4,4]$27, CMOS approximately $[2,2,4,4,4]$28, and SMOS approximately $[2,2,4,4,4]$29. ZipVoice-Distill at $[2,2,4,4,4]$30 NFE still obtains SIM-o approximately $[2,2,4,4,4]$31, WER approximately $[2,2,4,4,4]$32, and UTMOS approximately $[2,2,4,4,4]$33 (Zhu et al., 16 Jun 2025).

For dialogue generation, the benchmark compares Dia, MoonCast, and ZipVoice-Dialog (Zhu et al., 12 Jul 2025):

Setting	Model	Reported metrics
Model size & RTF	Dia	1.6 B params, RTF 1.66
Model size & RTF	MoonCast	2.7 B params, RTF 0.95
Model size & RTF	ZipVoice-Dialog	123 M params, RTF 0.063 (≈15× faster)
test-dialog-zh	MoonCast	cpSIM 0.463, WER 15.85 %, UTMOS 1.78
test-dialog-zh	ZipVoice-Dialog	cpSIM 0.556, WER 3.17 %, UTMOS 2.25; CMOS 0.00; SMOS 3.86
test-dialog-en	Dia	cpSIM 0.333, WER 11.80 %, UTMOS 1.87
test-dialog-en	MoonCast	cpSIM 0.356, WER 23.62 %, UTMOS 2.37
test-dialog-en	ZipVoice-Dialog	cpSIM 0.437, WER 3.25 %, UTMOS 3.07
Short English (cpWER)	MoonCast	WER 8.41 %, cpWER 16.53 %
Short English (cpWER)	ZipVoice-Dialog	WER 2.79 %, cpWER 3.27 %

The paper states that ZipVoice-Dialog uniformly outperforms both autoregressive (Dia) and hybrid AR/NAR (MoonCast) baselines in intelligibility, turn-taking, similarity, overall quality, and speed (Zhu et al., 12 Jul 2025).

The ablation findings isolate several mechanisms. For speaker-turn control, a single "|" separator yields WER $[2,2,4,4,4]$34 and cpWER $[2,2,4,4,4]$35, two tokens "[S1] [S2]" yield WER $[2,2,4,4,4]$36 and cpWER $[2,2,4,4,4]$37, and additive speaker-turn embedding yields WER $[2,2,4,4,4]$38 and cpWER $[2,2,4,4,4]$39 (Zhu et al., 12 Jul 2025). For curriculum learning, monologue pre-training yields WER approximately $[2,2,4,4,4]$40 on English and approximately $[2,2,4,4,4]$41 on Chinese, whereas without it WER is approximately $[2,2,4,4,4]$42 on English and $[2,2,4,4,4]$43 on Chinese, and cpSIM plunges. For training-data size, OpenDialog only at $[2,2,4,4,4]$44 k h yields WER $[2,2,4,4,4]$45 on English, cpSIM $[2,2,4,4,4]$46, and WER $[2,2,4,4,4]$47 on Chinese; in-house only at $[2,2,4,4,4]$48 k h yields higher WER $[2,2,4,4,4]$49 on English and cpSIM $[2,2,4,4,4]$50; combined data at $[2,2,4,4,4]$51 k h yields WER $[2,2,4,4,4]$52 on English, cpSIM $[2,2,4,4,4]$53, and WER $[2,2,4,4,4]$54 on Chinese with cpSIM $[2,2,4,4,4]$55 (Zhu et al., 12 Jul 2025).

The stated limitations of the base ZipVoice model are that the uniform-duration assumption is simplistic; further NFE reduction, such as $[2,2,4,4,4]$56 to $[2,2,4,4,4]$57 steps, remains challenging, especially without explicit durations; more powerful text encoders or cross-modal pretraining may further boost intelligibility; and robustness to noisy or low-resource prompts, multi-speaker mixing, and prosody control remains to be investigated (Zhu et al., 16 Jun 2025). A plausible implication is that the dialogue extension inherits both the efficiency advantages and some of the alignment-related constraints of the underlying average-upsampling flow-matching formulation.