---
title: 'Seed-TTS Eval: Zero-Shot TTS Benchmark'
url: https://www.emergentmind.com/topics/seed-tts-eval
type: topic
---

# Seed-TTS Eval: Zero-Shot TTS Benchmark

Seed-TTS Eval, also written as Seed-TTS-Eval, Seed-TTS Eval, or Seed-TTS test-en/test-zh in later papers, is a zero-shot text-to-speech benchmark derived from the Seed-TTS research line and used as a public comparison point for cross-sentence voice cloning, intelligibility, speaker similarity, and robustness. In its canonical form, a system is conditioned on a reference utterance and target text, and is evaluated on whether it can synthesize the target sentence while preserving the target speaker’s identity; later work also uses the benchmark for English-only classical TTS, adversarial or hard-text stress testing, streaming evaluation, and analyses of prosody or alignment failure modes [2406.02430].

## 1. Origin and benchmark identity

The benchmark originates in the evaluation framework introduced alongside the Seed-TTS family. In the original Seed-TTS zero-shot in-context learning setup, each sample contains a reference utterance and a target utterance from the same speaker, and the model synthesizes the target text conditioned on the reference audio prompt, enabling direct comparison to ground-truth human speech [2406.02430]. That framing established Seed-TTS as a voice-cloning and zero-shot generalization benchmark rather than only a conventional single-speaker TTS test.

Subsequent papers describe the benchmark with more explicit split names. M3-TTS states that it follows the Seed-TTS evaluation protocol and uses Seed-TTS test-en with **1,088 English utterances** from **Common Voice** and Seed-TTS test-zh with **2,020 Chinese utterances** from **DiDiSpeech**, both at **24 kHz**, in a zero-shot cross-sentence setting [2512.04720]. LLaDA-TTS reports the same test-en and test-zh splits and additionally uses **test-zh-hard**, a set of **400 adversarial Chinese utterances**, to stress robustness on hard text [2603.26364].

The benchmark nomenclature is not fully uniform across the literature. The original Seed-TTS paper describes an “objective set” with **1,000 English samples** from **Common Voice** and **2,000 Mandarin samples** from **DiDiSpeech**, whereas later papers following the Seed-TTS evaluation protocol report **1,088** English utterances and **2,020** Chinese utterances [2406.02430]. Stability-focused work introduces further subset labels such as **hardcase** and **meta_zh**, using Seed-TTS-Eval to separate hard texts from common Mandarin text [2509.19852]. This suggests that “Seed-TTS Eval” functions as both a benchmark family and a de facto evaluation protocol whose exact subset naming can vary by paper.

## 2. Splits, modalities, and protocol variants

The most common split structure is bilingual. PilotTTS evaluates on the standard Seed-TTS Eval split with **test-zh** for Chinese and **test-en** for English [2605.27258]. LLaDA-TTS adds **test-zh-hard**, and DiffRO reports results on **Chinese**, **English**, and a **hard subset** when evaluating pronunciation-oriented post-training [2606.09234; 2507.05911]. Chatterbox-Flash uses **Seed-TTS test-en** as one of two public English zero-shot TTS benchmarks, while Raon-OpenTTS treats Seed-TTS-Eval as an external benchmark for open-weight TTS comparison [2605.30748; 2605.20830].

Although the standard use case is zero-shot reference-conditioned synthesis, later work adapts the benchmark to other conditions. Bagpiper-TTS uses **Seed-TTS-Eval (En)** as a **classical TTS benchmark** under a **plain voice** setting to test whether a natural-language-guided speech synthesis system still performs standard text-to-speech well when the input is a natural-language prompt rather than a rigid slot-based format [2606.22811]. Because Bagpiper-TTS does **not accept audio prompts/reference speaker clips**, the paper explicitly states that **speaker similarity is not evaluated** on this benchmark.

Other papers use Seed-TTS-Eval to probe narrower failure modes. The attention-guidance study on CosyVoice2 uses Seed-TTS-Eval **hardcase** and **meta_zh** to evaluate **stability hallucinations**, defined as repetitive, endless, or omitted speech, especially on long or difficult inputs [2509.19852]. The classifier-free-guidance reweighting paper evaluates on **SEED-EN** and **SEED-ZH** to study the trade-off between text correctness and speaker similarity in flow-matching zero-shot TTS [2606.25672]. In that sense, Seed-TTS Eval has become a stress test for alignment, conditioning, and robustness, not merely a leaderboard for aggregate quality.

## 3. Metrics and what they operationalize

The benchmark is dominated by automatic intelligibility and similarity metrics. For English, papers typically report **WER**; for Chinese, many later papers report **CER**, though some papers describe Chinese results as WER-like recognition error under Paraformer-zh. Raon-OpenTTS gives the standard WER definition,
$$
\mathrm{WER} = \frac{S + D + I}{N},
$$
with \(S\), \(D\), and \(I\) denoting substitutions, deletions, and insertions, and \(N\) the number of reference words [2605.20830]. PilotTTS computes **CER** on Chinese with **Paraformer-zh** and **WER** on English with **Whisper** [2605.27258]. LLaDA-TTS likewise uses **Paraformer** for Chinese and **Whisper-large-v3** for English [2603.26364].

Speaker fidelity is usually reported as **SIM**, **SS**, or **SIM-o**, generally as cosine similarity between speaker embeddings. PilotTTS reports **Speaker Similarity (SIM)** as cosine similarity between speaker embeddings [2605.27258]. M3-TTS uses **SIM-o**, defined as cosine similarity between **WavLM-based ECAPA-TDNN** embeddings extracted from the prompt and synthesized speech [2512.04720]. LLaDA-TTS reports **Speaker similarity (SS)** using **WavLM-large cosine similarity** [2603.26364]. Chatterbox-Flash also reports **SIM-o**, and its human evaluation adds **SMOS**, a subjective similarity score to the reference [2605.30748].

Several papers add naturalness or quality metrics beyond WER/CER and SIM. M3-TTS reports **UTMOS** as an objective naturalness estimate and **NMOS** and **QMOS** as human subjective ratings [2512.04720]. Chatterbox-Flash reports **UTMOS** and human **NMOS** on Seed-TTS test-en [2605.30748]. Raon-OpenTTS uses **SMOS** and **CMOS** in its broader evaluation framework and reports subjective results as **mean score + 95% confidence interval** [2605.20830].

The protocol is therefore metric-rich but not perfectly standardized. Some papers report only intelligibility, some include similarity, and some add subjective ratings or hard-set stress tests. Bagpiper-TTS is a particularly clear example: on Seed-TTS-Eval it reports only **WER** in an English plain-voice setting and explicitly excludes speaker similarity because there is no reference audio conditioning [2606.22811].

## 4. Representative reported results

The benchmark has been used to compare autoregressive, non-autoregressive, diffusion, masked-diffusion, continuous-autoregressive, and natural-language-guided systems. Reported results are strong across many model families, but they are not always directly commensurate because papers differ in language split, subset, and whether speaker similarity is included.

| System | Reported Seed-TTS result | Context |
|---|---|---|
| PilotTTS | test-zh: CER **0.87%**, SIM **0.862**; test-en: WER **1.50%**, SIM **0.815** | Standard zero-shot split |
| E2E-TTS-Stage3 | test-zh CER **0.78**; test-en WER **1.56**; test-hard **6.61** | End-to-end discrete-token TTS |
| Bagpiper-TTS | Seed-TTS-Eval (En) WER **1.7%** | Plain voice; no speaker similarity |
| Raon-OpenTTS-1B | WER **1.78%**, SIM **0.749** | Ranked second on WER and first on SIM among recent open-weight baselines |
| M3-TTS-VAE | English WER **1.36%**, Chinese WER **1.31%** | NAR MM-DiT alignment |
| LLaDA-TTS (64 steps) | test-zh CER **0.98%**; test-en WER **1.96%**; test-zh-hard CER **7.04%** | Masked diffusion |
| SemaVoice | English WER **1.71%**; Chinese CER **1.18%**; Hard CER **8.09%** | Continuous autoregressive |
| Chatterbox-Flash | test-en SIM-o **0.704**, WER **1.96**, UTMOS **4.09** | Native streaming block diffusion |

PilotTTS presents one of the strongest bilingual zero-shot reports, claiming the **lowest WER of 1.50% on test-en**, the **highest speaker similarity on both test sets**, and near-best Chinese CER [2605.27258]. The end-to-end discrete-token training paper reports **0.78** on test-zh and **1.56** on test-en, describing them as a new SOTA with a **0.6B-parameter LLM** and **0.5B-parameter FM model** [2606.09234]. M3-TTS reports **1.36% English** and **1.31% Chinese** word error rates for its VAE variant, positioning the model as a state-of-the-art NAR system on Seed-TTS [2512.04720].

English-only evaluations show similar compression toward very low WER. Bagpiper-TTS reports **1.7% WER** on **Seed-TTS-Eval (En)** despite using natural-language prompts rather than a rigid TTS input format [2606.22811]. Raon-OpenTTS-1B reports **WER = 1.78%** and **SIM = 0.749**, which the paper states ranks **second on WER** and **first on SIM** among recent open-weight TTS baselines [2605.20830]. Chatterbox-Flash reports **WER 1.96** and **SIM-o 0.704** on test-en while emphasizing native streaming support rather than top-line WER alone [2605.30748].

A plausible implication is that direct leaderboard-style ranking across all reports is unsafe unless the compared papers share the same split, language, reference-conditioning regime, and metric set. The literature itself repeatedly mixes English-only, bilingual, hard-subset, and plain-voice settings.

## 5. What Seed-TTS Eval has driven methodologically

Because Seed-TTS Eval measures both recognition accuracy and, in many settings, speaker preservation, it has encouraged methods that explicitly address alignment and conditioning disentanglement. PilotTTS attributes its gains to **Q-Former-based conditioning**, **cross-sample paired training**, and a frozen **CAMPPlus** speaker embedding path, arguing that the dual-pathway design helps decouple speaker identity from speaking style while preserving content accuracy [2605.27258]. SemaVoice frames its improvements around **SFM-guided alignment** and a **patch-wise diffusion head**, explicitly targeting the mismatch between semantic-prosodic modeling and reconstruction-driven continuous representations [2605.16964].

Alignment-centric work uses the benchmark even more directly. The attention-guidance paper for CosyVoice2 introduces **Optimal Alignment Score (OAS)**, reports a **correlation coefficient of 0.638** between OAS and WER on **400 hard text sentences** from Seed-TTS-Eval hardcase, and shows that OAS regularization and attention-guided training reduce WER on hard texts without reducing SIM or UTMOS [2509.19852]. The classifier-free-guidance study decomposes the guidance field into **text**, **speaker**, and **joint residuals** and evaluates the resulting trade-off on **SEED-EN** and **SEED-ZH**, arguing that the joint residual is useful for balancing speaker similarity and text correctness [2606.25672].

The benchmark has also been used to justify holistic training strategies. The end-to-end discrete-token paper argues that separately training the tokenizer, LLM, FM model, and reward model creates downstream mismatch, and it uses stage-wise Seed-TTS gains—**1.16 / 2.09** in Stage 1, **0.86 / 1.72** in Stage 2, and **0.78 / 1.56** in Stage 3 for zh/en—to support its E2E optimization claim [2606.09234]. DiffRO uses Seed-TTS-Eval to show that differentiable reward optimization directly on codec tokens can improve pronunciation accuracy beyond SFT and DPO, reporting, for example, **zh 1.50 → 1.09**, **en 4.26 → 2.57**, and **hard 7.90 → 5.83** for SFT plus DiffRO-ASR [2507.05911].

This suggests that Seed-TTS Eval has become a methodological proving ground for alignment learning, reward shaping, disentangled conditioning, and training/inference mismatch reduction.

## 6. Limits of the benchmark and complementary evaluations

The strongest recurring limitation is that very low WER or CER does not fully characterize speech quality. The original Seed-TTS paper explicitly notes that lower WER does **not** necessarily imply better human preference and warns that a model rewarded for lower WER may produce **slower, clearer, more standardized** speech, which can improve ASR metrics while hurting naturalness [2406.02430]. Voxtral TTS and Chatterbox-Flash both report settings in which automatic metrics and human judgments do not align perfectly, especially for naturalness, expressivity, and speaker similarity [2603.25551; 2605.30748].

A second limitation is that Seed-TTS-style metrics underrepresent prosody diversity. ProsodyEval argues that a system can sound “correct” while still being bland or over-smoothed, and introduces **DS-WED** because conventional acoustic metrics correlate poorly with human judgments of prosodic variation. On its human-annotated dataset, **DS-WED vs PMOS** achieves \(\bar r = 0.77\), compared with **MCD** at \(\bar r = 0.66\) and **log \(F_0\) RMSE** at \(\bar r = 0.30\), and the paper applies the metric to **Seed-TTS test-en** to show that generative paradigm, duration perturbation, and DPO materially affect prosody diversity [2509.19928].

A third limitation is that WER and SIM do not measure instruction adherence. InstructTTSEval was introduced precisely because existing metrics tell whether speech is intelligible, high-quality, or speaker-similar, but do **not** tell whether a model can follow complex natural-language style instructions. It evaluates **Acoustic-Parameter Specification**, **Descriptive-Style Directive**, and **Role-Play**, each with **1,000 test cases per task per language**, using **Gemini** as an automatic judge [2506.16381]. TTScore makes a related point from a different angle by proposing **TTScore-int** and **TTScore-pro** as targeted, reference-free metrics for intelligibility and prosody beyond WER and pitch-only measures [2509.20485].

Finally, protocol heterogeneity affects interpretation. Some papers omit speaker similarity because they do not use a reference prompt; some compare on English only; some include hard subsets; and some rely on baseline numbers taken from other papers. A plausible implication is that Seed-TTS Eval is best understood as a central benchmark family for zero-shot TTS rather than a single immutable leaderboard. Its importance lies not only in the absolute scores it produces, but also in the way it has anchored comparisons across voice cloning, robustness, alignment, and controllability in contemporary speech synthesis research.

Source: https://www.emergentmind.com/topics/seed-tts-eval