Seed-TTS Eval: Zero-Shot TTS Benchmark
- Seed-TTS Eval is a zero-shot text-to-speech benchmark that evaluates voice cloning by synthesizing target sentences conditioned on reference utterances in bilingual settings.
- It employs multiple splits (test-en, test-zh, and test-zh-hard) to assess classical TTS, adversarial stress testing, and robustness in alignment and conditioning.
- The benchmark drives methodological advances by incentivizing improvements in alignment learning, disentangled conditioning, and reward shaping across diverse metrics.
Seed-TTS Eval, also written as Seed-TTS-Eval, Seed-TTS Eval, or Seed-TTS test-en/test-zh in later papers, is a zero-shot text-to-speech benchmark derived from the Seed-TTS research line and used as a public comparison point for cross-sentence voice cloning, intelligibility, speaker similarity, and robustness. In its canonical form, a system is conditioned on a reference utterance and target text, and is evaluated on whether it can synthesize the target sentence while preserving the target speaker’s identity; later work also uses the benchmark for English-only classical TTS, adversarial or hard-text stress testing, streaming evaluation, and analyses of prosody or alignment failure modes (Anastassiou et al., 2024).
1. Origin and benchmark identity
The benchmark originates in the evaluation framework introduced alongside the Seed-TTS family. In the original Seed-TTS zero-shot in-context learning setup, each sample contains a reference utterance and a target utterance from the same speaker, and the model synthesizes the target text conditioned on the reference audio prompt, enabling direct comparison to ground-truth human speech (Anastassiou et al., 2024). That framing established Seed-TTS as a voice-cloning and zero-shot generalization benchmark rather than only a conventional single-speaker TTS test.
Subsequent papers describe the benchmark with more explicit split names. M3-TTS states that it follows the Seed-TTS evaluation protocol and uses Seed-TTS test-en with 1,088 English utterances from Common Voice and Seed-TTS test-zh with 2,020 Chinese utterances from DiDiSpeech, both at 24 kHz, in a zero-shot cross-sentence setting (Wang et al., 4 Dec 2025). LLaDA-TTS reports the same test-en and test-zh splits and additionally uses test-zh-hard, a set of 400 adversarial Chinese utterances, to stress robustness on hard text (Fan et al., 27 Mar 2026).
The benchmark nomenclature is not fully uniform across the literature. The original Seed-TTS paper describes an “objective set” with 1,000 English samples from Common Voice and 2,000 Mandarin samples from DiDiSpeech, whereas later papers following the Seed-TTS evaluation protocol report 1,088 English utterances and 2,020 Chinese utterances (Anastassiou et al., 2024). Stability-focused work introduces further subset labels such as hardcase and meta_zh, using Seed-TTS-Eval to separate hard texts from common Mandarin text (Wang et al., 24 Sep 2025). This suggests that “Seed-TTS Eval” functions as both a benchmark family and a de facto evaluation protocol whose exact subset naming can vary by paper.
2. Splits, modalities, and protocol variants
The most common split structure is bilingual. PilotTTS evaluates on the standard Seed-TTS Eval split with test-zh for Chinese and test-en for English (Li et al., 26 May 2026). LLaDA-TTS adds test-zh-hard, and DiffRO reports results on Chinese, English, and a hard subset when evaluating pronunciation-oriented post-training (Gao et al., 8 Jun 2026, Gao et al., 8 Jul 2025). Chatterbox-Flash uses Seed-TTS test-en as one of two public English zero-shot TTS benchmarks, while Raon-OpenTTS treats Seed-TTS-Eval as an external benchmark for open-weight TTS comparison (Seo et al., 29 May 2026, Kim et al., 20 May 2026).
Although the standard use case is zero-shot reference-conditioned synthesis, later work adapts the benchmark to other conditions. Bagpiper-TTS uses Seed-TTS-Eval (En) as a classical TTS benchmark under a plain voice setting to test whether a natural-language-guided speech synthesis system still performs standard text-to-speech well when the input is a natural-language prompt rather than a rigid slot-based format (Tian et al., 22 Jun 2026). Because Bagpiper-TTS does not accept audio prompts/reference speaker clips, the paper explicitly states that speaker similarity is not evaluated on this benchmark.
Other papers use Seed-TTS-Eval to probe narrower failure modes. The attention-guidance study on CosyVoice2 uses Seed-TTS-Eval hardcase and meta_zh to evaluate stability hallucinations, defined as repetitive, endless, or omitted speech, especially on long or difficult inputs (Wang et al., 24 Sep 2025). The classifier-free-guidance reweighting paper evaluates on SEED-EN and SEED-ZH to study the trade-off between text correctness and speaker similarity in flow-matching zero-shot TTS (Shi et al., 24 Jun 2026). In that sense, Seed-TTS Eval has become a stress test for alignment, conditioning, and robustness, not merely a leaderboard for aggregate quality.
3. Metrics and what they operationalize
The benchmark is dominated by automatic intelligibility and similarity metrics. For English, papers typically report WER; for Chinese, many later papers report CER, though some papers describe Chinese results as WER-like recognition error under Paraformer-zh. Raon-OpenTTS gives the standard WER definition,
with , , and denoting substitutions, deletions, and insertions, and the number of reference words (Kim et al., 20 May 2026). PilotTTS computes CER on Chinese with Paraformer-zh and WER on English with Whisper (Li et al., 26 May 2026). LLaDA-TTS likewise uses Paraformer for Chinese and Whisper-large-v3 for English (Fan et al., 27 Mar 2026).
Speaker fidelity is usually reported as SIM, SS, or SIM-o, generally as cosine similarity between speaker embeddings. PilotTTS reports Speaker Similarity (SIM) as cosine similarity between speaker embeddings (Li et al., 26 May 2026). M3-TTS uses SIM-o, defined as cosine similarity between WavLM-based ECAPA-TDNN embeddings extracted from the prompt and synthesized speech (Wang et al., 4 Dec 2025). LLaDA-TTS reports Speaker similarity (SS) using WavLM-large cosine similarity (Fan et al., 27 Mar 2026). Chatterbox-Flash also reports SIM-o, and its human evaluation adds SMOS, a subjective similarity score to the reference (Seo et al., 29 May 2026).
Several papers add naturalness or quality metrics beyond WER/CER and SIM. M3-TTS reports UTMOS as an objective naturalness estimate and NMOS and QMOS as human subjective ratings (Wang et al., 4 Dec 2025). Chatterbox-Flash reports UTMOS and human NMOS on Seed-TTS test-en (Seo et al., 29 May 2026). Raon-OpenTTS uses SMOS and CMOS in its broader evaluation framework and reports subjective results as mean score + 95% confidence interval (Kim et al., 20 May 2026).
The protocol is therefore metric-rich but not perfectly standardized. Some papers report only intelligibility, some include similarity, and some add subjective ratings or hard-set stress tests. Bagpiper-TTS is a particularly clear example: on Seed-TTS-Eval it reports only WER in an English plain-voice setting and explicitly excludes speaker similarity because there is no reference audio conditioning (Tian et al., 22 Jun 2026).
4. Representative reported results
The benchmark has been used to compare autoregressive, non-autoregressive, diffusion, masked-diffusion, continuous-autoregressive, and natural-language-guided systems. Reported results are strong across many model families, but they are not always directly commensurate because papers differ in language split, subset, and whether speaker similarity is included.
| System | Reported Seed-TTS result | Context |
|---|---|---|
| PilotTTS | test-zh: CER 0.87%, SIM 0.862; test-en: WER 1.50%, SIM 0.815 | Standard zero-shot split |
| E2E-TTS-Stage3 | test-zh CER 0.78; test-en WER 1.56; test-hard 6.61 | End-to-end discrete-token TTS |
| Bagpiper-TTS | Seed-TTS-Eval (En) WER 1.7% | Plain voice; no speaker similarity |
| Raon-OpenTTS-1B | WER 1.78%, SIM 0.749 | Ranked second on WER and first on SIM among recent open-weight baselines |
| M3-TTS-VAE | English WER 1.36%, Chinese WER 1.31% | NAR MM-DiT alignment |
| LLaDA-TTS (64 steps) | test-zh CER 0.98%; test-en WER 1.96%; test-zh-hard CER 7.04% | Masked diffusion |
| SemaVoice | English WER 1.71%; Chinese CER 1.18%; Hard CER 8.09% | Continuous autoregressive |
| Chatterbox-Flash | test-en SIM-o 0.704, WER 1.96, UTMOS 4.09 | Native streaming block diffusion |
PilotTTS presents one of the strongest bilingual zero-shot reports, claiming the lowest WER of 1.50% on test-en, the highest speaker similarity on both test sets, and near-best Chinese CER (Li et al., 26 May 2026). The end-to-end discrete-token training paper reports 0.78 on test-zh and 1.56 on test-en, describing them as a new SOTA with a 0.6B-parameter LLM and 0.5B-parameter FM model (Gao et al., 8 Jun 2026). M3-TTS reports 1.36% English and 1.31% Chinese word error rates for its VAE variant, positioning the model as a state-of-the-art NAR system on Seed-TTS (Wang et al., 4 Dec 2025).
English-only evaluations show similar compression toward very low WER. Bagpiper-TTS reports 1.7% WER on Seed-TTS-Eval (En) despite using natural-language prompts rather than a rigid TTS input format (Tian et al., 22 Jun 2026). Raon-OpenTTS-1B reports WER = 1.78% and SIM = 0.749, which the paper states ranks second on WER and first on SIM among recent open-weight TTS baselines (Kim et al., 20 May 2026). Chatterbox-Flash reports WER 1.96 and SIM-o 0.704 on test-en while emphasizing native streaming support rather than top-line WER alone (Seo et al., 29 May 2026).
A plausible implication is that direct leaderboard-style ranking across all reports is unsafe unless the compared papers share the same split, language, reference-conditioning regime, and metric set. The literature itself repeatedly mixes English-only, bilingual, hard-subset, and plain-voice settings.
5. What Seed-TTS Eval has driven methodologically
Because Seed-TTS Eval measures both recognition accuracy and, in many settings, speaker preservation, it has encouraged methods that explicitly address alignment and conditioning disentanglement. PilotTTS attributes its gains to Q-Former-based conditioning, cross-sample paired training, and a frozen CAMPPlus speaker embedding path, arguing that the dual-pathway design helps decouple speaker identity from speaking style while preserving content accuracy (Li et al., 26 May 2026). SemaVoice frames its improvements around SFM-guided alignment and a patch-wise diffusion head, explicitly targeting the mismatch between semantic-prosodic modeling and reconstruction-driven continuous representations (Wang et al., 16 May 2026).
Alignment-centric work uses the benchmark even more directly. The attention-guidance paper for CosyVoice2 introduces Optimal Alignment Score (OAS), reports a correlation coefficient of 0.638 between OAS and WER on 400 hard text sentences from Seed-TTS-Eval hardcase, and shows that OAS regularization and attention-guided training reduce WER on hard texts without reducing SIM or UTMOS (Wang et al., 24 Sep 2025). The classifier-free-guidance study decomposes the guidance field into text, speaker, and joint residuals and evaluates the resulting trade-off on SEED-EN and SEED-ZH, arguing that the joint residual is useful for balancing speaker similarity and text correctness (Shi et al., 24 Jun 2026).
The benchmark has also been used to justify holistic training strategies. The end-to-end discrete-token paper argues that separately training the tokenizer, LLM, FM model, and reward model creates downstream mismatch, and it uses stage-wise Seed-TTS gains—1.16 / 2.09 in Stage 1, 0.86 / 1.72 in Stage 2, and 0.78 / 1.56 in Stage 3 for zh/en—to support its E2E optimization claim (Gao et al., 8 Jun 2026). DiffRO uses Seed-TTS-Eval to show that differentiable reward optimization directly on codec tokens can improve pronunciation accuracy beyond SFT and DPO, reporting, for example, zh 1.50 → 1.09, en 4.26 → 2.57, and hard 7.90 → 5.83 for SFT plus DiffRO-ASR (Gao et al., 8 Jul 2025).
This suggests that Seed-TTS Eval has become a methodological proving ground for alignment learning, reward shaping, disentangled conditioning, and training/inference mismatch reduction.
6. Limits of the benchmark and complementary evaluations
The strongest recurring limitation is that very low WER or CER does not fully characterize speech quality. The original Seed-TTS paper explicitly notes that lower WER does not necessarily imply better human preference and warns that a model rewarded for lower WER may produce slower, clearer, more standardized speech, which can improve ASR metrics while hurting naturalness (Anastassiou et al., 2024). Voxtral TTS and Chatterbox-Flash both report settings in which automatic metrics and human judgments do not align perfectly, especially for naturalness, expressivity, and speaker similarity (Liu et al., 26 Mar 2026, Seo et al., 29 May 2026).
A second limitation is that Seed-TTS-style metrics underrepresent prosody diversity. ProsodyEval argues that a system can sound “correct” while still being bland or over-smoothed, and introduces DS-WED because conventional acoustic metrics correlate poorly with human judgments of prosodic variation. On its human-annotated dataset, DS-WED vs PMOS achieves , compared with MCD at and log RMSE at , and the paper applies the metric to Seed-TTS test-en to show that generative paradigm, duration perturbation, and DPO materially affect prosody diversity (Yang et al., 24 Sep 2025).
A third limitation is that WER and SIM do not measure instruction adherence. InstructTTSEval was introduced precisely because existing metrics tell whether speech is intelligible, high-quality, or speaker-similar, but do not tell whether a model can follow complex natural-language style instructions. It evaluates Acoustic-Parameter Specification, Descriptive-Style Directive, and Role-Play, each with 1,000 test cases per task per language, using Gemini as an automatic judge (Huang et al., 19 Jun 2025). TTScore makes a related point from a different angle by proposing TTScore-int and TTScore-pro as targeted, reference-free metrics for intelligibility and prosody beyond WER and pitch-only measures (Ulgen et al., 24 Sep 2025).
Finally, protocol heterogeneity affects interpretation. Some papers omit speaker similarity because they do not use a reference prompt; some compare on English only; some include hard subsets; and some rely on baseline numbers taken from other papers. A plausible implication is that Seed-TTS Eval is best understood as a central benchmark family for zero-shot TTS rather than a single immutable leaderboard. Its importance lies not only in the absolute scores it produces, but also in the way it has anchored comparisons across voice cloning, robustness, alignment, and controllability in contemporary speech synthesis research.
