---
title: 'WSYue-eval: Cantonese ASR & TTS Benchmark'
url: https://www.emergentmind.com/topics/wsyue-eval
type: topic
---

# WSYue-eval: Cantonese ASR & TTS Benchmark

WSYue-eval is a comprehensive evaluation benchmark suite for Cantonese speech technology, especially Automatic Speech Recognition (ASR) and Text-to-Speech (TTS). Introduced with WenetSpeech-Yue, it consists of two main components—WSYue-ASR-eval and WSYue-TTS-eval—and is designed to overcome the limitations of prior evaluation sets, which were narrow in domain, lacked diversity, and had limited linguistic phenomena. Its stated purpose is to provide a rigorous standard for Cantonese speech understanding and generation tasks, with evaluation coverage spanning short and long utterances, code-switching, diverse acoustic conditions, and linguistically challenging TTS cases [2509.03959].

## 1. Definition and position within WenetSpeech-Yue

WSYue-eval is the benchmark layer associated with WenetSpeech-Yue, a large-scale Cantonese speech corpus with multi-dimensional annotation. The benchmark is explicitly divided into two parts: WSYue-ASR-eval, a manually curated and annotated test set for assessing ASR, and WSYue-TTS-eval, an evaluation set for TTS with both “base” and “coverage” subsets. This division reflects the dual emphasis of the broader resource on speech understanding and speech generation [2509.03959].

The benchmark is described as comprehensive because it targets multiple failure modes that are central in Cantonese speech processing but often underrepresented in earlier resources. These include utterance-length variation, code-switching between Cantonese and English, routine and challenging acoustic conditions, and a range of paralinguistic attributes such as emotion, age, and gender. A plausible implication is that WSYue-eval is intended not merely as a held-out test set, but as a stress-test for robustness across realistic Hong Kong and overseas Cantonese usage conditions.

WSYue-eval samples from all ten WenetSpeech-Yue source domains: Storytelling, Entertainment, Drama, Culture, Vlog, Commentary, Education, Podcast, News, and Others. This domain span is central to its design, since representational breadth is treated as a benchmark objective rather than a by-product of corpus scale [2509.03959].

## 2. WSYue-ASR-eval: manually annotated ASR benchmark

WSYue-ASR-eval is a manually curated and annotated test set for Cantonese ASR. Each utterance receives multiple manual labels: text transcription, emotion, age, and gender. The benchmark is partitioned into two subsets to evaluate model robustness under real-world scenarios: a Short subset covering 0–10 seconds and a Long subset covering 10–30 seconds [2509.03959].

| Set | Duration | Hours |
|---|---:|---:|
| Short | 0–10 s | 9.46 |
| Long | 10–30 s | 1.97 |

The corresponding utterance counts are 2,861 for Short and 838 for Long. The source summary also presents these values in a table labeled “WSYue-ASR-eval Subsets,” where the third column is titled “Speakers,” although the entries given are 2861 and 838. The benchmark description therefore preserves those values as reported, without further reinterpretation [2509.03959].

The ASR evaluation design emphasizes diversity and realism. The utterances are sourced from the full variety of WenetSpeech-Yue domains. Scenarios include short and long utterances, routine and challenging acoustic conditions such as background noise and overlapping speech, code-switching, and emotion or tone intended to test expressiveness and paralinguistic features. The inclusion of many utterances with interleaved Cantonese and English is explicitly motivated as crucial in Hong Kong contexts [2509.03959].

The benchmark metadata schema is also unusually rich for an ASR test set. Example JSON records include fields such as `utt_id`, `rover_result`, `confidence`, `jyutping_confidence`, `duration`, `speaker_id`, `gender`, `age`, `sample_rate`, `DNSMOS`, `SNR`, timestamp boundaries, and domain-level `meta_info`. This suggests that WSYue-ASR-eval can support not only aggregate recognition scoring but also stratified analyses by speaker profile, recording quality, and domain.

## 3. WSYue-TTS-eval: zero-shot and generalization-oriented TTS benchmark

WSYue-TTS-eval is designed for zero-shot TTS evaluation and generalization assessment. It contains two subsets: Base and Coverage. The Base subset consists of 1,000 prompt-text pairs sampled from CommonVoice `yue`, and is intended to emulate real-world conversational text. The Coverage subset is a carefully constructed set spanning a broad range of domains and linguistically challenging phenomena [2509.03959].

The Coverage subset is built through an explicit multi-step process. Seed text curation is manual and aims to ensure diversity across domains such as news, daily life, entertainment, education, and literature. Linguistic phenomena are then explicitly included, with stated categories including code-switching, polyphonic characters, tone sandhi, numerals, named entities, emotions, discourse particles, slang, and more. Large language models are prompted to extend seeds into fluent, idiomatic Cantonese sentences, and all generated samples are reviewed by humans for accuracy and authenticity [2509.03959].

Representative examples given for the Coverage subset illustrate this design logic. “今日下午 meeting 約咗幾點？” is used as an example of code-switching in daily conversation; “四萬零五百五十五個燈泡亮起咗。” illustrates numerals; and “香港天氣晴朗，氣溫攝氏三十二度。” corresponds to a news or weather style sentence. These examples indicate that WSYue-TTS-eval is structured to probe linguistic generalization rather than only in-domain fluency.

A common misconception would be to treat the TTS benchmark as a standard sentence list for MOS-style testing alone. The benchmark description does not support that reading. It is explicitly framed as both a standard synthesis benchmark and a generalization benchmark, with the Coverage subset constructed to expose weaknesses in handling varied linguistic scenarios.

## 4. Domains, metadata, and annotation schema

WSYue-eval is characterized by broad domain coverage and multi-dimensional annotation. Across the benchmark, the stated annotation types include transcriptions, speaker metadata, emotion labels, text confidence, audio quality, timestamps, and domain and meta information. Specifically enumerated fields are: human-verified text for ASR evaluation; age, gender, and speaker ID; emotion labels; probability or confidence score for transcription reliability; DNSMOS and SNR for audio quality; start and end timestamps for character-level alignment; and contextual metadata such as program and region [2509.03959].

These annotations serve both evaluative and diagnostic functions. In the ASR setting, text confidence and acoustic quality attributes allow analyses that separate recognition difficulty due to language content from difficulty due to recording conditions. In the TTS setting, timestamps and metadata are relevant to alignment and prompt conditioning. This suggests that the benchmark is designed to support failure attribution, not only benchmark ranking.

The ten domains sampled by WSYue-eval—Storytelling, Entertainment, Drama, Culture, Vlog, Commentary, Education, Podcast, News, and Others—also matter methodologically. Because the benchmark is sourced from all WenetSpeech-Yue domains, it is not restricted to a single speech style. That property is directly tied to the benchmark’s claim of representational breadth [2509.03959].

## 5. Evaluation protocols and metrics

For ASR, the principal metric is Mixed Error Rate (MER), defined as
$$
\mathrm{MER} = \frac{S + D + I}{N}
$$
where $S$ denotes substitutions, $D$ deletions, $I$ insertions, and $N$ the total number of reference units. The unit is character-level for Chinese and word-level for English, which is described as naturally handling code-switching. The benchmark protocol evaluates each model on both WSYue-ASR-eval subsets, and the associated model-development procedure described in the source uses two-stage training: first with all mixed-confidence labels, then fine-tuning on high-confidence data for maximal accuracy [2509.03959].

For TTS, the benchmark uses several objective and subjective metrics. MER is again used, now applied to ASR-transcribed audio from the TTS system. SIM is defined as cosine similarity between speaker embedding vectors of reference and synthesized speech, using Wespeaker. UTMOSv2 is used as an objective metric for naturalness of synthetic speech. Human MOS is also included: native listeners, with $N=10$, rate intelligibility (I-MOS), similarity (S-MOS), and accent nativeness (A-MOS) on a 5-point scale, reported with 95% confidence intervals [2509.03959].

The metric suite reflects the fact that Cantonese TTS evaluation cannot be reduced to naturalness alone. MER targets content preservation, SIM targets speaker consistency, and MOS targets perceptual judgments. A plausible implication is that WSYue-TTS-eval is intended to discourage systems that optimize one axis at the expense of the others.

## 6. Relationship to earlier benchmarks and demonstrated utility

The benchmark is positioned against previous Cantonese ASR and TTS datasets such as Common Voice and MDCC, which are described as small, mono-domain, and lacking code-switching, acoustic diversity, or rich annotation; the summary further characterizes earlier datasets as smaller than 300 hours. WSYue-eval, by contrast, is described as larger and more varied than existing test sets, with rich annotations, systematic coverage of code-switching, numerals, tone sandhi, polyphonic characters, and a comprehensive TTS evaluation setup [2509.03959].

The source further describes WSYue-eval as complementary to existing resources and as setting a new standard for fair and challenging evaluation. Interpreted conservatively, this means that its contribution lies less in introducing a wholly new task and more in consolidating multiple sources of evaluation difficulty—linguistic, acoustic, and speaker-related—into one Cantonese benchmark suite.

Its utility is demonstrated through experiments reported in the WenetSpeech-Yue study. For ASR, models trained on WenetSpeech-Yue and evaluated on WSYue-ASR-eval are reported to outperform strong competitive baselines including SenseVoice, TeleASR, Paraformer, and Whisper-large-v3. The best reported model, U2pp-Conformer-LLM-Yue, reaches 4.73% MER on WSYue-ASR-eval Short and 7.91% MER on WSYue-ASR-eval Long [2509.03959].

For TTS, the reported objective results indicate that CosyVoice2-Yue achieves the lowest MER, reported as 10.33–9.49%, and the highest SIM, reported as 0.83–0.84, on both WSYue-TTS-eval base and coverage, outperforming Step-Audio-TTS-3B and Edge-TTS. The subjective results state that fine-tuned models achieve more than 4.3 out of 5 on I-MOS, S-MOS, and A-MOS across all axes. The benchmark authors interpret these results as evidence that WSYue-eval is challenging and discriminative, and that improvements on the benchmark translate reliably to real-world performance [2509.03959].

Source: https://www.emergentmind.com/topics/wsyue-eval