WenetSpeech-Yue: Cantonese Speech Corpus

• WenetSpeech-Yue is a large-scale Cantonese speech corpus with rich multi-dimensional annotations (e.g., speaker attributes, quality scores) designed to boost ASR and TTS performance.
• It is constructed via a fully automated six-stage WenetSpeech-Pipe that segments, annotates, and filters 21,800 hours of Cantonese speech across diverse domains.
• The included WSYue-eval benchmark and detailed metadata enable robust evaluation and advanced research in speech recognition, synthesis, and dialect-specific applications.

Searching arXiv for the specified paper and closely related corpus work to ground the article. Using the arXiv search interface to verify bibliographic details for (Li et al., 4 Sep 2025). WenetSpeech-Yue is a large-scale Cantonese speech corpus with multi-dimensional annotation for automatic speech recognition (ASR) and text-to-speech (TTS), released together with the WSYue-eval benchmark. It is presented as the first large-scale Cantonese speech corpus with multi-dimensional annotation for ASR and TTS, and is designed to address the limited annotated resources that have hindered progress in Yue Chinese despite the language being spoken by approximately 84.9 million native speakers worldwide. The corpus is constructed with WenetSpeech-Pipe, an integrated six-stage pipeline, and covers 21,800 hours across 10 domains, with annotations including ASR transcription, text confidence, speaker identity, age, gender, speech quality scores, character-level timestamps, domain tags, and reference to the original program or link (Li et al., 4 Sep 2025).

1. Problem setting and design objective

The stated motivation for WenetSpeech-Yue is the mismatch between the scale of Cantonese usage and the scarcity of annotated resources for core speech tasks. In the formulation of the dataset paper, ASR and TTS are regarded as the most established and fundamental tasks in speech understanding and generation, yet Cantonese has had limited annotated resources and consequently suboptimal ASR and TTS performance.

WenetSpeech-Yue is explicitly framed as a corpus for both speech understanding and speech generation rather than for ASR alone. That framing is reflected in the annotation design: each utterance is not only transcribed, but also associated with confidence measures, speaker attributes, signal-quality descriptors, timing information, and source metadata. The resulting object is therefore not merely a collection of waveform–text pairs, but a multi-label corpus intended to support training, filtering, benchmarking, and analysis across several downstream settings.

The companion benchmark, WSYue-eval, extends that design objective by separating evaluation into ASR and TTS components. The ASR side targets short and long utterances, code-switching, and diverse acoustic conditions, while the TTS side distinguishes standard testing from generalization testing. This division suggests a deliberate attempt to evaluate both canonical performance and robustness under phonological, stylistic, and acoustic variation.

2. WenetSpeech-Pipe

WenetSpeech-Pipe is described as a fully automated, six-stage pipeline that ingests in-the-wild Cantonese audio and produces utterance-level segments richly annotated for downstream ASR and TTS (Li et al., 4 Sep 2025). The six modules are as follows.

Audio Collection: The pipeline crawls long-form recordings from sources including YouTube, podcasts, TikTok, Bilibili, and audiobooks across ten domains. A Voice Activity Detection (VAD) model is then used to break each recording into short, speech-only clips with average length approximately 11.4 s.

Speaker Attributes Annotation: Speaker diarization with pyannote.audio assigns a local speaker ID to each segment. Age and gender classifiers, exemplified by Vox-Profile, tag each segment with speaker age group and gender. The age groups are CHILD, TEEN, YOUTH, MIDDLE_AGE, and OLD; the gender labels are M and F.

Speech Quality Annotation: Segment-level signal and perceptual quality indicators are computed. SNR is estimated via Brouhaha, perceptual MOS is predicted with DNSMOS using non-intrusive P.835, and effective bandwidth is detected as max frequency from the spectral content.

Automatic Speech Recognition: Each segment is decoded by three diverse Cantonese ASR systems—SenseVoice, Whisper, and TeleASR—yielding three independent text hypotheses per clip.

Text Postprocessing: The pipeline converts traditional Chinese to simplified Chinese with OpenCC, removes punctuation and special symbols, normalizes numbers and dates with cn2an, and inserts spaces between Cantonese and English tokens.

Recognizer Output Voting (ROVER + LLM corrector): The three normalized transcriptions are aligned via dynamic programming. Outlier hypotheses are filtered with an edit-distance threshold, voting is performed at each token position, and the majority word is selected. The fraction of votes is recorded as the utterance-level text confidence score. In parallel, pinyin voting produces a “jyutping confidence.” Refinement is then performed by an LLM, specifically Qwen3-4B, for minimal context-aware corrections, and final forced alignment yields character-level timestamps.

By chaining these modules, the pipeline attaches high-quality ASR transcription, confidence, speaker ID, age, gender, SNR, MOS, bandwidth, character timestamps, domain tags, and source reference to each utterance. This architecture is central to the corpus’ claim of multi-dimensional annotation.

3. Scale, composition, and metadata schema

WenetSpeech-Yue contains 21,800 hours of Cantonese speech, with average segment length 11.4 s and approximately 6.9 million utterances. The number of distinct speaker-IDs is given as $O(10\ \text{thousands})$, derived via diarization. The corpus spans ten domains: Storytelling, Entertainment, Drama, Culture, Vlog, Commentary, Education, Podcast, News, and Others.

The corpus is also partitioned by label confidence, using the text-confidence score produced by ROVER. The three splits are:

Confidence split	Criterion	Duration
Strong	conf $> 0.9$	6,771.4 h
Moderate	$0.8 <$ conf $\le 0.9$	10,615.0 h
Weak	$0.6 <$ conf $\le 0.8$	4,488.1 h

Speech quality is heterogeneous by design. DNSMOS ranges from 2.0 to 4.4 with mean approximately 3.0, SNR ranges from $-5$ dB to $80$ dB with median approximately $30$ dB, and sampling rate ranges from 8 kHz to 32 kHz. Speaker attributes are not balanced: gender is approximately 70% male and 30% female, while age is skewed toward MIDDLE_AGE at 50.6%.

Per-utterance metadata are stored in JSON fields that include:

$0.8 <$0

The combination of confidence tiers, signal-quality descriptors, and speaker metadata is structurally important. It supports data selection strategies, quality-aware filtering, and attribute-conditioned modeling without requiring external relabeling.

4. WSYue-eval

WSYue-eval is a comprehensive Cantonese benchmark with two components: WSYue-ASR-eval and WSYue-TTS-eval (Li et al., 4 Sep 2025).

WSYue-ASR-eval consists of two human-annotated subsets, each with transcript and emotion, age, and gender tags. The Short subset covers utterances from 0 to 10 s and contains 2,861 utterances totaling 9.46 h. The Long subset covers utterances from 10 to 30 s and contains 838 utterances totaling 1.97 h. The benchmark covers in-house dialogue and reading, open-source corpora including CommonVoice yue/zh-HK, MDCC, Daily_Use, and Commands, as well as Cantonese-English code-switching and various acoustic conditions.

WSYue-TTS-eval is divided into a Base subset and a Coverage subset. The Base subset contains 1,000 prompts drawn from CommonVoice yue in daily conversation style. The Coverage subset contains approximately 1,000 prompts manually curated and expanded by an LLM to span multiple domains—news, announcements, literature, poetry, services, and entertainment—and challenging phenomena including polyphonic characters, tone sandhi, numerals, named entities, code-switching, colloquialisms, and strong emotion.

The ASR and TTS components of WSYue-eval serve different evaluation roles. WSYue-ASR-eval emphasizes recognition robustness under variation in duration, source, language mixing, and acoustics. WSYue-TTS-eval explicitly separates standard evaluation from generalization testing, enabling assessment of zero-shot TTS generalization and robustness.

5. ASR models, training protocol, and reported performance

The ASR experiments distinguish systems without LLM augmentation from a system with LLM augmentation. The non-LLM models are SenseVoice-small-Yue with 234 M parameters, U2pp-Conformer-Yue with 130 M, and Whisper-medium-Yue with 769 M. The LLM-based model is U2pp-Conformer-LLM-Yue, described as encoder + Qwen3-4B with 4.2 G total.

Training follows a two-stage protocol. Stage 1 trains on all segments with confidence $> 0.6$. Stage 2 fine-tunes on high-confidence segments with confidence $> 0.9$0 only. Evaluation uses Mixed Error Rate (MER), defined as Chinese character edits plus English word edits:

$> 0.9$1

On WSYue-ASR-eval, the reported Stage 2 results are:

• SenseVoice-small-Yue: MER-Short 5.23%, MER-Long 8.63%
• U2pp-Conformer-Yue: MER-Short 5.05%, MER-Long 8.89%
• Whisper-medium-Yue: MER-Short 5.05%, MER-Long 8.05%
• U2pp-Conformer-LLM-Yue: MER-Short 4.73%, MER-Long 7.91%

U2pp-Conformer-LLM-Yue is reported as SOTA on both subsets (Li et al., 4 Sep 2025). The paper further states that all Yue-fine-tuned models outperform competitive baselines such as SenseVoice, TeleASR, Whisper-large, and FireRedASR on in-house, open-source, and WSYue-ASR-eval sets.

The training protocol and confidence-aware data partition are notable because they operationalize the corpus’ annotation scheme rather than treating confidence as passive metadata. The reported gains therefore function not only as model comparisons, but also as evidence for the practical value of confidence-tiered supervision.

6. TTS models, evaluation protocol, and metric trade-offs

The TTS experiments include pretrained zero-shot models, fine-tuned Cantonese variants, and additional baselines. The pretrained zero-shot systems are Llasa-1B and CosyVoice2. Fine-tuned models are Llasa-1B-Yue and CosyVoice2-Yue, trained on a 12,000 h high-quality subset selected by DNSMOS $> 0.9$2 and SNR $> 0.9$3 dB. Additional baselines are Step-Audio-TTS-3B, described as lightweight zero-shot, and Edge-TTS, described as commercial single-speaker.

Objective metrics are MER, computed via U2pp-Conformer-Yue; speaker similarity SIM, computed as Wespeaker embedding cosine; and UTMOSv2 with higher being better. Subjective metrics are I-MOS for intelligibility, S-MOS for speaker similarity, and A-MOS for Cantonese nativeness, all on a 5-point scale with 95% confidence intervals.

For the fine-tuned Cantonese models, the reported WSYue-TTS-eval results are:

• CosyVoice2-Yue: MER-Base 10.33%, SIM-Base 0.821, MER-Cov 9.49%, SIM-Cov 0.834, UTMOSv2 3.021, I-MOS $> 0.9$4, S-MOS $> 0.9$5, A-MOS $> 0.9$6
• Llasa-1B-Yue: MER-Base 10.89%, SIM-Base 0.762, MER-Cov 12.78%, SIM-Cov 0.772, UTMOSv2 2.696, I-MOS $> 0.9$7, S-MOS $> 0.9$8, A-MOS $> 0.9$9

The paper states that both fine-tuned models dramatically reduce MER over their zero-shot counterparts; one example given is CosyVoice2 improving from 14.38% to 10.33% on Base (Li et al., 4 Sep 2025). It also states that fine-tuning boosts perceived naturalness and speaker consistency.

A notable result is the divergence between textual accuracy and perceptual quality. Edge-TTS has MER approximately 8.3%, which is lower than the fine-tuned models’ MER, but it yields lower MOS due to mechanical prosody. This directly cautions against treating low MER as a sufficient proxy for high-quality Cantonese synthesis.

7. Research utility, implications, and stated future directions

The paper identifies two principal outcomes. First, the combination of data and pipeline yields SOTA ASR results, with all Yue-destined ASR models outperforming open-source, commercial, and LLM-based Cantonese systems across benchmarks, and with U2pp-Conformer-LLM-Yue performing best on WSYue-ASR-eval. Second, TTS fine-tuning on WenetSpeech-Yue substantially improves zero-shot synthesis quality, intelligibility, and accent nativeness.

The rich metadata are presented as enabling several classes of work. The paper explicitly highlights flexible two-stage training via confidence tiers, style- and speaker-aware modeling via diarization and age/gender labels, and high-fidelity tasks such as VC and codec research through SNR, MOS, and bandwidth tags. These are direct consequences of the annotation design rather than auxiliary observations.

The stated future directions are self-supervised pretraining on 21,800 h for multilingual or dialect-adapted LLMs, multi-task learning spanning ASR, TTS, speaker-ID, and age-ID, prosody and style transfer as well as expressive TTS via speaker embeddings and domain labels, and extension of the same pipeline to other Chinese dialects. A plausible implication is that the corpus is intended not only as a benchmark resource, but also as infrastructure for data-centric methodology in Cantonese speech modeling.

All code, metadata schemas, and benchmarks are available at https://github.com/ASLP-lab/WenetSpeech-Yue.