WenetSpeech-Pipe: Cantonese Speech Corpus Pipeline
- WenetSpeech-Pipe is a modular, Cantonese-aware pipeline that converts raw audio into a large-scale corpus with multi-dimensional annotations ideal for ASR and TTS.
- The six-module architecture integrates audio collection, speaker attributes, quality scoring, multi-ASR, text normalization, and output voting to enhance transcription accuracy.
- It leverages detailed metadata including speaker identity, acoustic quality, and confidence scores to support robust training and evaluation of speech applications.
WenetSpeech-Pipe is an integrated pipeline for building large-scale speech corpus with multi-dimensional annotation tailored for speech understanding and generation. It comprises six modules—Audio Collection, Speaker Attributes Annotation, Speech Quality Annotation, Automatic Speech Recognition, Text Postprocessing and Recognizer Output Voting—and serves as the data engineering backbone used to build WenetSpeech-Yue, a 21,800-hour Cantonese speech corpus with annotations including ASR transcription, text confidence, speaker identity, age, gender, speech quality scores, and additional metadata (Li et al., 4 Sep 2025).
1. Definition and system role
WenetSpeech-Pipe is presented as a modular, Cantonese-aware pipeline that transforms raw, in-the-wild recordings into a structured corpus that enables state-of-the-art Cantonese ASR and TTS. Its stated purpose is to address scale and cost, quality and reliability, rich, multi-dimensional labels, and TTS suitability in large-scale Cantonese data construction (Li et al., 4 Sep 2025).
The pipeline is designed for settings in which existing Cantonese corpora are tiny, often less than 200 hours, and rely heavily on manual transcription and labeling. In response, WenetSpeech-Pipe automates collection, segmentation, transcription, quality scoring, and metadata extraction. It also addresses the fact that web audio is noisy and ASR errors are non-trivial, especially for a dialect with limited training data, by combining acoustic quality scoring, multi-system ASR, recognizer fusion, and utterance-level confidence estimation (Li et al., 4 Sep 2025).
A distinguishing property is its emphasis on multi-dimensional annotation. In addition to text–audio pairs, it adds speaker identity, age, gender, acoustic quality scores, domain labels, character-level timestamps, and text confidence or Jyutping confidence. The paper explicitly connects these annotations to ASR, multi-speaker TTS, style modeling, self-supervised learning, and evaluation. This suggests that WenetSpeech-Pipe is not merely a transcription pipeline, but a corpus-construction framework intended to support multiple downstream speech tasks (Li et al., 4 Sep 2025).
2. Six-module architecture and data flow
The pipeline is organized as a linear sequence of six modules. The stated data flow is: Raw long audios → VAD segmentation → per-segment speaker labels and attributes plus quality metrics → multi-ASR transcripts → text normalization → ROVER+LLM fusion → forced alignment → JSON metadata (Li et al., 4 Sep 2025).
| Module | Main function | Output |
|---|---|---|
| Audio Collection | Collect and segment long recordings | Short audio segments with coarse domain labels |
| Speaker Attributes Annotation | Infer speaker identity, age, gender | speaker_id, gender, age |
| Speech Quality Annotation | Score acoustic quality | [SNR](https://www.emergentmind.com/topics/electroweak-symmetry-non-restoration-snr), DNSMOS, sample_rate |
| Automatic Speech Recognition | Produce multiple hypotheses | Raw outputs from SenseVoice, Whisper, TeleASR |
| Text Postprocessing | Normalize heterogeneous ASR text | Normalized transcripts |
| Recognizer Output Voting | Fuse hypotheses and align text | rover_result, confidence, jyutping_confidence, timestamp |
The Audio Collection module gathers real-world Cantonese speech from YouTube, podcasts, TikTok, Bilibili, audiobooks, and related sources, covering Storytelling, Drama, Commentary, Vlogs, Food, Entertainment, News, Education, Music, and Others. A voice activity detection module splits long files into short clips, and the resulting utterances average 11.40 s in WenetSpeech-Yue. The output consists of short audio segments, time offsets per parent recording, and coarse domain labels later stored in meta_info.domain (Li et al., 4 Sep 2025).
The Speaker Attributes Annotation module combines speaker diarization and speaker-trait inference. Speaker diarization uses pyannote.audio 2.1 and outputs local speaker labels at the level of each long recording. Age and gender inference uses Vox-Profile and predicts age category and gender for each segment. The paper reports that the corpus is dominated by middle-aged male speakers, approximately 50.6% of total, with under-representation of female speakers across age groups (Li et al., 4 Sep 2025).
The Speech Quality Annotation module computes three quality dimensions: signal-to-noise ratio with Brouhaha, predicted Mean Opinion Score with DNSMOS P.835, and bandwidth or sampling rate. Reported distributions span from −5 dB to 80 dB for SNR, roughly 2.0–4.4 for DNSMOS, and 8 kHz to 32 kHz for stored sampling rates. For TTS subset construction, the paper keeps segments with DNSMOS and SNR dB, yielding 12,000 hours of high-quality speech (Li et al., 4 Sep 2025).
The Automatic Speech Recognition module runs three systems independently for each segment: SenseVoice, Whisper, and TeleASR. The rationale given is that these systems differ in architecture, training data volume and domain, and internal LLMs, so they have complementary error distributions. Their raw outputs are passed to subsequent normalization and fusion stages (Li et al., 4 Sep 2025).
3. Text normalization, voting, and confidence estimation
The Text Postprocessing module standardizes the outputs of different ASR systems into a unified format suitable for alignment and voting. The paper enumerates four sequential operations: punctuation and symbol removal; traditional-to-simplified conversion with OpenCC; text normalization of numbers and dates with cn2an/an2cn; and word spacing for mixed scripts with Pangu. The described motivation includes script mismatch across recognizers, non-lexical tokens such as [笑], punctuation and emoji, numerical and date formats, and Cantonese–English code-switching (Li et al., 4 Sep 2025).
These operations are explicitly tied to Cantonese-specific phenomena. Whisper outputs Cantonese mostly in traditional characters, whereas other systems may vary; OpenCC is therefore used to unify outputs in simplified Chinese. Pangu inserts spaces between Chinese characters and English words, which the paper connects to MER computation with character-level errors for Chinese and word-level errors for English. This suggests that the normalization stage is also an evaluation-conditioning stage, because it harmonizes transcripts before both voting and model training (Li et al., 4 Sep 2025).
Recognizer Output Voting is based on Recognizer Output Voting Error Reduction. The normalized texts from SenseVoice, Whisper, and TeleASR are aligned with a dynamic programming algorithm to create a confusion network. Candidate filtering by edit distance excludes a system from voting when its output is too far from the average transcription of the other two. Majority voting then selects the word or character with the highest frequency at each aligned position, producing the consensus transcription (Li et al., 4 Sep 2025).
The same module computes an utterance-level text confidence score from the average voting frequency across positions and reports thresholds such as 0.6, 0.8, and 0.9. It also extends voting to Cantonese Jyutping, yielding jyutping_confidence. After ROVER, the pipeline applies a minimal corrective pass using Qwen3-4B. The LLM takes the ROVER consensus transcription and the original three ASR outputs as context, and is instructed to make minimal necessary corrections while preserving spoken content. Forced alignment is then performed with a pre-trained acoustic model to obtain per-character start and end timestamps, stored as timestamp in the JSON metadata (Li et al., 4 Sep 2025).
The paper explicitly attributes several benefits to this module: it reduces systematic biases from any single recognizer, filters out outlier hypotheses, provides a principled way to quantify label reliability, and enhances lexical and phonetic consistency for a tonal, code-switching dialect like Cantonese. A plausible implication is that confidence scoring is not an ancillary metadata field but a central mechanism for later data partitioning and training strategy (Li et al., 4 Sep 2025).
4. Metadata design and corpus partitions
All metadata is stored in a single JSON per corpus. The paper lists basic fields such as utt_id, audio_path, and duration; textual annotations such as rover_result, confidence, and jyutping_confidence; speaker attributes including speaker_id, gender, and age; speech quality fields including SNR, DNSMOS, and sample_rate; temporal structure via character-level timestamp; and extended metadata in meta_info, including program, region, link, and domain (Li et al., 4 Sep 2025).
The annotation design combines automatic and manual procedures. Automatic components include ASR transcripts from SenseVoice, Whisper, and TeleASR; text normalization and postprocessing; ROVER and confidence scoring; Jyutping confidence; speaker diarization; age and gender prediction; acoustic quality annotation; and character timestamps from forced alignment. Manual procedures are reserved for evaluation sets, where the paper specifies multiple rounds of human transcription and labeling for WSYue-ASR-eval and human curation and validation for WSYue-TTS-eval Coverage (Li et al., 4 Sep 2025).
Confidence is also used to partition the 21,800 hours of WenetSpeech-Yue into three subsets. Strong labels are defined by confidence and amount to 6,771.43 hours; Moderate labels by and amount to 10,615.02 hours; Weak labels by and amount to 4,488.13 hours. Segments with confidence are discarded (Li et al., 4 Sep 2025).
This partition is subsequently exploited in a two-stage ASR training strategy: Stage 1 uses mixed medium- and high-confidence data with confidence , and Stage 2 fine-tunes on the high-confidence subset with confidence . The article reports that this two-stage strategy is applied after the pipeline rather than within it. That distinction is important: WenetSpeech-Pipe constructs and stratifies the corpus, while downstream training procedures consume those strata (Li et al., 4 Sep 2025).
5. Relation to the WeNet and WenetSpeech ecosystem
WenetSpeech-Pipe belongs to a broader WeNet-centered ecosystem in which corpus construction, ASR runtime design, and downstream speech generation are tightly connected. The production-oriented WeNet toolkit introduced a unified two-pass model, U2, with a shared encoder, CTC first-pass search, attention second-pass rescoring, dynamic chunk-based attention, TorchScript export, LibTorch deployment, and on-the-fly feature extraction (Yao et al., 2021). The notion of a “WenetSpeech-Pipe” in that earlier context referred to a complete ASR pipeline built around WeNet’s U2 model and system design, with streaming and non-streaming deployment through a single exported model (Yao et al., 2021).
The earlier WenetSpeech corpus for Mandarin organized large-scale web speech into strong, weak, and unlabeled partitions using a confidence score derived from a CTC-based end-to-end force alignment and label error detection pipeline. It stored all metadata in a single JSON file and explicitly linked candidate generation, validation, and confidence-based filtering to downstream ASR recipes for Kaldi, ESPnet, and WeNet (Zhang et al., 2021). WenetSpeech-Pipe generalizes that corpus-engineering pattern from Mandarin ASR data preparation to a multi-dimensional dialectal pipeline oriented to both ASR and TTS (Li et al., 4 Sep 2025).
Subsequent WenetSpeech projects extend the same pattern. WenetSpeech4TTS derived a 12,800-hour Mandarin TTS corpus from WenetSpeech by re-merging and extending segments, enhancing audio, detecting and removing multi-speaker segments, re-transcribing with Paraformer, and scoring each segment’s quality with DNSMOS P.808 (Ma et al., 2024). WenetSpeech-Wu provides an approximately 8,000-hour Wu dialect corpus with multi-dimensional annotation, task-specific quality tiers, and benchmark suites spanning ASR, AST, speaker attributes, emotion, TTS, and instruct TTS (Wang et al., 16 Jan 2026). WenetSpeech-Chuan introduces a 10,013-hour Sichuanese corpus and a Chuan-Pipeline with VAD, diarization, speaker clustering, WVMOS quality scoring, multi-ASR plus LLM fusion, multimodal punctuation, and confidence-based strong and weak label partitioning (Dai et al., 22 Sep 2025). This suggests that WenetSpeech-Pipe is best understood as a family of WenetSpeech-style data pipelines rather than a single monolithic implementation.
In a complementary direction, WEST repositions the WeNet ecosystem as a full-stack, LLM-based speech toolkit for recognition, synthesis, understanding, dialogue, and multimodal capabilities, while explicitly building on WeNet 2.0 and reusing its data packaging ideas (Zhang et al., 24 Sep 2025). For streaming ASR, “Adapting Whisper for Streaming Speech Recognition via Two-Pass Decoding” uses the WeNet toolkit and a U2 structure to attach a CTC branch to Whisper and retain Whisper’s decoder for rescoring, which parallels the earlier WeNet production emphasis on unified two-pass inference (Zhou et al., 13 Jun 2025). These developments do not redefine WenetSpeech-Pipe, but they place it within an expanding architecture in which dataset construction, streaming ASR, and speech generation are increasingly co-designed.
6. Empirical outcomes, benchmarks, and limitations
WenetSpeech-Pipe is validated through the release of WenetSpeech-Yue and WSYue-eval. The resulting corpus covers 21,800 hours across 10 domains, with an average segment length of 11.40 s, and provides a 12,000-hour TTS-ready subset selected by DNSMOS and SNR thresholds (Li et al., 4 Sep 2025). The benchmark suite includes WSYue-ASR-eval and WSYue-TTS-eval, with ASR test subsets for short and long utterances and TTS test subsets for base and coverage conditions (Li et al., 4 Sep 2025).
On WSYue-ASR-eval, the paper reports that high-confidence labels from the Recognizer Output Voting module are the main driver of performance gains. For example, Stage 1 to Stage 2 reductions on WSYue-ASR-eval include Whisper-medium-Yue from 7.27 to 5.05 on Short and from 11.19 to 8.05 on Long; U2pp-Conformer-Yue from 7.62 to 5.05 on Short and from 12.01 to 8.89 on Long; and U2pp-Conformer-LLM-Yue from 6.81 to 4.73 on Short and from 10.75 to 7.91 on Long (Li et al., 4 Sep 2025). The reported interpretation is explicit: the confidence produced by ROVER is crucial.
On WSYue-TTS-eval, the TTS-ready subset identified by WenetSpeech-Pipe is used to fine-tune Llasa-1B and CosyVoice2 for Cantonese. The paper reports objective and subjective improvements over zero-shot baselines. For example, CosyVoice2-Yue obtains Base MER 10.33%, Coverage MER 9.49%, SIM 0.821 and 0.834, and UTMOSv2 3.021, while Llasa-1B-Yue obtains Base MER 10.89%, Coverage MER 12.78%, SIM 0.762 and 0.772, and UTMOSv2 2.696. Subjective MOS scores include I-MOS 4.45 ± 0.16 for CosyVoice2-Yue and S-MOS 4.11 ± 0.37 plus A-MOS 4.34 ± 0.34 for Llasa-1B-Yue (Li et al., 4 Sep 2025).
The paper does not present a dedicated limitations section, but it identifies several issues. The age–gender distribution shows strong bias toward middle-aged male speakers, with female speakers under-represented. Coverage of sub-varieties or regional accents is not explicitly quantified. Although ROVER, candidate filtering, and LLM correction reduce transcription errors, the bulk of labels remains implicitly noisy rather than manually verified. Direct application to other scripts or languages would require adaptations because the pipeline uses tools such as OpenCC and cn2an (Li et al., 4 Sep 2025).
Future directions are implied rather than formalized. The paper points to application to other dialects and languages, improvement of speaker balance and domain coverage, and use of multi-dimensional labels for style transfer, emotional TTS, and self-supervised pretraining (Li et al., 4 Sep 2025). A plausible implication, reinforced by later WenetSpeech-Yue, WenetSpeech-Wu, and WenetSpeech-Chuan releases, is that WenetSpeech-Pipe has become a template for dialectal speech-data engineering within the WeNet ecosystem rather than a Cantonese-only artifact (Wang et al., 16 Jan 2026).
