WenetSpeech-Chuan-Eval: Sichuanese ASR/TTS Benchmark
- WenetSpeech-Chuan-Eval is a benchmark suite with two held-out evaluation sets (ASR and TTS) derived from 10,013 hours of Sichuanese speech.
- It provides rigorous, reproducible evaluations through manually refined transcriptions and detailed metadata for dialect-specific analysis.
- The benchmark’s easy and hard subsets enable systematic diagnosis of system performance in both automatic speech recognition and text-to-speech tasks.
Searching arXiv for the target paper and closely related benchmark context. WenetSpeech-Chuan-Eval denotes the evaluation layer released with WenetSpeech-Chuan, a 10,013-hour Sichuanese speech corpus, and consists of two held-out benchmarks: WSC-Eval-ASR for automatic speech recognition and WSC-Eval-TTS for text-to-speech. The suite is built from raw Sichuanese data processed by the Chuan-Pipeline and then professionally and manually corrected, yielding high-quality, human-verified transcriptions and rich metadata intended for rigorous, reproducible evaluation in Sichuanese dialect speech processing (Dai et al., 22 Sep 2025).
1. Definition, scope, and relation to the parent corpus
WenetSpeech-Chuan-Eval is a benchmark suite rather than a training partition. Its two components, WSC-Eval-ASR and WSC-Eval-TTS, are curated test sets derived from the broader WenetSpeech-Chuan resource, but they are not part of the Strong/Weak label training splits. Their role is to function as gold-standard, held-out evaluation benchmarks with higher manual quality control than the main corpus.
This distinction is central to the benchmark’s design. The parent corpus contains 10,013 hours of Sichuanese speech with “Strong” and “Weak” label partitions, whereas WenetSpeech-Chuan-Eval is constructed from raw Sichuanese data processed by the Chuan-Pipeline and then manually corrected. The benchmark retains similar annotation dimensions—transcription, domain labels, and paralinguistic metadata—but is explicitly configured for evaluation rather than model fitting (Dai et al., 22 Sep 2025).
A recurrent misunderstanding is to treat WenetSpeech-Chuan-Eval as a generic subset of WenetSpeech-Chuan. In the released formulation, it is instead a standardized evaluation suite with task-specific benchmark design. For ASR, the benchmark measures recognition accuracy across easy and difficult acoustic and domain conditions. For TTS, it is designed to assess synthesis accuracy, naturalness, accent quality, and speaker similarity.
2. Benchmark composition and internal structure
The suite is divided into an ASR benchmark and a TTS benchmark, each further structured to expose difficulty variation rather than to supply training or development splits.
| Component | Structure | Stated function |
|---|---|---|
| WSC-Eval-ASR | Easy and Hard subsets; total duration 9.7 hours | Standardized benchmark for Sichuanese ASR |
| WSC-Eval-TTS | Easy and Hard subsets | Benchmark for synthesis accuracy, naturalness, accent quality, and speaker similarity |
WSC-Eval-ASR has a total duration of 9.7 hours. Its Easy subset covers audiobook and reading domains and lasts 8.55 hours. Its Hard subset covers short videos, entertainment, and drama and lasts 1.15 hours. All audio samples are annotated with speaker attributes, including age, gender, and emotional state. The paper does not state the number of utterances or speakers for WSC-Eval-ASR, but the domain split implies a deliberate contrast between relatively controlled reading-style material and more acoustically variable content (Dai et al., 22 Sep 2025).
The ASR benchmark is evaluation-only. No train/dev partition is defined within WSC-Eval-ASR. Its held-out status is therefore part of the benchmark contract, and its Easy/Hard split should be understood as a difficulty stratification rather than a conventional dataset partition. The domain distribution suggests that the Easy split emphasizes cleaner, read or scripted speech, while the Hard split targets more spontaneous, noisy, and conversational material from short-form media and entertainment sources.
WSC-Eval-TTS is organized into WSC-Eval-TTS-easy and WSC-Eval-TTS-hard. The easy subset contains sentences with dialectal words across diverse domains and is intended to test basic dialectal correctness and intelligibility. The hard subset contains long sentences and LLM-generated sentences in varied styles, including tongue twisters, folk sayings, and emotional speech. The hard split is therefore a stress test for prosody, long-range coherence, and accurate dialectal accent in complex content (Dai et al., 22 Sep 2025).
For TTS evaluation, the benchmark uses prompt recordings from 10 speakers—5 male and 5 female—sourced from MagicData and internal recordings. Each speaker records 200 sentences. The paper states that this design ensures balance across gender and provides diversity in age and accent variations. For subjective evaluation, 30 speech samples are evaluated, with 20 drawn from WSC-Eval-TTS-easy and 10 from WSC-Eval-TTS-hard. The underlying test sets contain more items; the 30 samples are the subset used for human MOS evaluation.
3. Annotation design and quality control
The annotation scheme in WenetSpeech-Chuan-Eval is intentionally richer than a plain transcript-only benchmark. In WSC-Eval-ASR, orthographic transcriptions are first produced by the Chuan-Pipeline, specifically by LLM-GER combining multiple ASR outputs, and are then manually refined by professional human annotators. This process is intended to ensure high transcription accuracy and to correct residual errors from automatic processing (Dai et al., 22 Sep 2025).
Speaker metadata in WSC-Eval-ASR includes gender labels, age stage labels, and emotion labels. The age stages are children, teenager, young, middle-aged, and old. The emotion categories are happy, angry, sad, neutral, fearful, surprised, and disgusted. Domain labels—such as audiobook, reading, short videos, entertainment, and drama—are also part of the benchmark structure and are used to define the Easy and Hard splits. Although the paper explicitly discusses balance across gender, age, and accent variations for the TTS evaluation set, it does not provide numeric balance claims for ASR; the more conservative reading is that the ASR metadata supports fine-grained subgroup analysis.
In WSC-Eval-TTS, the primary annotations are text scripts and prompt audio. The easy scripts are sentences with dialectal words from diverse domains, while the hard scripts include long and LLM-generated sentences, including tongue twisters, folk sayings, and emotional speech. The prompt side is multi-speaker and dialect-accented, with the speaker pool designed to provide balanced prompts across gender and diverse age and accent variation (Dai et al., 22 Sep 2025).
The paper does not enumerate phonetic or prosodic labels for the Eval sets. There is no explicit tone-marking scheme or phoneme-level annotation in the benchmark description. Annotation is primarily at the text and speaker-attribute level. However, the broader WenetSpeech-Chuan pipeline uses LLM-GER to correct and normalize Sichuanese dialectal expressions while maintaining token length and semantics, and it also uses a multimodal punctuation prediction method for the larger corpus, combining audio-based pause durations via Kaldi forced alignment with text-based BiLSTM punctuation prediction. This is relevant context because it indicates that the evaluation sets inherit conventions from a pipeline built to preserve dialectal form while aligning punctuation with actual speech pauses.
4. Evaluation protocols, metrics, and baseline systems
ASR evaluation on WSC-Eval-ASR uses Character Error Rate, reported in percent. The paper gives the standard definition:
where is the number of substitutions, the number of deletions, the number of insertions, and the number of characters in the reference transcription. CER is reported separately for WSC-Eval-ASR Easy, WSC-Eval-ASR Hard, WSC-Eval-ASR Total, MagicData-Conversation, MagicData-Daily-Use, and the average across all ASR test sets (Dai et al., 22 Sep 2025).
The ASR protocol distinguishes systems evaluated without LLM integration from systems evaluated with LLM integration. The former category includes SenseVoice-small, Whisper-small, FireRedASR-AED, Paraformer, Paraformer-WSC-Finetune, Paraformer + internal data, and Paraformer-WSC-Finetune + internal data. The latter includes Kimi-Audio, FireRedASR-LLM, Qwen2.5-omni, Qwen2.5-omni-WSC-Finetune, Qwen2.5-omni + internal data, and Qwen2.5-omni-WSC-Finetune + internal data. Paraformer and Qwen2.5-omni are fine-tuned on the Strong Label portion of WenetSpeech-Chuan, specified as 3,714 hours with confidence greater than or equal to 0.9, and further experiments add 1,000 hours of internal high-quality dialect data (Dai et al., 22 Sep 2025).
TTS evaluation combines objective and subjective metrics. Objective intelligibility is measured by CER on synthesized speech using FireRedASR. Speaker similarity is measured as SIM, reported in percent, using WavLM-Large as the speaker embedding backbone. Subjective evaluation uses three MOS-family scales: Intelligibility-MOS (IMOS), Speaker-MOS (SMOS), and Accent-MOS (AMOS), each on a 1–5 Likert scale with higher scores indicating better performance. AMOS specifically evaluates how natural and accurate the Sichuanese accent sounds (Dai et al., 22 Sep 2025).
The TTS listening protocol uses 10 native Sichuanese raters and 10 non-expert listeners, and 30 samples are evaluated for MOS scoring. Table 4 reports CER, SIM, IMOS, SMOS, and AMOS separately for the easy and hard splits. The baseline systems are Step-Audio-TTS, CosyVoice 2.0, Qwen-TTS, CosyVoice2-WSC, and CosyVoice2-WSC-SFT. In the latter two systems, WenetSpeech-Chuan is used to adapt the CosyVoice2 LLM component to Sichuanese, and CosyVoice2-WSC-SFT is further supervised fine-tuned with 100 hours of high-quality data from two fixed speakers for comparison with the single-speaker commercial Qwen-TTS baseline.
5. Experimental results and diagnostic value
The benchmark’s empirical results are organized to expose both absolute performance and failure modes. On WSC-Eval-ASR, the Hard subset consistently produces higher CER than the Easy subset across models, confirming that the split is not cosmetic but operationally diagnostic. Among open-source systems, FireRedASR-AED achieves an average CER of 15.14% across WSC-Eval-ASR and MagicData sets. Fine-tuning on WenetSpeech-Chuan substantially improves both conventional ASR and omni-modal systems: Paraformer improves from an average CER of 16.52% to 14.58%, an 11.7% relative improvement, while Qwen2.5-omni improves from 17.69% to 15.74%, an 11.0% relative improvement. With an additional 1,000 hours of internal dialect data, Paraformer-WSC-Finetune + internal data reaches an average CER of 13.38%, described as state-of-the-art among open-source systems (Dai et al., 22 Sep 2025).
These results are diagnostically informative because they show where foundation models fail before dialect adaptation. The paper notes that omni LLMs such as Qwen2.5-omni and Kimi-Audio show higher error rates on MagicData-Conversation and WSC-Hard, indicating difficulty adapting to spontaneous dialectal speech. Fine-tuning on WenetSpeech-Chuan mitigates this domain mismatch, which in turn positions WSC-Eval-ASR as a benchmark for robustness to dialect variation and spontaneous media conditions, not only for clean recognition performance.
On WSC-Eval-TTS, the evaluation similarly distinguishes straightforward dialect synthesis from expressive and long-form synthesis. CosyVoice2-WSC, trained on WenetSpeech-Chuan, achieves on the easy split a CER of 4.28%, SIM of 72.78%, and improved IMOS, SMOS, and AMOS relative to the baseline CosyVoice2.0. On the hard split it yields a CER of 8.78%, SIM of 62.59%, and an AMOS of 3.92. CosyVoice2-WSC-SFT further improves performance: on the easy split it reaches a CER of 4.08%, SIM of 78.84%, and the highest IMOS, SMOS, and AMOS values, around 4.10–4.20; on the hard split it reaches a CER of 7.22% and still leads in AMOS (Dai et al., 22 Sep 2025).
The TTS findings support two conclusions. First, training on WenetSpeech-Chuan brings open-source dialect TTS close to the commercial Qwen-TTS baseline in intelligibility. Second, the benchmark’s harder subset reveals degradations that would be obscured by a single undifferentiated test set: long and expressive Sichuanese content increases CER and lowers SIM and MOS scores, even when basic dialectal correctness remains strong. WSC-Eval-TTS is therefore a diagnostic benchmark for prosody, accent realization, and robustness under complex sentence design.
6. Access, usage conventions, and broader significance
The project page states that the corpus, benchmarks, models, and receipts are publicly available at https://github.com/ASLP-lab/WenetSpeech-Chuan. The paper does not describe licensing specifics in its text. It does, however, make the intended use of the evaluation suite clear: WSC-Eval-ASR and WSC-Eval-TTS should be treated as test sets rather than as training material, and results should be reported with the benchmark’s prescribed split structure and metric family (Dai et al., 22 Sep 2025).
This usage model aligns WenetSpeech-Chuan-Eval with the benchmarking logic established by WenetSpeech for Mandarin, where held-out, manually transcribed evaluation sets are separated from large-scale training partitions (Zhang et al., 2021). In the Sichuanese setting, the benchmark extends that methodology from standard Mandarin to dialectal speech. The paper explicitly situates this within a broader resource gap: Sichuanese is described as a major Chinese dialect cluster with approximately 120 million speakers, while existing open resources such as MagicData Conversation at 4.53 hours and MagicData Daily-Use at 6.4 hours are much smaller, and KeSpeech covers some Southwestern Mandarin accents rather than full dialect (Dai et al., 22 Sep 2025).
The benchmark’s significance is not limited to system comparison. Because the ASR side includes gender, age, and emotion labels, and the TTS side explicitly balances prompt design across gender, age, and accent variation, the suite also supports bias analysis. The paper frames this as part of a broader AI equity agenda: mainstream systems often underperform on dialectal and accented speech, and a publicly available benchmark with high-quality evaluation sets can make those disparities measurable.
A further implication is methodological rather than directly claimed. The existing WSC-Eval-TTS protocol already combines objective metrics with IMOS, SMOS, and AMOS; a plausible extension would be to complement this with direct human-likeness evaluation frameworks such as the Audio Turing Test, which was proposed for Chinese TTS as an alternative to MOS saturation effects (Wang et al., 16 May 2025). Even without such an extension, WenetSpeech-Chuan-Eval already functions as a primary open benchmark for Sichuanese ASR and TTS: it defines held-out tasks, standardized splits, reproducible metrics, and strong open-source baselines within a resource regime that had previously been severely underspecified (Dai et al., 22 Sep 2025).
