Dialectal CER and LID Evaluation

• Dialectal CER and LID are evaluation metrics that quantify ASR transcription errors and language identification accuracy on dialects and accents.
• Benchmarks such as ML-SUPERB 2.0 reveal significant performance gaps in SOTA multilingual models when processing non-standard language varieties.
• Adaptation strategies like low-rank adaptation and data augmentation demonstrate practical improvements, boosting LID accuracy while reducing CER.

Dialectal Character Error Rate (CER) and Language Identification (LID) are the principal metrics for evaluating automatic speech recognition (ASR) and spoken language recognition systems on non-standard language varieties, including dialects and regional accents. Recent research demonstrates that state-of-the-art (SOTA) multilingual ASR and LID models—based on large-scale self-supervised or supervised pre-trained speech foundation models (SFMs)—exhibit significant performance degradation on dialectal and accented speech relative to mainstream language varieties. These findings have prompted the introduction of comprehensive benchmarks, notably ML-SUPERB 2.0, the development of regularization and adaptation strategies, and in-depth analyses of the inductive biases in current models (Wang et al., 30 May 2025, Bafna et al., 31 May 2025, Chen et al., 8 Sep 2025).

1. Definitions: Metrics for Dialectal Evaluation

Character Error Rate (CER) is defined as

$$\mathrm{CER} = \frac{\text{\#substitutions} + \text{\#insertions} + \text{\#deletions}}{\text{reference character count}} \times 100\%$$

and quantifies ASR transcription error at the character level. For dialectal evaluation, CER is computed per dialect and then averaged.

Language Identification (LID) accuracy is given by

$$\mathrm{ACC} = \frac{1}{M} \sum_{i=1}^{M} \mathbf{1}(\hat{\ell}_i = \ell_i) \times 100\%$$

where $\ell_i$ and $\hat{\ell}_i$ are the gold and predicted language labels for utterance $i$. Other metrics such as macro F1 and Equal Error Rate (EER) are used, but accuracy is the official ML-SUPERB 2.0 metric (Chen et al., 8 Sep 2025). All metrics can be stratified by dialect, forming the basis for "dialectal CER" and "dialectal LID".

2. Benchmarking: ML-SUPERB 2.0 and Experimental Protocols

ML-SUPERB 2.0 is a large-scale public benchmark designed for inclusive ASR and LID evaluation, covering over 200 languages, accents, and dialects. The dialectal evaluation splits consist of 93 labeled varieties from multiple public and hidden corpora, with each variety allocated 10 minutes of audio for testing. Systems are scored both on a "general multilingual" set and a "dialectal/accented" set (Chen et al., 8 Sep 2025).

Baseline models include seven self-supervised learning (SSL) encoders (e.g., XLSR-53, MMS-1B, XEUS, WavLM) followed by CTC-trained downstream networks, as well as supervised models such as Whisper Large v3 and OWSM 4B. For dialectal tests, each model's ability to generalize to under-represented, divergent, or low-resourced dialects is the focus (Chen et al., 8 Sep 2025).

3. Failure Modes and Analysis of Modern LID on Dialectal Speech

Empirical analysis shows that SOTA LID models suffer pronounced accuracy drops on dialectal or L2-accented speech. For ECAPA-TDNN (trained on VoxLingua-107), mean LID accuracy falls from ≈87.6% for US/UK/CA English to ≈55.8% on L2 English and from ≈93.4% to ≈61.3% for German (Bafna et al., 31 May 2025). A common error is "accent-language confusion": e.g., Dutch-accented English is misclassified as "Dutch" in 82.6% of errors, evidencing over-reliance on substrate-language phonotactics.

Detailed block-permutation tests reveal that popular LID models (ECAPA-TDNN, MMS-1B) are nearly invariant to sub-second input permutation (ΔAcc < 3% for 0.25s blocks), indicating that such models discard longer-range lexical and grammatical context in favor of short-range accentual cues. In contrast, sequence-sensitive models maintain higher error rates upon such permutations, indicating better use of global context.

4. Model Adaptation and Regularization Strategies for Dialectal Robustness

Several adaptation regimes have been studied for improving dialectal CER and LID:

• Frozen-upstream training: Only the downstream network is trained; cheapest but underfits dialects.
• Partial fine-tuning: A carefully selected block of deep SFM layers is unfrozen and jointly optimized, providing adaptive capacity for dialectal features.
• Low-Rank Adaptation (LoRA): Injects trainable low-rank matrices into self-attention projections of SFMs, achieving comparable or better performance than full fine-tuning (for Transformers) with parameter efficiency.
• Data Augmentation: Augmenting training data with additional labeled samples from sources such as Common Voice v20 is especially critical for low-resource or few-shot dialects. Adding ≈2,000 utterances per language yielded >35 pp LID improvement and 9 pp CER reduction for few-shot languages (Wang et al., 30 May 2025).
• LID-aware CTC loss (multi-task): An auxiliary CTC loss for LID is introduced at selected encoder layers alongside the ASR CTC loss. The multi-task objective is

$$\mathcal{L} = (1-\lambda)\,\mathcal{L}_{\mathrm{CTC}} + \lambda \frac{1}{|\mathcal{M}|} \sum_{\ell\in\mathcal{M}} \mathcal{L}_{\mathrm{LID}}^{(\ell)}$$

with $\lambda=0.3$ effective for regularizing representations and improving dialectal generalization (Wang et al., 30 May 2025).

5. Quantitative Results: Dialectal CER and LID Performance

The challenge baseline XEUS achieves 33.6% LID accuracy and 68.7% CER on the dialectal split. Best submissions have achieved 56.6% LID accuracy (+23 pp, 68% relative gain) and 38.5% CER (–30.2 pp, 44% relative reduction) on dialectal/accidental data, with all 93 varieties benefitting (Chen et al., 8 Sep 2025).

For adaptation strategies on ML-SUPERB 2.0 (MMS-1B, layers 25–36):

Method	LID-ACC DL	CER DL
Baseline	58.4%	32.7%
Frozen-upstream	50.6%	41.7%
Partial fine-tune	68.4%	34.5%
LoRA	64.9%	28.5%
+DataAug +LIDCTC	74.2%	31.5%

On the 20 few-shot languages, augmenting the training set and using LID-aware CTC produces a jump from 2.4% to 40.7% LID accuracy and from 24.6% to 15.1% CER (Wang et al., 30 May 2025). Submitted models that explicitly fine-tuned on authentic dialect data, used language-aware adapters, or applied pseudo-labeling reported the greatest performance gains.

6. Mitigation Strategies: Chunking and Sequence-Level Integration

Modern LID models can be made more robust to dialect and accent variation through inference-time and architectural changes:

• Input-chunking + majority vote: At inference, segment utterances into 1–2 second chunks, classify each chunk, and aggregate by majority vote. This empirically boosts LID accuracy on L2 accents (e.g., from 23.5% to 51.5% for ECAPA-TDNN on EdAcc) without retraining (Bafna et al., 31 May 2025).
• Sequence-level integration: Incorporating explicit phoneme or discrete-SSL unit sequence encoders (e.g., "phoneseqs" via phone recognizers with deep transformers) into LID or ASR models enables the system to learn global lexical and phonotactic structure. Joint training brings +9.5 pp LID accuracy on L2 English and +34.4 pp on challenging CV-L2 splits, at minor cost to in-domain performance.

For ASR, the same principles hold: chunk-based ensemble decoding and explicit long-range sequence modeling mitigate dialectal CER by preventing excessive reliance on local acoustic features and introducing phonotactic constraints.

7. Limitations, Open Problems, and Research Directions

Persistent challenges include:

• Orthographic bias: Disparities in dialectal performance are often exacerbated by mismatches in orthography, corpus coverage, and transcription standards.
• Model inductive bias: SOTA models are still predominantly "accent classifiers," lacking explicit mechanisms for capturing language- or dialect-level grammar and lexicon beyond local phones.
• Lack of fine-grained analysis: Absence of per-dialect and sociolect breakdown limits fairness/bias studies, despite improvements reported for all 93 tested varieties (Chen et al., 8 Sep 2025).

Ongoing research targets:

• Increasing dialectal inclusivity through open corpus and annotation release.
• Integrating external LLMs and hierarchical multi-task learning for finer phoneme/grapheme/LID coupling (Wang et al., 30 May 2025).
• Self-training with unlabeled dialectal data, and dynamic curriculum strategies for LID supervision.
• Expansion of benchmarks to include task-level spoken language understanding and translation on dialectal speech (Chen et al., 8 Sep 2025).

These directions aim to ensure that large-scale ASR and LID systems provide uniformly strong performance across all language varieties, further closing the dialectal robustness gap exposed by recent community challenges.