Benchmarking Multilingual Speech Models on Pashto: Zero-Shot ASR, Script Failure, and Cross-Domain Evaluation

Abstract: Pashto is spoken by approximately 60--80 million people but has no published benchmarks for multilingual automatic speech recognition (ASR) on any shared public test set. This paper reports the first reproducible multi-model evaluation on public Pashto data, covering zero-shot ASR, script-level failure, and cross-domain evaluation of fine-tuned models. For zero-shot ASR, ten models (all seven Whisper sizes, MMS-1B, SeamlessM4T-v2-large, and OmniASR-CTC-300M) are evaluated on the FLEURS Pashto test set and a filtered Common Voice~24 subset; zero-shot Whisper WER ranges from 90% to 297%, with the medium model collapsing to 461% on Common Voice~24 consistent with decoder looping. SeamlessM4T achieves 39.7% WER on Common Voice~24 (the best zero-shot result reported to date, as of submission); MMS-1B achieves 43.8% on FLEURS. For script failure, a language-identification audit shows that no Whisper model produces Pashto-script output in more than 0.8% of utterances, while MMS-1B, SeamlessM4T, and OmniASR each exceed 93% Pashto-script fidelity; WER alone does not reveal this failure, since a model generating Arabic-script output on Pashto audio has not achieved ASR in any interpretable sense. For cross-domain evaluation, five fine-tuned Pashto ASR models are evaluated on both test sets: published WER figures of 14% degrade to 32.5--59% on out-of-distribution sets, while one augmented model achieves 35.1% on both sets with zero cross-domain degradation. Character-class error stratification confirms that Pashto-unique phonemes (the retroflex series and lateral fricatives) account for disproportionate error mass. All evaluations cover read speech only. Five structural impediments to cumulative progress are identified and five ordered research priorities are argued.

• The paper establishes a comprehensive benchmark for evaluating multilingual ASR models on Pashto, focusing on zero-shot performance and script fidelity issues.
• The paper reveals severe performance degradation in Whisper variants—with WERs up to 461%—and identifies specific failure modes such as decoder looping and script mismatches.
• The paper demonstrates that fine-tuning with data augmentation significantly mitigates cross-domain performance drops, ensuring more consistent ASR outcomes.

Benchmarking Multilingual Speech Models on Pashto: Zero-Shot ASR, Script Failure, and Cross-Domain Evaluation

Introduction

This paper establishes a rigorous and reproducible benchmark for evaluating multilingual automatic speech recognition (ASR) models on Pashto, focusing on zero-shot recognition, script fidelity, and cross-domain generalization (2604.04598). In the context of low-resource spoken language processing, Pashto presents several formidable challenges: the lack of shared public ASR test sets, the absence of open-source TTS systems, orthographic complexity due to script extension, and distinctive phonemic attributes not represented in most cross-lingual training corpora. The presented benchmark covers controlled (FLEURS) and crowd-sourced (Common Voice 24) read-speech settings, with ten prominent multilingual ASR models and five fine-tuned models under systematic evaluation.

Zero-Shot Benchmarking: Model Coverage and Primary Findings

The evaluation encompasses all Whisper model sizes, MMS-1B, SeamlessM4T-v2-large, and OmniASR-CTC-300M under a fully deterministic protocol. A key empirical finding is that, when transcribing Pashto audio, the Whisper family suffers catastrophic performance degradation, manifesting in both exceedingly high word error rates (WERs) and script-level failures. Whisper-medium, in particular, exhibits decoder instability, reflected in a 461% WER on CV24_filtered—a product of decoder looping and non-pashto output pathways.

Figure 1: Distributions of zero-shot WER for all evaluated models, highlighting extreme performance degradation in Whisper, and significantly superior results from SeamlessM4T, MMS-1B, and OmniASR.

SeamlessM4T-v2-large achieves a WER of 39.7% on CV24_filtered, outperforming other zero-shot baselines. MMS-1B and OmniASR exhibit WERs of 43.8% and 45.1% on FLEURS and CV24_filtered, respectively—providing the first interpretable reference points for zero-shot Pashto ASR on public data.

Script Fidelity Auditing and Failure Modes

An original contribution is the explicit quantification of script fidelity—measuring whether model outputs are rendered in the authentic Pashto codepoint inventory. Most Whisper outputs contain either Arabic/Dari/Urdu script or entirely indeterminate content, with a maximum of 0.8% of true Pashto-script output for any size. This fundamental failure mode is not apparent from WER alone, as WER can be inflated by orthographically similar but semantically invalid script substitutions. In contrast, MMS-1B, SeamlessM4T, and OmniASR produce Pashto-script outputs in over 93% of utterances.

Figure 2: Per-model script output distributions on FLEURS; Whisper variants overwhelmingly fail to generate Pashto script, in stark contrast to MMS-1B, SeamlessM4T, and OmniASR.

Further, two Whisper-specific failure regimes are identified: (1) systematic substitution of Pashto-unique phonemes with Arabic-script analogues, and (2) language-agnostic decoder looping, especially prominent in the medium size. These failure mechanisms are decoupled from acoustics and must be addressed at the script-handling and output-decoding layers, not at the representation level.

Cross-Domain Generalization of Fine-Tuned Models

The paper assesses five fine-tuned models spanning three architectures (Whisper Base, W2V-BERT 2.0, XLS-R) and two training corpora, explicitly examining out-of-distribution generalization. The widely cited WERs of 13.96–14% are shown to hold only in in-domain scenarios and degrade sharply (to 32.5–59%) when evaluated on independent data. Data augmentation—specifically speed perturbation and additive noise—enables w2v-b2-aug to maintain a virtually identical WER (35.1%) across both FLEURS and CV24_filtered.

Figure 3: Cross-domain generalization scatterplot; points above the diagonal represent significant degradation on CV24 relative to FLEURS. Points highlighted show cross-domain penalty for non-augmented models.

Figure 4: Comparative WER on FLEURS and CV24_filtered for top zero-shot and all fine-tuned systems. Fine-tuned models vary widely in robustness; w2v-b2-aug eliminates cross-domain performance drops.

Phoneme Class Error Analysis

A granular stratification of errors by character class demonstrates that the primary challenge for Pashto ASR remains the accurate modeling of phonemes that are unique or nearly unique to Pashto, particularly lateral fricatives and the four-member retroflex stop series. Fine-tuned models, despite domain adaptation, allocate disproportionate error mass to these phones, indicating incomplete transfer and representation in encoder pretraining.

Figure 5: Character-class-specific WER deviation for pashto-asr-v3 on FLEURS, with the greatest elevation localized to voiced and voiceless lateral fricatives and retroflex stops.

Implications and Open Research Problems

Practical Implications:

• Whisper models are unreliable for Pashto ASR in both zero-shot and fine-tuned settings unless explicit Pashto-script output is validated.
• MMS-1B, SeamlessM4T, and OmniASR offer interpretable zero-shot baselines for future research, provided script fidelity is systematically monitored.
• Cross-domain WER degradation invalidates in-domain performance reporting as a sufficient metric for real-world or even platform-transfer generalization.

Open Research Priorities:

1. Open-Source TTS: The most urgent infrastructural gap is the absence of a peer-reviewed, open-source TTS system for Pashto, despite the existence of substantial synthetic speech corpora and applicable architectures (e.g., VITS, Kokoro-82M).
2. Spontaneous and Dialectal Corpora: All current results are limited to read speech; spontaneous speech data collection and dialectal annotation are both feasible and would immediately improve representational coverage.
3. Benchmark Adoption: Consistent adoption of the provided evaluation and normalization protocol would establish comparability across the field.
4. G2P and Pronunciation Lexicons: A rule-based G2P for Pashto, grounded in contemporary generative phonology, would unlock advanced TTS architectures and fine-grained error analyses.
5. Unexplored Architectures: Zero-shot evaluation of XLS-R and HuBERT encoders on the provided benchmarks is warranted, as is controlled fine-tuning of WavLM and HuBERT.

Conclusion

This work codifies the first deterministic, public, and multi-system ASR benchmark for Pashto, resolving reproducibility and comparability barriers that previously precluded cumulative progress. The explicit discrimination of script fidelity as a required evaluation axis distinguishes acoustic-phonetic mismatches from catastrophic script-level failures, especially critical for the extended Arabic script languages. Strong empirical evidence is provided for the necessity of cross-domain evaluation and augmentation, as well as for the non-monotonic behavior of popular ASR architectures in low-resource settings. Future progress will hinge on the adoption of these benchmarks, the systematic release and evaluation of TTS systems, and the collection of spontaneous and richly annotated spoken Pashto corpora.