Factors affecting ASR performance: A study using state of the art ASR models in Indic Languages

Abstract: ASR performance varies across languages, speakers, and recording conditions, yet systematic analysis for Indic languages remain limited. We present a large-scale study of decoded outputs from multiple open-source ASR models evaluated on diverse Indian speech datasets in zero-shot settings. We analyze linguistic, speaker-level, and acoustic factors across Hindi, Bengali, Kannada, Telugu, and Marathi. We examine correlations between WER and speaker traits such as average word length, speaking rate, and utterance duration across multiple model dataset pairs. For Hindi, we further analyze audio factors including telephone codecs, bit depth, resampling, and background noise. Results reveal both cross lingual patterns and language-specific sensitivities, showing how speaker behavior and signal processing choices affect ASR robustness in real world Indic scenarios.

• The paper identifies key speaker-level factors—average word length, speaking rate, and utterance duration—that systematically influence ASR accuracy in Indic languages.
• It evaluates audio-level degradations, showing that techniques like classical interpolation outperform neural upsampling for maintaining robust ASR performance.
• The study reveals cross-lingual consistency in ASR error trends and emphasizes the need for tailored preprocessing and model retraining in realistic, bandwidth-constrained environments.

Systematic Analysis of Factors Affecting ASR Performance in Indic Languages

Introduction

This paper provides a comprehensive, empirical study dissecting the principal speaker-level and audio-level factors that influence automatic speech recognition (ASR) performance across five major Indic languages—Hindi, Bengali, Kannada, Telugu, and Marathi—using a suite of state-of-the-art ASR architectures in zero-shot conditions. The analysis spans multiple open-source models (Whisper, Wav2Vec2, Data2Vec, Conformer) and a diverse set of speech corpora reflecting real-world recording conditions, morphological variation, and acoustic scenarios. The paper bridges a crucial gap in unified, cross-lingual evaluation and identification of performance bottlenecks specific to the Indian linguistic and deployment environment, with rigorous experiments on both high-level (speaker, language) and low-level (signal processing) axes.

Methodological Framework

The evaluation isolates two axes of variability: (1) speaker-level factors—average word length (AWL), speaking rate (WPM), and utterance duration (AL)—reflecting linguistic complexity and spontaneous speech properties; and (2) audio-level factors, particularly for Hindi, modeling realistic telephony and low-fidelity degradations via codec simulations, bit-depth quantization, resampling/upsampling strategies, and additive noise scenarios. ASR model performance is assessed using Word Error Rate (WER), with consistent normalization protocols for hypothesis and reference text.

The experimental apparatus comprises both Indic-centric and multilingual ASR models, including proprietary and open collections, and benchmarks performance over leading publicly available datasets such as MUCS, Kathbath, IndicTTS, Common Voice, FLEURS, Vaani, and RESPIN.

Speaker-Level Factors: Linguistic and Temporal Effects

The study identifies systematic and, in many cases, language-generalizable trends in how speaker-level properties impact WER. Detailed quantitative trend analysis demonstrates:

• Average Word Length (AWL): WER follows a U-shaped trend, where both extremely short and long average word lengths induce higher errors. Medium-range AWLs minimize WER, suggesting a sweet spot in morphological regularity. This trend is highly consistent across all languages and model/dataset pairs.

Figure 1: WER trends in Hindi as a function of average word length (AWL), capturing the characteristic U-shaped dependency.

• Speaking Rate (WPM): WER-bias exhibits marked language-dependence. In Hindi, higher WPM actually lowers WER, contradicting the expected effect of faster articulation causing more errors. This is presumably mediated by hesitations and fluency disruptions at low rates. In contrast, other Indic languages display patterns more consistent with canonical ASR findings: aggressive articulation (very high WPM) degrades recognition, likely due to phoneme coarticulation and slurring.
• Utterance Duration: Short utterances result in elevated WER, attributable to boundary effects and limited phonetic context, while very long durations see accumulative errors. The trajectory is stable across all model families analyzed.

This robust demarcation of speaker-level factor trends provides fundamental constraints for future corpus design and benchmarking protocols for Indic ASR, allowing modelers to anticipate and normalize for these variabilities.

Audio-Level Factors: Signal Degradation and ASR Robustness

A central thrust of the study is the stringent evaluation of ASR resilience to realistic signal distortions and telephony pathologies, particularly for Hindi. Four classes of degradations are considered:

• Amplitude Quantization: Models remain stable up to 10–12 bit PCM quantization, with sharp WER increases at 8 bits and a catastrophic rise at 6 bits.

Figure 2: ASR performance as a function of amplitude quantization, illustrating the critical degradation below 10-bit precision.

• Telephony Bandwidth and Codecs: GSM (2G) simulation yields substantial performance degradation due to its narrow bandwidth and low bit-rate quantization. 3G/4G (narrowband and wideband) processing, as well as Opus (5G), maintain near-parity with the 16 kHz original, firmly establishing bandwidth preservation as the dominant factor in telephony ASR efficacy.
• Downsampling and Restoration: Upon 4 kHz and 8 kHz downsampling, resynthesis through classical interpolation (linear, polyphase via soxr) incurs moderate WER rises. Notably, neural super-resolution techniques (VoiceFixer, AudioSR) exacerbate WER despite their perceptual advantages for human listeners, as they inject confounding frequency artifacts not modeled during ASR training.

Figure 3: Effect of 4 kHz downsampling and alternative restoration strategies on Hindi ASR WER; neural restoration aggravates error rates.

• Additive Noise: As expected, all models display monotonic WER increases with declining SNR for white noise, environmental, and competing background speech conditions. Noteworthy is the finding that Whisper architectures manifest greater robustness to overlapping speech compared to Conformer models; in contrast, both architectures show comparable stability for stationary or natural non-speech interference.

Figure 4: WER as a function of SNR under white Gaussian noise for Hindi ASR, evidencing typical monotonic degradation.

Implications and Future Directions

The cross-lingual consistency of morpho-temporal biases indicates that ASR improvements will require structural advances in morphological modeling and attention mechanisms for boundary contexts. The pronounced codec and bandwidth effects, especially under GSM constraints, highlight the necessity of tailored frontend pre-processing and model retraining under bandwidth-limited conditions for reliable field deployment.

A particularly significant, contradictory finding is that neural audio restoration—though optimal for human perception—deteriorates ASR performance substantially across state-of-the-art architectures, emphasizing the misalignment between perceptual audio enhancement and ASR model training objectives. Consequently, classical interpolation methods remain preferred for practical ASR system design, barring the availability of ASR-tuned neural upsamplers.

Robustness to aggressive quantization above 10-bits validates the deployment of such systems on edge devices with bandwidth and storage constraints, delineating safety bounds for signal fidelity. The improved resilience of Whisper-style models to speech-on-speech interference encourages their adoption in overlapping real-world dialog environments such as call centers and automated telephony.

In the broader context, the findings provide a blueprint for future ASR development—demanding task-specific signal restoration, large-scale augmentation pipelines mimicking real deployment corruptions, and error weighting according to morpho-temporal factor distributions in target populations. Further research could focus on jointly optimized restoration-ASR pipelines and multilingual retraining under cross-lingual and inter-speaker heterogeneity.

Conclusion

This work establishes a rigorous empirical paradigm for analyzing the multifactorial underpinnings of ASR performance in Indic languages. The unified framework and multi-language evaluation not only expose the structural weaknesses of contemporary ASR models in morphologically complex, bandwidth-constrained, and noisy environments, but also offer actionable insights into robust system design for broad deployment. Prioritizing bandwidth preservation, precise amplitude encoding, and rejecting naive perceptual audio restorers are critical for effective ASR in realistic Indian contexts, setting a new reference for both development and deployment of Indic ASR systems (2606.09335).