Twists, Humps, and Pebbles: Multilingual Speech Recognition Models Exhibit Gender Performance Gaps (2402.17954v3)

Abstract: Current automatic speech recognition (ASR) models are designed to be used across many languages and tasks without substantial changes. However, this broad language coverage hides performance gaps within languages, for example, across genders. Our study systematically evaluates the performance of two widely used multilingual ASR models on three datasets, encompassing 19 languages from eight language families and two speaking conditions. Our findings reveal clear gender disparities, with the advantaged group varying across languages and models. Surprisingly, those gaps are not explained by acoustic or lexical properties. However, probing internal model states reveals a correlation with gendered performance gap. That is, the easier it is to distinguish speaker gender in a language using probes, the more the gap reduces, favoring female speakers. Our results show that gender disparities persist even in state-of-the-art models. Our findings have implications for the improvement of multilingual ASR systems, underscoring the importance of accessibility to training data and nuanced evaluation to predict and mitigate gender gaps. We release all code and artifacts at https://github.com/g8a9/multilingual-asr-gender-gap.

Evaluation of Gender Performance Disparities in Multilingual ASR Models

Introduction

The paper investigates gender performance gaps in multilingual Automatic Speech Recognition (ASR) systems, specifically focusing on OpenAI's Whisper and Meta's SeamlessM4T models. The paper systematically examines how these models perform across different genders using a diverse set of corpora that contains speech in 19 languages from seven distinct language families. Despite advancements in multilingual ASR, the paper reveals persistent and varying gender disparities that necessitate closer examination for future model improvements and fairer implementations.

Methodology

Multilingual ASR Models: The models evaluated are Whisper and SeamlessM4T, both state-of-the-art representatives in multilingual and multitask ASR. These models are selected due to their prominence and diverse language handling capabilities.

Datasets: The analysis utilizes three datasets—Mozilla Common Voice (CV), Google Fleurs, and Meta VoxPopuli. These datasets provide a range of recording conditions from read speech (CV, Fleurs) to spontaneous speech (VoxPopuli), ensuring varied acoustic environments.

Language Selection: The paper includes 19 languages with sufficient gender-tagged speech data, ensuring a balanced representation across seven language families. This selection helps in providing comprehensive insights into linguistic and gender performance gaps.

Evaluation Metrics: The paper uses standard ASR metrics—Word Error Rate (WER) and Character Error Rate (CER). Gender performance gaps are evaluated using a Pairwise Comparison Metric (PCM), examining relative performance differences between male and female speakers.

Key Findings

1. Gender Performance Gaps: Across all datasets and languages, the models exhibit variable performance trends, sometimes favoring female speakers and other times male speakers. It's particularly noted that Whisper and SeamlessM4T show gaps in read speech datasets (CV and Fleurs) more prominently than in spontaneous speech (VoxPopuli).
2. Phonetic Feature Analysis: Analyzing phonetic features like pitch, intensity, and speaking rate reveals no significant differences that can explain the observed gender gaps. Despite acoustic feature parity, the models still show varied recognition performance across genders.
3. Gender Probing: Utilizing gender probes suggests that multilingual ASR models encode gender information differently in their internal states. A negative correlation exists between the model’s ability to distinguish gender and the performance gap, indicating that easier gender differentiation within the model's internal representation correlates with reduced gender bias in ASR performance.

Implications

Practical Implications: The findings suggest that current multilingual ASR models still harbor gender biases, which can result in unequal service quality. This insight is crucial for developers aiming to fine-tune ASR systems for fairer performance. Ensuring gender-balanced training data and incorporating fairness in model assessment become essential steps forward.

Theoretical Implications: The correlation between internal gender encoding and performance gaps provides an avenue for deeper exploration into model interpretability. Future research could leverage these insights to develop more inclusive and fair ASR models. The paper underscores the need for gender-aware model evaluations and potentially new metrics that can more holistically capture fairness dimensions.

Future Research Directions: To build on these findings, future work could:

• Extend evaluations to other sociodemographic factors such as age, dialect, and socio-cultural background.
• Investigate mitigation strategies for identified gender biases, including data augmentation and adversarial training techniques.
• Explore the creation and evaluation of new datasets with ideally balanced and diverse speaker distributions to provide reliable fairness assessments.
• Examine intrinsic and extrinsic biases using comparative analyses across multiple ASR models and languages.

Conclusion

The paper provides a rigorous and comprehensive assessment of gender performance disparities in leading multilingual ASR models. The findings encapsulate the persisting gender biases that require targeted interventions for future model iterations. By highlighting the nuanced relationship between gender-encoded internal states and performance gaps, the paper charts a path for more comprehensive fairness evaluations in speech recognition technologies. Future research and development efforts must prioritize gender inclusivity to ensure fair and robust ASR systems that can serve diverse user bases equitably.