Towards Building Large Scale Datasets and State-of-the-Art Automatic Speech Translation Systems for 14 Indian Languages (2411.04699v3)

Abstract: Speech translation for Indian languages remains a challenging task due to the scarcity of large-scale, publicly available datasets that capture the linguistic diversity and domain coverage essential for real-world applications. Existing datasets cover a fraction of Indian languages and lack the breadth needed to train robust models that generalize beyond curated benchmarks. To bridge this gap, we introduce BhasaAnuvaad, the largest speech translation dataset for Indian languages, spanning over 44 thousand hours of audio and 17 million aligned text segments across 14 Indian languages and English. Our dataset is built through a threefold methodology: (a) aggregating high-quality existing sources, (b) large-scale web crawling to ensure linguistic and domain diversity, and (c) creating synthetic data to model real-world speech disfluencies. Leveraging BhasaAnuvaad, we train IndicSeamless, a state-of-the-art speech translation model for Indian languages that performs better than existing models. Our experiments demonstrate improvements in the translation quality, setting a new standard for Indian language speech translation. We will release all the code, data and model weights in the open-source, with permissive licenses to promote accessibility and collaboration.

• The paper introduces the BhasaAnuvaad dataset, comprising over 44,400 hours of audio and 17 million text segments to address the scarcity of AST resources for Indian languages.
• The paper evaluates current AST systems and reveals that they struggle with spontaneous speech due to disfluencies and colloquial variations.
• The paper leverages a mix of curated, web-mined, and synthetic data to improve AST training and inspire future advancements in model architectures.

Overview and Analysis of "BhasaAnuvaad: A Speech Translation Dataset for 13 Indian Languages"

The paper "BhasaAnuvaad: A Speech Translation Dataset for 13 Indian Languages" presents a substantial contribution to the field of Automatic Speech Translation (AST) by addressing the critical scarcity of translation datasets for Indian languages. Despite India's linguistic diversity, spanning 22 officially recognized languages, AST development in this region has been hindered by a lack of comprehensive resources—especially when compared to high-resource languages like English. This research evaluates current AST systems, identifying key deficiencies, and introduces the BhasaAnuvaad dataset, a comprehensive resource that spans 13 Indian languages along with English, comprising over 44,400 hours of audio and 17 million text segments.

Current State of AST Systems

The paper begins with an evaluation of widely-used AST systems on Indian languages, with a focus on systems’ performance on both read and spontaneous speech. The authors find that existing systems perform adequately on read speech but significantly struggle with spontaneous speech, marked by disfluencies such as pauses, hesitations, and colloquial language, which are prevalent in everyday communication. This performance gap underscores the necessity for datasets that better reflect real-world conditions, thus motivating the creation of BhasaAnuvaad.

BhasaAnuvaad Dataset Composition

The BhasaAnuvaad dataset is notable for its scale and diversity. It includes speech-to-text pairs for both English-to-Indic and Indic-to-English translations. The dataset integrates three types of data sources: curated datasets from existing resources, large-scale web-mined data, and synthetic data generation. This variety aims to cover a broad spectrum of language use cases and promote robust AST system development.

1. Curated Datasets: The authors have incorporated high-quality datasets from sources including TED Talks and educational platforms which contribute to a broad base of multilingual content.
2. Web Mining: By employing advanced mining techniques, the authors have extracted parallel data from accessible online content, enriching the dataset while ensuring diversity in linguistic style and context.
3. Synthetic Data Generation: To further overcome the limitations of existing resources, the research employs synthetic data generation techniques. This approach enables the simulation of spontaneous and varied speech patterns, which are essential for training models capable of handling real-world dialogue.

Practical and Theoretical Implications

The introduction of BhasaAnuvaad stands to significantly impact both practical applications and theoretical advancements in AST:

• Practical Implications: Researchers and developers working with Indian languages now have access to a rich resource to train and evaluate AST systems that can perform better in real-world scenarios involving spontaneous speech. This can lead to more effective communication tools in various sectors including business, education, and public administration.
• Theoretical Developments: The dataset also provides an opportunity to develop and refine models that accommodate code-switching and dialectal variations common in Indian languages. It challenges existing models to adapt to and excel in conditions that more closely mirror actual user experiences.

Future Directions

The paper's contribution also points to future research avenues, particularly in enhancing model architectures and training methodologies to further improve AST performance on spontaneous speech and lower-resource language pairs. The potential for language-specific adaptations and innovations, parallel resource expansion to other underrepresented languages in the AST field, and novel model evaluation frameworks that encapsulate diverse linguistic phenomena are promising areas for ongoing investigation.

In conclusion, "BhasaAnuvaad" represents a significant step forward in the resource landscape for Indian language AST. While the dataset by itself does not solve all challenges associated with multilingual speech translation, it provides a formidable foundation upon which further advancements can be built. The research not only presents a valuable resource but also invites further exploration into more sophisticated model architectures capable of leveraging the multifaceted nature of this dataset.