No Language Left Behind: Scaling Human-Centered Machine Translation (2207.04672v3)

Abstract: Driven by the goal of eradicating language barriers on a global scale, machine translation has solidified itself as a key focus of artificial intelligence research today. However, such efforts have coalesced around a small subset of languages, leaving behind the vast majority of mostly low-resource languages. What does it take to break the 200 language barrier while ensuring safe, high quality results, all while keeping ethical considerations in mind? In No Language Left Behind, we took on this challenge by first contextualizing the need for low-resource language translation support through exploratory interviews with native speakers. Then, we created datasets and models aimed at narrowing the performance gap between low and high-resource languages. More specifically, we developed a conditional compute model based on Sparsely Gated Mixture of Experts that is trained on data obtained with novel and effective data mining techniques tailored for low-resource languages. We propose multiple architectural and training improvements to counteract overfitting while training on thousands of tasks. Critically, we evaluated the performance of over 40,000 different translation directions using a human-translated benchmark, Flores-200, and combined human evaluation with a novel toxicity benchmark covering all languages in Flores-200 to assess translation safety. Our model achieves an improvement of 44% BLEU relative to the previous state-of-the-art, laying important groundwork towards realizing a universal translation system. Finally, we open source all contributions described in this work, accessible at https://github.com/facebookresearch/fairseq/tree/nllb.

No Language Left Behind: Scaling Human-Centered Machine Translation

The paper "No Language Left Behind: Scaling Human-Centered Machine Translation" addresses the significant challenges associated with extending machine translation (MT) support to over 200 languages, with a specific focus on low-resource languages. The researchers developed several novel techniques in data collection, model architecture, and evaluation metrics to tackle these challenges. This essay explores the main methodologies, results, and implications of these research efforts.

Core Contributions

1. FLORES-200 Benchmark Dataset: Building on the prior FLORES-101, the team created FLORES-200, a multilingual benchmark dataset that covers 204 languages by professionally translating 3001 English sentences. This dataset allows for rigorous evaluation of translation quality across an unprecedented number of translation directions—over 40,000.
2. Bitext Mining with LASER3: The researchers developed LASER3, a set of multilingual sentence encoders trained using a teacher-student architecture. This approach enabled bitext mining at a massive scale, creating 1.1 billion sentence pairs in 148 languages, significantly enriching low-resource language corpora.
3. Conditional Computation Using Mixture-of-Experts: To efficiently manage the varying resource availability among languages, the paper employs a Sparsely Gated Mixture of Experts (MoE) architecture. This strategy conditions the computational pathways in the model based on the input, thereby optimizing parameter utilization and reducing interference among languages.
4. Self-Supervision and Curriculum Learning: Integration of self-supervised learning with large-scale monolingual data and a curriculum learning strategy for phased training addresses overfitting issues in low-resource settings. These methods improve generalization and translation quality for low-resource language pairs.
5. Documenting and Sharing Contributions: All datasets, models, and tools developed during this research have been open-sourced. This transparency supports further research and practical deployment of MT systems for low-resource languages.

Numerical Results

The paper presents robust numerical results demonstrating the efficacy of their methods:

• Translation Quality: The model achieves a 44% relative BLEU score improvement over the previous state-of-the-art, indicating substantial advancements in translation accuracy.
• FLORES-200 Coverage: Performance assessments using the FLORES-200 benchmark reveal consistent improvements across high-resource and low-resource language pairs, validating the practical effectiveness of the proposed techniques.
• Bitext Mining Scale: By mining over 1.1 billion sentence pairs, the researchers dramatically increase the available bitext for many low-resource languages.

Architectural Innovations

1. Conditional Compute with MoE: The MoE model effectively manages the dynamic computational needs of different languages by activating only a subset of the total parameters for any given input. This targeted activation not only augments parameter efficiency but also mitigates detrimental interference across languages.
2. Training Strategies: Leveraging self-supervised learning and a phased curriculum training regimen significantly enhances model robustness, especially for low-resource languages. This combination ensures that high-resource languages benefit from exhaustive training while preventing overfitting for low-resource languages.

Practical and Theoretical Implications

Practical Implications: The advancements proposed in this paper hold significant promise for democratizing access to high-quality MT for low-resource languages. By making these languages more accessible digitally, we enable broader participation in global knowledge sharing and foster cultural preservation. Additionally, these methods can be employed to create more inclusive tools, such as translation support for Wikipedia, thus enriching information accessibility.

Theoretical Implications: The work contributes to the growing body of knowledge on multilingual models and conditional computation. The insights gained from effectively managing model capacity and leveraging large-scale self-supervised learning can inform future research on extending multilingual capabilities even further, potentially encompassing more dialects and regional variations.

Future Directions

The research opens several avenues for future exploration:

• Broader Application of MoE: Investigating the use of MoE in other NLP tasks and its potential integration with emerging model architectures can further optimize multilingual NLP models.
• Extended Language Support: Additional research into low-resource languages, particularly those without standardized written forms or predominantly oral languages, could expand the reach of inclusive MT systems.
• Ethical Considerations: Continued reflection on data ownership and ethical deployment will be critical, particularly in ensuring the benefits of technological advancements are equitably distributed among all language communities.

Conclusion

This paper makes significant strides toward the inclusion of low-resource languages in mainstream machine translation technologies. Through innovative datasets, advanced model architectures, and comprehensive evaluation frameworks, the researchers construct a blueprint for future efforts in expanding language coverage in AI. The open-source approach ensures that these contributions will have a lasting impact on the field, fostering further innovation and application in multilingual NLP.