Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation (2004.11867v1)

Abstract: Massively multilingual models for neural machine translation (NMT) are theoretically attractive, but often underperform bilingual models and deliver poor zero-shot translations. In this paper, we explore ways to improve them. We argue that multilingual NMT requires stronger modeling capacity to support language pairs with varying typological characteristics, and overcome this bottleneck via language-specific components and deepening NMT architectures. We identify the off-target translation issue (i.e. translating into a wrong target language) as the major source of the inferior zero-shot performance, and propose random online backtranslation to enforce the translation of unseen training language pairs. Experiments on OPUS-100 (a novel multilingual dataset with 100 languages) show that our approach substantially narrows the performance gap with bilingual models in both one-to-many and many-to-many settings, and improves zero-shot performance by ~10 BLEU, approaching conventional pivot-based methods.

Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation

This paper discusses enhancements to massively multilingual neural machine translation (NMT) systems, with a focus on bolstering both multilingual and zero-shot translation capabilities. The authors address two critical challenges: the limited modeling capacity of multilingual NMT systems and the issue of off-target translation in zero-shot scenarios.

Key Contributions and Findings

1. Modeling Capacity Enhancements:
   • The authors propose augmenting the capacity of multilingual NMT models through deeper architectures and language-specific components. Their investigation reveals substantial improvements in translation quality by adopting deeper Transformer models and incorporating language-aware layer normalization (LaLn) and language-aware linear transformations (LaLt).
   • Experiments demonstrate that these modifications notably narrow the performance gap between multilingual and bilingual NMT models, especially in low-resource scenarios.
2. Zero-Shot Translation Improvements:
   • A major focus of the paper is zero-shot translation, wherein a model translates between language pairs unseen during training. To tackle the prevalent issue of off-target translation—where models translate into an unintended language—the authors introduce the random online backtranslation (ROBt) algorithm.
   • The ROBt technique shows significant improvement in translation quality, notably enhancing the BLEU score by approximately 10 points for zero-shot translation, achieving performance levels near conventional pivot-based methods.
3. Empirical Evaluation:
   • The research utilizes the OPUS-100 dataset, containing data for 100 languages, as a benchmark for evaluating the proposed methods. This dataset is significant for its scale, allowing the exploration of massively multilingual settings not commonly addressed in prior studies.
   • With empirical results highlighting a 92.6% win ratio against baseline models, the enhanced NMT systems confirm the feasibility of managing numerous translation directions effectively.
4. Impact of Dataset and Training Strategies:
   • The paper also assesses the influence of training data size within the OPUS-100 dataset, observing that low-resource languages particularly benefit from the increased model expressivity provided by LaLn and LaLt.
   • The paper emphasizes that while increased model capacity universally benefits translation quality, it is most effective when combined with data-centric techniques like ROBt for zero-shot scenarios.

Implications and Future Work

The enhancements proposed in this paper represent a forward step in the scalability and effectiveness of multilingual NMT architectures. By successfully integrating model expressivity improvements with practical data augmentation techniques, the research sets a foundation for further studies in optimizing NMT systems for extensive multilingual use.

Future work could delve into reducing the computational overhead associated with language-aware transformations or exploring alternative backtranslation strategies that might further refine zero-shot translation quality. Additionally, advancements in generative modeling and unsupervised learning, as suggested by the authors, could present promising avenues for overcoming current limitations in zero-shot translation, possibly further transcending the efficacy of existing pivot-based frameworks.

Overall, this paper provides substantial insights into the challenges and solutions associated with massively multilingual NMT, contributing both practical strategies and theoretical advancements to the field.