Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond (1812.10464v2)

Abstract: We introduce an architecture to learn joint multilingual sentence representations for 93 languages, belonging to more than 30 different families and written in 28 different scripts. Our system uses a single BiLSTM encoder with a shared BPE vocabulary for all languages, which is coupled with an auxiliary decoder and trained on publicly available parallel corpora. This enables us to learn a classifier on top of the resulting embeddings using English annotated data only, and transfer it to any of the 93 languages without any modification. Our experiments in cross-lingual natural language inference (XNLI dataset), cross-lingual document classification (MLDoc dataset) and parallel corpus mining (BUCC dataset) show the effectiveness of our approach. We also introduce a new test set of aligned sentences in 112 languages, and show that our sentence embeddings obtain strong results in multilingual similarity search even for low-resource languages. Our implementation, the pre-trained encoder and the multilingual test set are available at https://github.com/facebookresearch/LASER

Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond

The paper "Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond" introduces a novel approach to learning joint multilingual sentence representations covering 93 languages from over 30 different language families and utilizing 28 distinct scripts. The primary motivation behind this work lies in the development of universal language-agnostic sentence embeddings that perform robustly across various tasks and languages, leveraging shared information to benefit low-resource languages and enable zero-shot transfer across different languages.

Methodology

The proposed architecture employs a single, language-agnostic BiLSTM encoder trained on publicly available parallel corpora. This shared encoder is supplemented by an auxiliary decoder, facilitating the learning of sentence embeddings from a variety of multilingual corpora through a common Byte-Pair Encoding (BPE) vocabulary of 50,000 operations. The encoder output is max-pooled to obtain fixed-length sentence embeddings, which are crucial for diverse NLP tasks.

The training strategy focuses on using two target languages (English and Spanish) for scaling purposes, rather than relying on an expensive and often unavailable N-way parallel corpus. This results in efficiency both in terms of computational resources and training duration. The pre-trained encoder is subsequently evaluated on several multilingual NLP tasks without any task-specific fine-tuning.

Experimental Evaluation

The efficacy of the proposed method is demonstrated across multiple tasks:

1. Cross-Lingual Natural Language Inference (XNLI): The zero-shot transfer performance is benchmarked on the XNLI dataset, consisting of 15 languages. The proposed embeddings yield superior results, surpassing several existing models, including multilingual BERT, especially in terms of maintaining stability across languages. The average accuracy drop for transfer languages compared to English is minimal, showcasing the model's robustness.
2. Cross-Lingual Document Classification (MLDoc): The embeddings are evaluated on the MLDoc dataset, wherein the proposed method achieves the highest accuracy for 5 out of 7 languages, demonstrating its strength in zero-shot document classification.
3. Bitext Mining (BUCC): The approach achieves new state-of-the-art F1 scores on the BUCC mining task by implementing an advanced scoring function to address scale inconsistency issues inherent in cosine similarity-based methods.
4. Multilingual Similarity Search (Tatoeba): The paper introduces a new test set of aligned sentences in 112 languages. Results show that the proposed embeddings achieve low similarity error rates for a substantial number of these languages, even those with minimal training data.

Implications and Future Directions

The empirical results indicate that joint training on a diverse set of languages significantly enhances the embeddings' generalizability and performance on multilingual tasks, underlining the benefits of a shared multilingual model over separate language-specific models. This work opens up avenues for improved cross-lingual transfer, especially for low-resource languages.

Future directions could include exploring alternative encoder architectures such as Transformer models, integrating monolingual data through techniques like back-translation, using pre-trained monolingual word embeddings, as well as developing language-agnostic preprocessing methods to further enhance the generalizability and applicability of the model.

Conclusion

This paper showcases a comprehensive and effective methodology for learning massively multilingual sentence embeddings that facilitate zero-shot cross-lingual transfer. The presented approach sets new benchmarks on several widely recognized evaluation tasks and fosters future advancements in the development of multilingual NLP models that can seamlessly support a broad spectrum of languages with minimal resource dependencies. The work exemplifies a significant step forward in universal LLMing and has potential implications for a wide array of multilingual applications.