Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation (2004.09813v2)

Abstract: We present an easy and efficient method to extend existing sentence embedding models to new languages. This allows to create multilingual versions from previously monolingual models. The training is based on the idea that a translated sentence should be mapped to the same location in the vector space as the original sentence. We use the original (monolingual) model to generate sentence embeddings for the source language and then train a new system on translated sentences to mimic the original model. Compared to other methods for training multilingual sentence embeddings, this approach has several advantages: It is easy to extend existing models with relatively few samples to new languages, it is easier to ensure desired properties for the vector space, and the hardware requirements for training is lower. We demonstrate the effectiveness of our approach for 50+ languages from various language families. Code to extend sentence embeddings models to more than 400 languages is publicly available.

Overview of "Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation"

The paper "Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation" by Nils Reimers and Iryna Gurevych introduces a method for extending monolingual sentence embeddings to new languages via a process called multilingual knowledge distillation. This approach leverages knowledge distillation to map translated sentences to the same location in the vector space as their original counterparts, enabling the creation of multilingual versions from existing monolingual models.

Methodology

The core idea is to employ a teacher-student training schema where the teacher model is a pre-existing monolingual sentence embedding model. The embeddings for source language sentences are generated using this teacher model. These sentences are then translated into target languages, and the student model is trained to produce embeddings that mimic the teacher's embeddings for both the original and translated sentences.

The authors utilize mean squared loss to ensure that the distances in the vector space are preserved during translation:

$$\frac{1}{|\mathcal{B}|} \sum_{j \in \mathcal{B}} \left[ (M(s_j) - \hat M(s_j))^2 + (M(s_j) - \hat M(t_j))^2 \right]$$

For the teacher model, the authors predominantly use an English SBERT model fine-tuned on datasets like AllNLI and the STS benchmark. The student model primarily used for experiments is XLM-R, although other architectures can also be applied.

Training Data

The paper highlights the importance of parallel datasets for training the student model. To this end, the authors utilize a variety of datasets, such as GlobalVoices, TED2020, NewsCommentary, WikiMatrix, Tatoeba, Europarl, JW300, OpenSubtitles2018, and UNPC. These datasets cover a wide range of languages and topics, allowing the authors to demonstrate the effectiveness of their approach across 50+ languages.

Experiments and Results

The authors conducted extensive experiments for tasks like multilingual and cross-lingual semantic textual similarity (STS), bitext retrieval, and cross-lingual similarity search. Below are key findings from their experiments:

1. Multilingual Semantic Textual Similarity (STS):
   • The authors tested their models on the multilingual STS 2017 dataset, comparing it to other systems like LASER, mUSE, and LaBSE.
   • Their approach using multilingual knowledge distillation showed superior performance, especially for cross-lingual setups, achieving higher Spearman rank correlations.
2. Bitext Retrieval:
   • The authors evaluated their models on the BUCC bitext retrieval task. Here, LASER and LaBSE performed better for exact translation retrieval.
   • However, the proposed method demonstrated state-of-the-art performance for tasks involving identifying semantically similar but not necessarily identical sentences.
3. Tatoeba Similarity Search:
   • The methodology was particularly effective for lower-resource languages, achieving significant accuracy improvements on the Tatoeba dataset.
   • For certain languages with limited parallel data, the authors’ approach achieved higher accuracies compared to other models.

Implications and Future Work

The implications of this research are twofold—practical and theoretical:

• Practical Implications:
  • The proposed method allows for the rapid extension of sentence embedding models to multiple languages without needing substantial parallel data for each new language.
  • This can significantly reduce the computational resources required for training multilingual models, making it feasible to support a wide array of languages.
• Theoretical Implications:
  • The research highlights the effectiveness of knowledge distillation in transferring the properties of monolingual vector spaces to multilingual setups.
  • It opens up new avenues for investigating how well various properties of vector spaces can be preserved across languages and tasks.

The paper also addresses the "curse of multilinguality," where adding more languages can degrade performance due to fixed model capacity. By decoupling the embedding training into a two-step process—monolingual embedding followed by multilingual extension—this approach minimizes the potential language bias, ensuring better alignment of vector spaces.

Conclusion

The method proposed by Reimers and Gurevych provides a robust framework for generating multilingual sentence embeddings through knowledge distillation. By decoupling the embedding creation process from multilingual training, it simplifies the extension of sentence embeddings to new languages while preserving the desired properties of the vector space. This method not only shows significant improvements across various benchmarks but also offers an efficient way to address the challenges associated with multilingual NLP applications.