Language-agnostic BERT Sentence Embedding

Abstract: While BERT is an effective method for learning monolingual sentence embeddings for semantic similarity and embedding based transfer learning (Reimers and Gurevych, 2019), BERT based cross-lingual sentence embeddings have yet to be explored. We systematically investigate methods for learning multilingual sentence embeddings by combining the best methods for learning monolingual and cross-lingual representations including: masked language modeling (MLM), translation language modeling (TLM) (Conneau and Lample, 2019), dual encoder translation ranking (Guo et al., 2018), and additive margin softmax (Yang et al., 2019a). We show that introducing a pre-trained multilingual LLM dramatically reduces the amount of parallel training data required to achieve good performance by 80%. Composing the best of these methods produces a model that achieves 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba, well above the 65.5% achieved by Artetxe and Schwenk (2019b), while still performing competitively on monolingual transfer learning benchmarks (Conneau and Kiela, 2018). Parallel data mined from CommonCrawl using our best model is shown to train competitive NMT models for en-zh and en-de. We publicly release our best multilingual sentence embedding model for 109+ languages at https://tfhub.dev/google/LaBSE.

• The paper introduces language-agnostic sentence embeddings via dual encoder models and additive margin softmax, achieving 83.7% accuracy for bi-text retrieval.
• It leverages a combination of masked language modeling and translation language modeling to minimize required parallel data while ensuring robust cross-lingual performance.
• The released LaBSE model enhances multilingual applications such as NMT and translation pair mining, demonstrating competitive performance in low-resource settings.

Language-Agnostic BERT Sentence Embedding

Introduction

The paper "Language-agnostic BERT Sentence Embedding" (2007.01852) investigates efficient methods for learning multilingual sentence embeddings using a combination of monolingual and cross-lingual approaches. Key techniques explored include masked language modeling (MLM), translation language modeling (TLM), dual encoder translation ranking, and additive margin softmax. Leveraging a pre-trained multilingual LLM significantly reduces the necessary parallel data for achieving high performance. The paper presents a model that achieves 83.7% accuracy on bi-text retrieval across 112 languages, surpassing the previous benchmark of 65.5%. This model also performs competitively on monolingual transfer learning tasks.

Figure 1: Dual encoder model with BERT based encoding modules.

Methodology

Cross-lingual Sentence Embeddings

The foundation of this study is the use of dual encoder models, which implement paired encoding modules to generate bilingual sentence embeddings. By employing pre-trained LLMs rather than training from scratch, these models sidestep the extensive parallel data requirement that was characteristic of previous methods. The central loss function used, additive margin softmax, enhances the separation between translations and non-translations, significantly improving retrieval accuracy.

Figure 2: Negative sampling example in a dual encoder framework. [Left]: The in-batch negative sampling in a single core; [Right]: Synchronized multi-accelerator negative sampling using n TPU cores and batch size 8 per core with examples from other cores are all treated as negatives.

Pre-training Strategies

Combining MLM and TLM, the approach utilizes a large multilingual pre-trained model to craft strong multilingual sentence embeddings. MLM, a variant of the cloze task, predicts words masked in a sentence using surrounding context. TLM adapts this to multilingual sentences by concatenating translation pairs, enabling cross-language embedding learning. These embeddings have proven to be highly effective, equating to the performance on large bitext retrieval tasks like the UN corpus and BUCC.

Experimental Results

The dual encoder models were benchmarked on extensive multilingual datasets, highlighting their performance, particularly on languages without explicit training data. In Tatoeba evaluations, models achieved impressive accuracy of 83.7% across 112 languages, demonstrating robustness in low-resource settings. Comprehensive testing revealed that while additive margin softmax consistently improved cross-lingual embedding quality, pre-training also reduced the need for large volumes of parallel data.

Figure 3: Average P@1~(\%) on UN retrieval task of models trained with different margin values.

Practical Implications

The LaBSE model is publicly available for use in multilingual applications. Its capacity to mine high-quality translation pairs from extensive web corpora, such as CommonCrawl, underscores its practical utility. Experiments show that the parallel data mined using LaBSE enhances the training of NMT models, bridging gaps for languages with minimal existing resources, while maintaining competitive translation quality.

Conclusion

The thorough exploration outlined in "Language-agnostic BERT Sentence Embedding" provides a methodical approach to constructing high-quality cross-lingual embeddings across diverse languages. Key contributions include reduced reliance on extensive parallel corpora, competitive performance on translation tasks, and efficient pre-training strategies. The release of LaBSE empowers researchers and developers to leverage these advancements in multilingual natural language processing contexts.