Word Translation Without Parallel Data (1710.04087v3)

Abstract: State-of-the-art methods for learning cross-lingual word embeddings have relied on bilingual dictionaries or parallel corpora. Recent studies showed that the need for parallel data supervision can be alleviated with character-level information. While these methods showed encouraging results, they are not on par with their supervised counterparts and are limited to pairs of languages sharing a common alphabet. In this work, we show that we can build a bilingual dictionary between two languages without using any parallel corpora, by aligning monolingual word embedding spaces in an unsupervised way. Without using any character information, our model even outperforms existing supervised methods on cross-lingual tasks for some language pairs. Our experiments demonstrate that our method works very well also for distant language pairs, like English-Russian or English-Chinese. We finally describe experiments on the English-Esperanto low-resource language pair, on which there only exists a limited amount of parallel data, to show the potential impact of our method in fully unsupervised machine translation. Our code, embeddings and dictionaries are publicly available.

Word Translation Without Parallel Data: A Detailed Overview

The paper "Word Translation Without Parallel Data" by Alexis Conneau et al. introduces a novel approach for learning cross-lingual word embeddings without the requirement of parallel corpora or bilingual dictionaries. Traditional methods for cross-lingual alignment typically rely on bilingual lexicons or parallel datasets. These methods face limitations, particularly for low-resource languages or language pairs without shared alphabets. The research presented in this paper addresses these challenges by aligning monolingual word embedding spaces using unsupervised methods.

Methodology

The core methodology employed in this research involves adversarial training and the application of the Procrustes refinement. The process is divided into several key steps:

1. Initial Mapping via Adversarial Training: The initial transformation matrix $W$ is obtained by training the system in a two-player adversarial game. A discriminator attempts to differentiate between the aligned source language embeddings $W \mathbf{X}$ and the target language embeddings $\mathbf{Y}$, while $W$ is trained to make $W \mathbf{X}$ indistinguishable from $\mathbf{Y}$.
2. Procrustes Refinement: The rough alignment obtained from adversarial training serves as a basis for further refinement. Using the most frequent words as anchor points ensures a more accurate linear mapping $W$. This step leverages the closed-form Procrustes solution to realign thesource and target embedding spaces more precisely.
3. Cross-Domain Similarity Local Scaling (CSLS): To address the hubness problem in high-dimensional spaces, the authors introduce CSLS as a method to scale the similarity measure. This approach refines the neighborhood graph's similarities, enhancing the retrieval accuracy for word translation tasks.
4. Unsupervised Validation Criterion: Selection of the best model in an unsupervised context is non-trivial. The paper proposes a validation criterion based on the average cosine similarity of deemed translations, ensuring effective model selection and hyper-parameter tuning.

Empirical Results

The researchers evaluate their method across various tasks and language pairs, producing notable results:

• Word Translation Retrieval: The method shows competitive performance with supervised approaches, sometimes even surpassing them. For instance, on English-Italian (en-it), the proposed unsupervised method achieves a precision of 66.2%, outperforming the best supervised approach which scores 63.7%.
• Sentence Translation Retrieval: The refined embeddings significantly improve performance in sentence retrieval tasks. The precision for retrieving English sentences using Italian queries (English-Italian direction) increased from 53.5% to 69.5% using CSLS.
• Cross-Lingual Word Similarity: On the SemEval 2017 dataset, the proposed method exhibits strong correlations with human judgments of word similarity, further validating the quality of the embeddings.

Implications and Future Directions

The implications of this research are substantial, particularly for low-resource languages. The ability to build high-quality bilingual dictionaries without the need for parallel corpora or extensive lexicons opens new avenues for machine translation and other cross-lingual NLP applications. Additionally, the robust performance on language pairs without shared alphabets (e.g., English-Chinese) demonstrates the method's versatility.

Future developments could explore more sophisticated adversarial techniques and consider joint training of embeddings to further enhance alignment quality. Another promising direction is integrating LLMing techniques to refine translations at the sentence level, reducing errors seen in word-by-word translation examples such as in the analysis of English-Esperanto pairs.

Conclusion

This paper presents a significant advance in the field of cross-lingual word embeddings by demonstrating that unsupervised techniques can achieve, and sometimes surpass, the performance of supervised methods. The innovations in adversarial training, Procrustes refinement, and CSLS collectively contribute to a robust framework for word translation without parallel data. The release of high-quality dictionaries and embeddings as part of this research offers valuable resources for the NLP community, supporting further exploration and development in multilingual contexts.

References

The research builds on foundational work in the area of word embeddings and bilingual lexicon induction, citing important contributions from Mikolov et al., Smith et al., Artetxe, and others. The use of fastText embeddings and references to statistical decipherment align this research with broader efforts to harness the distributional hypothesis for multilingual applications. The implementation details and comprehensive empirical validation underscore the practical viability of the proposed approach.

In conclusion, "Word Translation Without Parallel Data" illustrates a significant step forward in leveraging unsupervised methods for cross-lingual NLP, with extensive implications for future research and application development in multilingual machine learning.