Offline bilingual word vectors, orthogonal transformations and the inverted softmax

Abstract: Usually bilingual word vectors are trained "online". Mikolov et al. showed they can also be found "offline", whereby two pre-trained embeddings are aligned with a linear transformation, using dictionaries compiled from expert knowledge. In this work, we prove that the linear transformation between two spaces should be orthogonal. This transformation can be obtained using the singular value decomposition. We introduce a novel "inverted softmax" for identifying translation pairs, with which we improve the precision @1 of Mikolov's original mapping from 34% to 43%, when translating a test set composed of both common and rare English words into Italian. Orthogonal transformations are more robust to noise, enabling us to learn the transformation without expert bilingual signal by constructing a "pseudo-dictionary" from the identical character strings which appear in both languages, achieving 40% precision on the same test set. Finally, we extend our method to retrieve the true translations of English sentences from a corpus of 200k Italian sentences with a precision @1 of 68%.

• The paper demonstrates that orthogonal transformations derived via SVD provide an optimal linear mapping between monolingual word vectors.
• It introduces an inverted softmax to counteract high-dimensional hubness, raising English-to-Italian translation precision from 34% to 43%.
• The approach leverages pseudo-dictionaries and large corpora to facilitate bilingual sentence retrieval with a precision @1 of 68%.

Analysis of Offline Bilingual Word Vectors, Orthogonal Transformations, and the Inverted Softmax

The paper explores approaches for generating bilingual word vectors through offline methods by leveraging pre-trained embeddings. The authors build upon the seminal work by Mikolov et al., introducing significant enhancements in aligning monolingual word vectors from different languages using orthogonal transformations and a novel inverted softmax mechanism to tackle the hubness problem. Such approaches contribute to the more robust identification of translation pairs and demonstrate impressive translation precision improvements.

Core Contributions

The paper presents several key contributions in this domain of study:

1. Orthogonal Transformations: The authors prove that the optimal linear transformation between vector spaces should be orthogonal. By leveraging the Singular Value Decomposition (SVD), they efficiently obtain these transformations, enabling the creation of a shared semantic space. This is a theoretical underpinning that unifies previous methods under a consistent framework.
2. Inverted Softmax Technique: Introducing the inverted softmax, the researchers address the hubness problem prevalent in high-dimensional vector spaces. By normalizing over source words instead of target words, they enhance the retrieval accuracy of translation pairs. This results in a substantial increase in translation precision, from 34% to 43% for English-to-Italian word translation.
3. Pseudo-Dictionaries: A method is introduced to construct pseudo-dictionaries using identical character strings found in different languages, enabling bilingual vector space learning without expert bilingual knowledge. This approach achieves 40% translation precision, underscoring the robustness of orthogonal transformations.
4. Bilingual Sentence Retrieval: Extending their methodology, the authors apply their transformations to retrieve sentence translations from substantial corpora, achieving remarkable precision. For example, identifying the correct translation of English sentences from 200k Italian sentences was achieved with a precision @1 of 68%.

Numerical Results and Implications

• Translation Precision: Utilizing the expert-crafted dictionary, the methods presented achieve significantly higher precision in translation tasks compared to previous methods. Particularly, they realize 43% precision for English-to-Italian word translation.
• Robustness: Experiments using pseudo-dictionaries reflect the method's adaptability and resilience, maintaining high translation precision in absence of expert bilingual input.
• Comparative Performance: The proposed method substantially outperforms Mikolov's original mapping, especially when leveraged through the inverted softmax and dimensionality reduction, highlighting its efficacy.

Practical and Theoretical Implications

The work has profound implications for both theoretical research and practical applications in the field of natural language processing, specifically in machine translation and cross-linguistic information retrieval. The notion that a simple orthogonal transformation, derivable through SVD, suffices for aligning semantic spaces, offers a compelling approach for future research in multilingual embeddings.

From a practical standpoint, the orthogonal transformation technique democratizes the creation of bilingual vectors by removing the dependency on extensive bilingual parallel corpora. This is particularly valuable for low-resource languages where such resources may not exist. Furthermore, the ability to efficiently translate sentences from large corpora without relying on word order adds practical value to translation systems in dynamic and multilingual environments.

Future Directions

The paper opens pathways for enhanced exploration in unsupervised and semi-supervised bilingual space learning. Future developments could investigate integrating advanced sentence embeddings for more syntactically and semantically rich translations. Additionally, exploring deeper connections with neural network-based approaches, such as leveraging transformers for improved context capture within the inverted softmax, could further refine translation outputs.

In conclusion, this work provides a robust framework for offline bilingual vector alignment, blending theoretical insights with practical innovations. It stands as a significant contribution to the domain, with substantial potential to influence future research directions in multilingual NLP tasks.