Unsupervised Alignment of Embeddings with Wasserstein Procrustes (1805.11222v1)

Abstract: We consider the task of aligning two sets of points in high dimension, which has many applications in natural language processing and computer vision. As an example, it was recently shown that it is possible to infer a bilingual lexicon, without supervised data, by aligning word embeddings trained on monolingual data. These recent advances are based on adversarial training to learn the mapping between the two embeddings. In this paper, we propose to use an alternative formulation, based on the joint estimation of an orthogonal matrix and a permutation matrix. While this problem is not convex, we propose to initialize our optimization algorithm by using a convex relaxation, traditionally considered for the graph isomorphism problem. We propose a stochastic algorithm to minimize our cost function on large scale problems. Finally, we evaluate our method on the problem of unsupervised word translation, by aligning word embeddings trained on monolingual data. On this task, our method obtains state of the art results, while requiring less computational resources than competing approaches.

Unsupervised Alignment of Embeddings with Wasserstein Procrustes

The paper "Unsupervised Alignment of Embeddings with Wasserstein Procrustes" tackles the longstanding problem of aligning two sets of high-dimensional point embeddings, a task of great relevance across various domains in machine learning, particularly NLP and computer vision. This work proposes a novel approach grounded in joint optimization of orthogonal and permutation matrices to achieve unsupervised alignment, utilizing a combination of the Wasserstein distance and the Procrustes analysis framework.

Overview and Approach

The primary objective of the research is to address the challenge of aligning two sets of embeddings without the need for supervised data. Specifically, the paper focuses on unsupervised word translation by aligning monolingual word embeddings. The problem is formulated as the optimization of an orthogonal matrix $\mathbf{Q}$ and a permutation matrix $\mathbf{P}$, which together define a linear transformation and matching between the point sets. This formulation is inherently non-convex, necessitating innovative strategies for reliable solution finding.

The authors introduce a stochastic optimization method to minimize a cost function designed for large-scale problems efficiently. The key contribution here is the introduction of a convex relaxation technique to initialize the optimization, borrowing insights from the graph isomorphism problem and applying principles from the Wasserstein distance for measuring proximity between sets.

Numerical Results

The paper provides a rigorous evaluation of the proposed method on the unsupervised word translation task. Involving multiple language pairs, the paper demonstrates that the approach yields state-of-the-art results. The evaluated language pairs include Spanish-English, French-English, and German-English, among others. The precision-at-one metric, a measure of accurate translation pairs, indicates that the Wasserstein Procrustes method performs competitively compared to existing unsupervised methods, such as adversarial learning frameworks and iterative closest point algorithms. Notably, it achieves these results while being computationally efficient, outperforming or closely matching competing methods with fewer computational resources.

Theoretical and Practical Implications

From a theoretical standpoint, the research draws an intriguing parallel between the tasks of graph matching and embedding alignment, suggesting potential for cross-pollination of techniques between these areas. The introduction of a convex relaxation for initializing non-convex optimization offers a valuable tool for tackling complex alignment problems, which may extend to broader applications beyond language translation.

Practically, the ability to align embeddings without explicit supervision holds promise for various applications, including cross-lingual information retrieval, image registration in computer vision, and more generalized data alignment tasks in machine learning workflows. The reduced computational burden of the method further enhances its applicability in large-scale industrial settings, opening the door for efficient integration into existing NLP pipelines.

Future Directions

While the presented method showcases robust performance, the paper motivates several future research directions. Firstly, deeper exploration into the theoretical guarantees and limitations of the convex relaxation technique can solidify its place in solving alignment problems. Additionally, identifying specific conditions under which the proposed method outperforms existing techniques can refine its practical applicability. Finally, evolving the algorithm to adapt to dynamic datasets in real-time scenarios could further extend its utility in fast-paced, data-driven environments.

In conclusion, "Unsupervised Alignment of Embeddings with Wasserstein Procrustes" offers significant advancements in the domain of unsupervised learning, demonstrating both theoretical innovation and practical effectiveness. The approach not only addresses current limitations but also paves the way for future research endeavors in high-dimensional data alignment.