Unsupervised MUSE Embedding Alignment

• The paper introduces an unsupervised framework that leverages Gaussian embeddings and Wasserstein-Procrustes optimization to align word spaces for bilingual lexicon induction.
• It employs a stochastic mini-batch method that alternates between orthogonal mapping and transport matrix estimation to efficiently refine both means and covariances.
• Empirical results demonstrate consistent performance gains for closely related languages while exposing challenges in aligning distant language pairs due to mismatched uncertainty patterns.

Unsupervised MUSE Embedding Alignment refers to a set of techniques for aligning independently trained monolingual word embedding spaces across languages without the use of supervision, generally by solving a coupled assignment-and-orthogonal-mapping (Wasserstein-Procrustes) problem. MUSE-style alignment has evolved from focusing exclusively on point-vector spaces to more general settings, including probabilistic (distributional) embeddings using Gaussian representations. The core goal is to recover a linear map and word correspondences such that words with identical or analogous meanings from distinct vocabularies are mapped close to each other in a joint vector space, enabling downstream tasks such as bilingual lexicon induction and transfer learning. Recent research advances include stochastic mini-batch optimization, explicit matching of distributional parameters, and empirical demonstrations of improved performance on cross-lingual tasks (Diallo et al., 2022).

1. Representation of Distributional Word Embeddings

Traditional unsupervised alignment methods operate on point-vector embeddings—each word $w$ is associated with a single vector $\mathbf{v}_w \in \mathbb{R}^d$ (e.g., skip-gram, GloVe, FastText). While convenient, this representation is limited in its ability to capture uncertainty, semantic asymmetry, or hierarchical relations.

Probabilistic (distributional) embeddings generalize this by associating with each word a multivariate Gaussian distribution: $$p_w(x) = \mathcal{N}(x; \boldsymbol{\mu}_w, \Sigma_w)$$ where $\boldsymbol{\mu}_w$ is the mean and $\Sigma_w$ is the covariance (often diagonal), encoding not just a location in the vector space but also the dispersion or semantic uncertainty. This representation enables the model to express the generality (broad/narrowness) of concepts and capture entailment patterns, as non-degenerate covariance matrices allow differentiation between, e.g., "animal" (broad) and "Labrador" (narrow) (Diallo et al., 2022).

2. Unsupervised Alignment Objective

Given monolingual Gaussian embeddings for a source and target language—$\{ (\mu^x_i, \Sigma^x_i) \}_{i=1}^n$ and $\{ (\mu^y_j, \Sigma^y_j) \}_{j=1}^m$—the alignment goal is to find an orthogonal mapping $Q \in \mathbb{R}^{d \times d}$ ($Q^\top Q=I$) and a permutation or transport matrix $P$ (often relaxed to a doubly-stochastic matrix) that aligns the two sets in a joint space.

The joint loss function is: $$\min_{Q \in \mathcal{O}_d,\,P \in \mathcal{P}_{n \times m}} \, \left\| M_x Q - P M_y \right\|_F^2 + \left\| \Sigma_x - P \Sigma_y \right\|_F^2$$ where $M_x$ and $M_y$ stack the means, and $\Sigma_x$ and $\Sigma_y$ stack the covariances. The first term performs Wasserstein-Procrustes alignment on the means; the second regularizes the assignment to encourage matches with similar semantic dispersion. The optimal $Q$ for known $P$ is given by the orthogonal Procrustes solution via SVD of $(M_x)^\top (P M_y)$ (Diallo et al., 2022).

3. Stochastic Mini-batch Optimization Scheme

Full-batch optimization is intractable for large vocabularies due to $O(nm)$ complexity. Instead, a stochastic mini-batch alternating approach is used:

1. Initialize $Q_0$ (identity) and $P_0$ by solving means-Procrustes on a small batch.
2. For each iteration $t$:
   • Sample batches of $b$ source and target Gaussians.
   • Compute batch transport $P_t$ via a linear program (Sinkhorn solver, $\varepsilon \approx 0.05$).
   • Compute gradient $G_t = -2(M_x^{(t)})^\top P_t M_y^{(t)}$ and update $Q$ with step-size $\alpha$.
   • Project $Q$ back to $\mathcal{O}_d$ by SVD.
   • Nested refinement: match and refine over covariances for $L \ll T$ inner steps. Key settings include $T \sim 5000$ iterations, initial $b = 500$ (doubling per epoch), $L = 2$, learning rate $\alpha \sim 0.1$, and covariance refinement with a lower learning rate (Diallo et al., 2022).

4. Evaluation Procedure and Empirical Results

Evaluation is based on bilingual lexicon induction: training on the top $20\,000$ frequent words per language (vocab $\sim 210\,000$), using precisely aligned FastText means and Vilnis-style variances from Wikipedia. The metric is Precision@1 (P@1), measured by nearest-neighbor search on the aligned means; incorporating covariance-aware distances during inference yields negligible gain.

A synopsis of main results (P@1, mean-only baseline vs. mean + covariance):

Language Pair	Baseline	+Covariance
en→fr	68.4	70.5
fr→en	69.3	71.8
en→es	67.4	70.8
es→en	71.3	73.2
en→de	62.1	64.1
de→en	59.8	60.1
en→ru	33.4	29.6
ru→en	49.7	41.2

In closely related languages (English–French, –Spanish, –German), dispersion matching yields a consistent 2–3 point increase in P@1. For distant pairs (English–Russian), the covariance term degrades performance, likely due to non-isomorphic structures and misaligned uncertainty patterns between the vocabularies (Diallo et al., 2022).

5. Theoretical and Practical Considerations

Probabilistic alignment confers several advantages:

• Richer geometric modeling: Covariances encode specificity and generalization, functioning as a second-order signal for more robust and meaningful matching.
• Regularization: Encouraging matched words to have similar uncertainty or dispersion stabilizes the transport plan and mitigates overfitting to mean-distance alignments.
• Entailment modeling: The volume of the Gaussian naturally supports hierarchical representation.

However, practical limitations exist:

• Increased computational overhead: Nested covariance refinement steps impose modest but nontrivial extra computation.
• Sensitivity to cross-linguistic and domain divergence: Forcing covariance alignment across typologically or domainwise disparate corpora can be detrimental, as variance patterns may not correspond meaningfully.
• Potential remedies: Incorporation of a seed lexicon, modeling full (non-diagonal) covariances, or learning an adaptive weighting between mean and covariance losses may address these issues (Diallo et al., 2022).

6. Broader Context and Implications

Unsupervised MUSE-style probabilistic alignment sits at the intersection of distribution-matching (e.g., Wasserstein-Procrustes optimizations), optimal transport, and cross-lingual semantic modeling. The explicit extension to Gaussian embeddings is a substantive generalization from the earlier point-vector paradigm, allowing the alignment process to exploit uncertainty and higher-order semantic structure.

The demonstrated empirical gains on related languages support the value of this extension, while the observed degradation on distant language pairs highlights an important challenge for future work: how to robustly align distributions when the isomorphism and semantic specificity assumptions break down.

The methodology proposed for aligning distributional word embeddings provides a template for further exploration in multilingual word alignment, entailment-aware representation transfer, and uncertainty modeling in natural language semantics (Diallo et al., 2022).