Inverted Softmax in Bilingual Embedding Retrieval

• Inverted softmax is a retrieval rule that normalizes over source embeddings to counteract the hubness phenomenon in high-dimensional bilingual spaces.
• It leverages an orthogonal transformation obtained via SVD and dimensionality reduction to sharpen alignment between pre-trained monolingual embeddings.
• Empirical results show that using inverted softmax increases precision @1 for English-to-Italian translation from 34% to 43%, even with noisy pseudo-dictionaries.

Searching arXiv for the specified paper and closely related context. to=arxiv_search.search 天天中彩票派奖 аԥсjson_stringify code: {"query":"ti:\"Offline bilingual word vectors, orthogonal transformations and the inverted softmax\" OR id:(Smith et al., 2017)","max_results":5,"sort_by":"relevance"} to=arxiv_search.search 大发快三的 аԥсjson_stringify code: {"query":"all:\"inverted softmax\" bilingual word vectors hubness orthogonal transformations","max_results":10,"sort_by":"relevance"} Inverted softmax is a retrieval rule for bilingual embedding alignment introduced in "Offline bilingual word vectors, orthogonal transformations and the inverted softmax" (Smith et al., 2017). It is defined in the setting where two pre-trained embedding spaces are aligned "offline" by a linear transformation and translation pairs are recovered from cross-space similarity scores. In that framework, the method replaces conventional nearest-neighbour retrieval and the usual target-normalized softmax with a source-normalized reweighting that penalizes hubs, that is, target vectors that become nearest-neighbours to many sources. In the same work, inverted softmax is presented together with an orthogonal alignment obtained by singular value decomposition, dimensionality reduction by discarding small singular-value directions, and extensions from word translation to sentence retrieval (Smith et al., 2017).

1. Offline bilingual alignment and the orthogonality constraint

The paper situates inverted softmax within the "offline" construction of bilingual word vectors. Rather than training bilingual vectors "online", it aligns two pre-trained monolingual embeddings with a linear transformation using bilingual correspondences. The central claim is that the linear transformation between the two spaces should be orthogonal, and that this transformation can be obtained using the singular value decomposition (Smith et al., 2017).

Let $X \in \mathbb{R}^{V_s \times d}$ denote the source-language embedding matrix and $Y \in \mathbb{R}^{V_t \times d}$ the target-language embedding matrix, with row-normalized vectors satisfying $\|x_j\|=\|y_i\|=1$. Given a seed dictionary $\{(x_i,y_i)\}_{i=1}^n$, the orthogonal map $O \in \mathbb{R}^{d \times d}$ is defined by the constrained maximization

$\max_{O\,:\,O^TO=I}\sum_i y_i^T\,O\,x_i,$

which is equivalent to the Procrustes objective

$\min\sum_i\|y_i-Ox_i\|^2.$

The analytic solution is obtained by forming $M = Y_D^T X_D$, computing its singular value decomposition $M = U \Sigma V^T$, and setting

$O = U\,V^T.$

The stated justification for orthogonality is that it forces the inverse map $Y \in \mathbb{R}^{V_t \times d}$0 to be the transpose and preserves lengths and inner-products. The paper further states that orthogonal transformations are more robust to noise, a property that later supports learning from a noisy pseudo-dictionary built from identical character strings occurring in both languages (Smith et al., 2017).

2. Similarity geometry and the inverted-softmax score

After alignment, the cosine-similarity matrix is

$Y \in \mathbb{R}^{V_t \times d}$1

A naïve translation rule selects

$Y \in \mathbb{R}^{V_t \times d}$2

that is, nearest-neighbour retrieval.

The paper also considers a conditional model based on the usual softmax over target words,

$Y \in \mathbb{R}^{V_t \times d}$3

where $Y \in \mathbb{R}^{V_t \times d}$4 is a temperature learned by maximizing log-likelihood on a held-out bilingual dictionary. However, the text explicitly states that this "forward" softmax still suffers from hubness.

The inverted softmax reverses the normalization direction: $Y \in \mathbb{R}^{V_t \times d}$5 An equivalent expression given in the text is

$Y \in \mathbb{R}^{V_t \times d}$6

The stated intuition is that a hub $Y \in \mathbb{R}^{V_t \times d}$7 acquires a large denominator because it is similar to many source vectors $Y \in \mathbb{R}^{V_t \times d}$8, and is therefore down-weighted. This makes inverted softmax a retrieval mechanism specifically designed to counter the hubness phenomenon in high-dimensional bilingual matching (Smith et al., 2017).

3. Retrieval procedure and computational form

The paper presents the use of inverted softmax as an explicit algorithm once $Y \in \mathbb{R}^{V_t \times d}$9, $\|x_j\|=\|y_i\|=1$0, the orthogonal map $\|x_j\|=\|y_i\|=1$1, the temperature $\|x_j\|=\|y_i\|=1$2, and a sample size $\|x_j\|=\|y_i\|=1$3 are fixed (Smith et al., 2017).

First, the similarity matrix is precomputed: $\|x_j\|=\|y_i\|=1$4 Then, for each source index $\|x_j\|=\|y_i\|=1$5, a sampled or full set $\|x_j\|=\|y_i\|=1$6 of size $\|x_j\|=\|y_i\|=1$7 is selected. For each target candidate $\|x_j\|=\|y_i\|=1$8, the denominator

$\|x_j\|=\|y_i\|=1$9

is computed, followed by the inverted-softmax score

$\{(x_i,y_i)\}_{i=1}^n$0

and finally

$\{(x_i,y_i)\}_{i=1}^n$1

The corresponding pseudocode sketch in the paper is:

$O \in \mathbb{R}^{d \times d}$6

This formulation makes explicit that the scoring rule depends both on the source-specific numerator $\{(x_i,y_i)\}_{i=1}^n$2 and on a target-specific denominator aggregated over source vectors. A plausible implication is that inverted softmax is not merely a post hoc ranking heuristic but a structured renormalization of the cross-space similarity matrix.

4. Relation to nearest-neighbour retrieval, forward softmax, and hubness

The paper contrasts three retrieval rules. Nearest-neighbour retrieval selects $\{(x_i,y_i)\}_{i=1}^n$3. Forward softmax uses $\{(x_i,y_i)\}_{i=1}^n$4 normalized over targets. Inverted softmax normalizes over sources for each target, thereby penalizing targets that are similar to many source embeddings (Smith et al., 2017).

This comparison is central because the method is motivated not by changing the embedding geometry alone, but by changing the retrieval geometry induced by the same similarity matrix. The paper states that nearest-neighbour retrieval suffers badly from hubs in high-dimensional spaces, and that forward softmax does not fix hubness. By contrast, inverted softmax demotes hubs precisely because the denominator for a target grows when that target is broadly similar across the source vocabulary.

A common misconception, directly addressed by this contrast, is that any softmax-based probabilistic interpretation of similarity automatically corrects retrieval bias. The formulation in the paper rejects that view: only the inverted normalization, not the standard target-normalized softmax, is presented as hubness-mitigating. This suggests that the crucial design choice is the axis of normalization rather than the mere use of exponentiated scores.

5. Orthogonal Procrustes, denoising, and their interaction with inverted softmax

The paper treats the orthogonal transform and inverted softmax as complementary components. The singular values in $\{(x_i,y_i)\}_{i=1}^n$5 are said to rank "stable" directions, and dropping the smallest $\{(x_i,y_i)\}_{i=1}^n$6 singular-value directions often improves retrieval by denoising (Smith et al., 2017).

This dimensionality reduction modifies the similarity matrix $\{(x_i,y_i)\}_{i=1}^n$7 before retrieval. The paper explicitly states that the quality of this matrix determines both the top-$\{(x_i,y_i)\}_{i=1}^n$8 neighbours and the inverted-softmax probabilities. It further states that a better, orthogonal, and lower-dimensional alignment sharpens the peaks in $\{(x_i,y_i)\}_{i=1}^n$9, which the inverted softmax then leverages to demote hubs.

The interaction is therefore two-stage. Orthogonal Procrustes imposes a constrained geometry that preserves lengths and inner-products, while singular-value truncation removes unstable directions. Inverted softmax then operates on the resulting score landscape. A plausible implication is that the method is best understood as a coupled alignment-and-retrieval pipeline rather than as an isolated scoring function.

6. Reported results and extensions

The paper reports empirical gains on English-to-Italian bilingual retrieval and sentence retrieval, and these results are the principal evidence for the usefulness of inverted softmax in the proposed framework (Smith et al., 2017).

For an English$O \in \mathbb{R}^{d \times d}$0Italian experiment with a 5 k-word train set and a 1.5 k-word test set, the paper reports the following precision @1 figures:

Method	Precision @1
Nearest-neighbour on Mikolov’s (least-squares) map	34 %
+ inverted softmax (same map)	42 %
+ dimensionality reduction (drop small SVD singular-value directions)	43 %

In the abstract, this improvement is summarized as raising the precision @1 of Mikolov's original mapping from 34% to 43% when translating a test set composed of both common and rare English words into Italian.

The paper also introduces a pseudo-dictionary from identical strings. Instead of using a curated bilingual list, it extracts all tokens appearing identically in both source and target vocabularies, with English$O \in \mathbb{R}^{d \times d}$1Italian yielding approximately 47 k pairs. Learning $O \in \mathbb{R}^{d \times d}$2 on this noisy seed still gives 40 % @1, whereas the data block states 1 %–6 % using Mikolov/Dinu retrieval. In the abstract, this is summarized as achieving 40% precision on the same test set without expert bilingual signal.

A further extension uses Europarl for sentence-level alignment. Each sentence is represented by its normalized word-sum vector; aligned sentences are treated as a "phrase dictionary"; and the same SVD-based orthogonal alignment is applied. The paper reports bilingual word vectors that, when combined with inverted softmax, recover word translations @1 at approximately 43 % for En$O \in \mathbb{R}^{d \times d}$3It and approximately 38 % for It$O \in \mathbb{R}^{d \times d}$4En. It also reports sentence retrieval from a corpus of 200 k Italian sentences with approximately 68 % @1 accuracy, while the abstract states a precision @1 of 68% for retrieving the true translations of English sentences from a corpus of 200k Italian sentences.

Taken together, these results place inverted softmax as a re-normalization over source embeddings that combats hubness within an orthogonally aligned bilingual embedding space. The paper’s own summary is that, coupled with a provably-optimal, single-SVD orthogonal alignment and optional dimensionality reduction, it raises bilingual-mapping precision from approximately 34 % to approximately 43 % @1 on a standard English$O \in \mathbb{R}^{d \times d}$5Italian benchmark (Smith et al., 2017).