Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
8 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Harnessing the Universal Geometry of Embeddings (2505.12540v3)

Published 18 May 2025 in cs.LG

Abstract: We introduce the first method for translating text embeddings from one vector space to another without any paired data, encoders, or predefined sets of matches. Our unsupervised approach translates any embedding to and from a universal latent representation (i.e., a universal semantic structure conjectured by the Platonic Representation Hypothesis). Our translations achieve high cosine similarity across model pairs with different architectures, parameter counts, and training datasets. The ability to translate unknown embeddings into a different space while preserving their geometry has serious implications for the security of vector databases. An adversary with access only to embedding vectors can extract sensitive information about the underlying documents, sufficient for classification and attribute inference.

Summary

  • The paper introduces vec2vec, a novel technique that maps text embeddings to a shared latent space without requiring paired data.
  • It details a modular architecture using MLPs, residual connections, and adversarial training to ensure robust translation between embedding spaces.
  • The study highlights security implications by demonstrating how translated embeddings can expose sensitive information through attribute inference and inversion.

This paper introduces "vec2vec" (2505.12540), a novel method for translating text embeddings between different vector spaces without requiring any paired data, access to the original encoders, or predefined sets of matches. The core idea is based on the "Strong Platonic Representation Hypothesis," which conjectures that neural networks trained with similar objectives and modalities, even with different architectures and data, converge to a universal latent semantic structure. The vec2vec method learns to map embeddings to and from this universal latent space, enabling unsupervised translation.

The primary application demonstrated is enhancing the interoperability of embedding models. For instance, if you have a vector database generated by an unknown model (M1M_1) and wish to use tools or perform analysis that requires embeddings from a known model (M2M_2), vec2vec can translate the M1M_1 embeddings into M2M_2's space. This translation aims to preserve the semantic geometry, meaning that documents close in M1M_1's space should have their translated embeddings close in M2M_2's space, and ideally, the translated embedding F(M1(di))F(M_1(d_i)) should be very close to the "true" embedding M2(di)M_2(d_i).

A significant implication highlighted is the security of vector databases. If an adversary gains access to a dump of embedding vectors, even without knowing the model that generated them, they can use vec2vec to translate these embeddings into a space of a known model. Once translated, they can apply techniques like attribute inference or embedding inversion to extract sensitive information from the original documents.

vec2vec Method: Architecture and Optimization

The vec2vec architecture is modular and employs Multilayer Perceptrons (MLPs) with residual connections, layer normalization, and SiLU nonlinearities. It consists of:

  1. Input Adapters (A1,A2A_1, A_2): These modules transform embeddings from their specific source spaces (RdM1R^{d_{M_1}} or RdM2R^{d_{M_2}}) into a shared universal latent representation of dimension ZZ. So, A1:RdM1RZA_1: \mathbb{R}^{d_{M_1}} \to \mathbb{R}^{Z} and A2:RdM2RZA_2: \mathbb{R}^{d_{M_2}} \to \mathbb{R}^{Z}.
  2. Shared Backbone (TT): This network takes the adapted embeddings and extracts a common latent embedding, mapping RZRZ\mathbb{R}^{Z} \to \mathbb{R}^{Z}.
  3. Output Adapters (B1,B2B_1, B_2): These modules translate the common latent embeddings back into the specific encoder spaces. So, B1:RZRdM1B_1: \mathbb{R}^{Z} \to \mathbb{R}^{d_{M_1}} and B2:RZRdM2B_2: \mathbb{R}^{Z} \to \mathbb{R}^{d_{M_2}}.

Using these components, two main types of functions are defined:

  • Translation functions:
    • F1=B2TA1F_1 = B_2 \circ T \circ A_1: Translates from M1M_1's space to M2M_2's space.
    • F2=B1TA2F_2 = B_1 \circ T \circ A_2: Translates from M2M_2's space to M1M_1's space.
  • Reconstruction mappings:
    • R1=B1TA1R_1 = B_1 \circ T \circ A_1: Encodes from M1M_1's space to latent and decodes back to M1M_1's space.
    • R2=B2TA2R_2 = B_2 \circ T \circ A_2: Encodes from M2M_2's space to latent and decodes back to M2M_2's space.

The optimization process is adversarial, involving generator networks (the adapters and backbone) and discriminator networks. Discriminators (D1,D2D_1, D_2) operate on the output embeddings, and D1,D2D_1^\ell, D_2^\ell operate on the latent representations.

The overall loss function combines adversarial losses and generator-specific constraints:

θ=argminθmaxD1,D2,D1,D2Ladv+λgenLgen\theta^* = \arg \min_{\theta} \max_{D_1, D_2, D_{1}^{\ell}, D_{2}^{\ell}} \mathcal{L}_{\text{adv}} + \lambda_{\text{gen}} \mathcal{L}_{\text{gen}}

  1. Adversarial Loss (Ladv\mathcal{L}_{\text{adv}}): This loss encourages the translated embeddings (and their latent representations) to be indistinguishable from real embeddings (and their latent representations) in the target space. It uses a standard GAN loss formulation: Ladv=LGAN(D1,F1)+LGAN(D2,F2)+LGAN(D1,TA1)+LGAN(D2,TA2)\mathcal{L}_{\text{adv}} = \mathcal{L}_{\text{GAN}}(D_1, F_1) + \mathcal{L}_{\text{GAN}}(D_2, F_2) + \mathcal{L}_{\text{GAN}}(D_{1}^{\ell}, T \circ A_1) + \mathcal{L}_{\text{GAN}}(D_{2}^{\ell}, T \circ A_2).
  2. Generator Loss (Lgen\mathcal{L}_{\text{gen}}): This comprises three components:
    • Reconstruction Loss (Lrec\mathcal{L}_{\text{rec}}): Ensures that an embedding, when mapped to the latent space and back to its original embedding space, remains close to the initial representation. Lrec=ExpR1(x)x22+EyqR2(y)y22\mathcal{L}_{\text{rec}} = \mathbb{E}_{x \sim p}\|R_1(x)-x\|_2^2 + \mathbb{E}_{y \sim q}\|R_2(y)-y\|_2^2, where xx are embeddings from M1M_1 and yy are from M2M_2.
    • Cycle-Consistency Loss (LCC\mathcal{L}_{\text{CC}}): Ensures that translating an embedding to the other space and back again results in minimal change. LCC=ExpF2(F1(x))x22+EyqF1(F2(y))y22\mathcal{L}_{\text{CC}} = \mathbb{E}_{x \sim p}\|F_2(F_1(x))-x\|_2^2 + \mathbb{E}_{y \sim q}\|F_1(F_2(y))-y\|_2^2.
    • Vector Space Preservation Loss (LVSP\mathcal{L}_{\text{VSP}}): Encourages the preservation of pairwise relationships (dot products) between embeddings after translation. For a batch of BB embeddings xix_i from M1M_1 and yiy_i from M2M_2: LVSP=1Bi=1Bj=1B[M1(xi)M1(xj)F1(M1(xi))F1(M1(xj))22+M2(yi)M2(yj)F2(M2(yi))F2(M2(yj))22]\mathcal{L}_{\text{VSP}} = \frac{1}{B} \sum_{i=1}^{B} \sum_{j=1}^{B} \left[ \|M_1(x_i) \cdot M_1(x_j) - F_1(M_1(x_i)) \cdot F_1(M_1(x_j))\|_2^2 + \|M_2(y_i) \cdot M_2(y_j) - F_2(M_2(y_i)) \cdot F_2(M_2(y_j))\|_2^2 \right]. The combined generator loss is Lgen=λrecLrec+λCCLCC+λVSPLVSP\mathcal{L}_{\text{gen}} = \lambda_{\text{rec}}\mathcal{L}_{\text{rec}} + \lambda_{\text{CC}}\mathcal{L}_{\text{CC}} + \lambda_{\text{VSP}}\mathcal{L}_{\text{VSP}}.

Implementation and Training

  • Network Architecture: MLPs with residual connections, LayerNorm, and SiLU activations. Discriminators are similar but without residual connections.
  • Training Data: The method is trained using unpaired sets of embeddings. For example, 1 million embeddings from model M1M_1 (e.g., GTR) generated from one set of NQ documents, and 1 million embeddings from model M2M_2 (e.g., GTE) generated from a disjoint set of NQ documents. The paper shows that even 50,000 embeddings can yield good results (Table A.5).
  • Training Process: Due to GAN instability, selecting the best model from multiple initializations is recommended.
  • Computational Cost: Training a single vec2vec model pair took 1-7 days on GPUs like NVIDIA 2080Ti, L4, A40, or A100.

Practical Application: Implementing vec2vec

To implement vec2vec for translating embeddings from an unknown source model M1M_1 to a known target model M2M_2:

  1. Collect Unpaired Embeddings:
    • Obtain a large set of embeddings {ui}\{u_i\} from the source M1M_1. These could be from a compromised vector database.
    • Generate a large set of embeddings {vj}\{v_j\} using your known model M2M_2 from a corpus that is distributionally similar (e.g., same language, general domain) to the source documents of {ui}\{u_i\}. Crucially, {ui}\{u_i\} and {vj}\{v_j\} do not need to correspond to the same documents.
  2. Define vec2vec Model Components:
    • Implement A1,A2,T,B1,B2A_1, A_2, T, B_1, B_2 as MLPs. The paper suggests MLPs with residual connections, LayerNorm, and SiLU. The latent dimension ZZ is a hyperparameter.
    • Implement discriminators D1,D2,D1,D2D_1, D_2, D_1^\ell, D_2^\ell as MLPs.
  3. Train the Model:
    • Initialize the networks.
    • Iteratively update the generator parameters (θ={A1,A2,T,B1,B2}\theta = \{A_1, A_2, T, B_1, B_2\}) and discriminator parameters using the adversarial and generator losses described above.
    • You'll need to choose hyperparameters λgen,λrec,λCC,λVSP\lambda_{\text{gen}}, \lambda_{\text{rec}}, \lambda_{\text{CC}}, \lambda_{\text{VSP}}.
    • The paper used Adam optimizer.
    • Save model checkpoints regularly and select the best performing one based on a validation set or heuristics related to loss convergence, as GAN training can be unstable.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
# Pseudocode for a training step
# x_m1: batch of embeddings from M1
# x_m2: batch of embeddings from M2

# --- Train Discriminators ---
# Latent discriminators
z_m1_real = T(A1(x_m1))
z_m2_real = T(A2(x_m2))
# (No explicit fake latent samples for D_latent, it operates on generator's latent output)

# Output discriminators
x_m1_to_m2_fake = F1(x_m1) # B2(T(A1(x_m1)))
x_m2_to_m1_fake = F2(x_m2) # B1(T(A2(x_m2)))

# Calculate discriminator losses (e.g., using BCEWithLogitsLoss)
# d_loss_latent_m1 = loss_gan_D(D_latent_m1, z_m1_real.detach(), T(A2(x_m2)).detach()) # Comparing T(A1(x)) vs T(A2(y))
# d_loss_latent_m2 = loss_gan_D(D_latent_m2, z_m2_real.detach(), T(A1(x_m1)).detach())
# d_loss_output_m1 = loss_gan_D(D1, x_m1_real, x_m2_to_m1_fake.detach()) # Comparing M1 real vs M2->M1 fake
# d_loss_output_m2 = loss_gan_D(D2, x_m2_real, x_m1_to_m2_fake.detach()) # Comparing M2 real vs M1->M2 fake
# d_loss = d_loss_latent_m1 + d_loss_latent_m2 + d_loss_output_m1 + d_loss_output_m2
# Optimize discriminators

# --- Train Generators ---
# Adversarial losses for generator
# g_loss_adv_latent_m1 = loss_gan_G(D_latent_m1, T(A2(x_m2))) # Fool D_latent_m1 with T(A2(y))
# g_loss_adv_latent_m2 = loss_gan_G(D_latent_m2, T(A1(x_m1))) # Fool D_latent_m2 with T(A1(x))
# g_loss_adv_output_m1 = loss_gan_G(D1, x_m2_to_m1_fake)     # Fool D1 with M2->M1 fake
# g_loss_adv_output_m2 = loss_gan_G(D2, x_m1_to_m2_fake)     # Fool D2 with M1->M2 fake
# g_loss_adv = g_loss_adv_latent_m1 + g_loss_adv_latent_m2 + g_loss_adv_output_m1 + g_loss_adv_output_m2

# Reconstruction losses
x_m1_reconstructed = R1(x_m1) # B1(T(A1(x_m1)))
x_m2_reconstructed = R2(x_m2) # B2(T(A2(x_m2)))
loss_rec = mse_loss(x_m1_reconstructed, x_m1) + mse_loss(x_m2_reconstructed, x_m2)

# Cycle-consistency losses
x_m1_cycled = F2(F1(x_m1)) # B1(T(A2( B2(T(A1(x_m1))) )))
x_m2_cycled = F1(F2(x_m2)) # B2(T(A1( B1(T(A2(x_m2))) )))
loss_cc = mse_loss(x_m1_cycled, x_m1) + mse_loss(x_m2_cycled, x_m2)

# Vector Space Preservation Loss (simplified conceptual calculation)
# pairwise_dots_m1_orig = calculate_pairwise_dots(x_m1)
# pairwise_dots_m1_translated = calculate_pairwise_dots(F1(x_m1))
# loss_vsp_m1 = mse_loss(pairwise_dots_m1_translated, pairwise_dots_m1_orig)
# (Similarly for x_m2 and F2)
# loss_vsp = loss_vsp_m1 + loss_vsp_m2

# g_loss_total = g_loss_adv + lambda_rec * loss_rec + lambda_cc * loss_cc + lambda_vsp * loss_vsp
# Optimize generators

  1. Translate and Utilize Embeddings:
    • Once trained, use the F1=B2TA1F_1 = B_2 \circ T \circ A_1 function to translate the source embeddings {ui}\{u_i\} into M2M_2's space: vi=F1(ui)v'_i = F_1(u_i).
    • These translated embeddings viv'_i can then be used with tools designed for M2M_2, for similarity search within M2M_2's space, or for information extraction attacks.

Experimental Results and Performance

  • Translation Quality: vec2vec achieves high cosine similarity (up to 0.92) between translated embeddings and ground-truth target embeddings (Table 2). Top-1 accuracy (correctly matching an embedding to its true counterpart in a large set) reaches up to 100% for some model pairs. It significantly outperforms naïve baselines (no translation) and oracle-aided optimal assignment (which assumes candidate targets are known) in cross-backbone model pairs.
  • Robustness: The method shows good performance on out-of-distribution datasets (e.g., trained on NQ from Wikipedia, tested on tweets from TweetTopic or medical records from MIMIC, see Table 3, Table A.2, A.3).
  • Cross-Modality: vec2vec can translate between unimodal text embeddings and multimodal embeddings like CLIP (Table 4), albeit with slightly lower performance than text-to-text, but still outperforming baselines.
  • Information Extraction:
    • Attribute Inference: Translated embeddings allow for effective zero-shot attribute inference (e.g., inferring topics of tweets or diseases from medical records). Performance often surpasses naïve baselines and sometimes even an oracle baseline that uses true embeddings in the target space (Table 5).
    • Embedding Inversion: Translated embeddings retain enough semantic information for off-the-shelf zero-shot inversion models to reconstruct parts of the original text (e.g., names, topics from Enron emails, see Figure 5, Figure 6). Accuracy varied, reaching up to 80% for some pairs.

Limitations and Considerations

  • GAN Training Instability: A key limitation is the inherent instability of GAN training. The authors mitigate this by training multiple initializations and selecting the best. More robust training procedures are future work.
  • Computational Resources: Training is computationally intensive.
  • Hyperparameter Tuning: The various loss weights (λrec,λCC,λVSP\lambda_{\text{rec}}, \lambda_{\text{CC}}, \lambda_{\text{VSP}}) and architectural choices (MLP depth/width, latent dimension ZZ) are hyperparameters that may require tuning for optimal performance on new pairs of embedding models. The ablation paper (Table A.4) shows that removing any of the loss components (VSP, CC, latent GAN) or skipping hyperparameter tuning degrades performance.

Conclusion

vec2vec demonstrates a practical method for unsupervised translation of text embeddings between different, incompatible spaces. This has dual implications: it enables greater interoperability between embedding-based systems and poses a security risk for vector databases by allowing attackers to translate and then analyze embeddings generated by unknown models. The success of vec2vec lends strong support to the Strong Platonic Representation Hypothesis, suggesting a universal underlying geometric structure in text embeddings.

Youtube Logo Streamline Icon: https://streamlinehq.com

HackerNews

  1. Harnessing the Universal Geometry of Embeddings (122 points, 39 comments)
  2. Text Embeddings are All Alike (5 points, 0 comments)
Reddit Logo Streamline Icon: https://streamlinehq.com

Reddit