Harnessing the Universal Geometry of Embeddings (2505.12540v3)

Abstract: We introduce the first method for translating text embeddings from one vector space to another without any paired data, encoders, or predefined sets of matches. Our unsupervised approach translates any embedding to and from a universal latent representation (i.e., a universal semantic structure conjectured by the Platonic Representation Hypothesis). Our translations achieve high cosine similarity across model pairs with different architectures, parameter counts, and training datasets. The ability to translate unknown embeddings into a different space while preserving their geometry has serious implications for the security of vector databases. An adversary with access only to embedding vectors can extract sensitive information about the underlying documents, sufficient for classification and attribute inference.

• The paper introduces vec2vec, a novel technique that maps text embeddings to a shared latent space without requiring paired data.
• It details a modular architecture using MLPs, residual connections, and adversarial training to ensure robust translation between embedding spaces.
• The study highlights security implications by demonstrating how translated embeddings can expose sensitive information through attribute inference and inversion.

This paper introduces "vec2vec" (2505.12540), a novel method for translating text embeddings between different vector spaces without requiring any paired data, access to the original encoders, or predefined sets of matches. The core idea is based on the "Strong Platonic Representation Hypothesis," which conjectures that neural networks trained with similar objectives and modalities, even with different architectures and data, converge to a universal latent semantic structure. The vec2vec method learns to map embeddings to and from this universal latent space, enabling unsupervised translation.

The primary application demonstrated is enhancing the interoperability of embedding models. For instance, if you have a vector database generated by an unknown model ($M_1$) and wish to use tools or perform analysis that requires embeddings from a known model ($M_2$), vec2vec can translate the $M_1$ embeddings into $M_2$'s space. This translation aims to preserve the semantic geometry, meaning that documents close in $M_1$'s space should have their translated embeddings close in $M_2$'s space, and ideally, the translated embedding $F(M_1(d_i))$ should be very close to the "true" embedding $M_2(d_i)$.

A significant implication highlighted is the security of vector databases. If an adversary gains access to a dump of embedding vectors, even without knowing the model that generated them, they can use vec2vec to translate these embeddings into a space of a known model. Once translated, they can apply techniques like attribute inference or embedding inversion to extract sensitive information from the original documents.

vec2vec Method: Architecture and Optimization

The vec2vec architecture is modular and employs Multilayer Perceptrons (MLPs) with residual connections, layer normalization, and SiLU nonlinearities. It consists of:

1. Input Adapters ($A_1, A_2$): These modules transform embeddings from their specific source spaces ($R^{d_{M_1}}$ or $R^{d_{M_2}}$) into a shared universal latent representation of dimension $Z$. So, $A_1: \mathbb{R}^{d_{M_1}} \to \mathbb{R}^{Z}$ and $A_2: \mathbb{R}^{d_{M_2}} \to \mathbb{R}^{Z}$.
2. Shared Backbone ($T$): This network takes the adapted embeddings and extracts a common latent embedding, mapping $\mathbb{R}^{Z} \to \mathbb{R}^{Z}$.
3. Output Adapters ($B_1, B_2$): These modules translate the common latent embeddings back into the specific encoder spaces. So, $B_1: \mathbb{R}^{Z} \to \mathbb{R}^{d_{M_1}}$ and $B_2: \mathbb{R}^{Z} \to \mathbb{R}^{d_{M_2}}$.

Using these components, two main types of functions are defined:

• Translation functions:
  • $F_1 = B_2 \circ T \circ A_1$: Translates from $M_1$'s space to $M_2$'s space.
  • $F_2 = B_1 \circ T \circ A_2$: Translates from $M_2$'s space to $M_1$'s space.
• Reconstruction mappings:
  • $R_1 = B_1 \circ T \circ A_1$: Encodes from $M_1$'s space to latent and decodes back to $M_1$'s space.
  • $R_2 = B_2 \circ T \circ A_2$: Encodes from $M_2$'s space to latent and decodes back to $M_2$'s space.

The optimization process is adversarial, involving generator networks (the adapters and backbone) and discriminator networks. Discriminators ($D_1, D_2$) operate on the output embeddings, and $D_1^\ell, D_2^\ell$ operate on the latent representations.

The overall loss function combines adversarial losses and generator-specific constraints:

$$\theta^* = \arg \min_{\theta} \max_{D_1, D_2, D_{1}^{\ell}, D_{2}^{\ell}} \mathcal{L}_{\text{adv}} + \lambda_{\text{gen}} \mathcal{L}_{\text{gen}}$$

1. Adversarial Loss ($\mathcal{L}_{\text{adv}}$): This loss encourages the translated embeddings (and their latent representations) to be indistinguishable from real embeddings (and their latent representations) in the target space. It uses a standard GAN loss formulation: $$\mathcal{L}_{\text{adv}} = \mathcal{L}_{\text{GAN}}(D_1, F_1) + \mathcal{L}_{\text{GAN}}(D_2, F_2) + \mathcal{L}_{\text{GAN}}(D_{1}^{\ell}, T \circ A_1) + \mathcal{L}_{\text{GAN}}(D_{2}^{\ell}, T \circ A_2)$$.
2. Generator Loss ($\mathcal{L}_{\text{gen}}$): This comprises three components:
   • Reconstruction Loss ($\mathcal{L}_{\text{rec}}$): Ensures that an embedding, when mapped to the latent space and back to its original embedding space, remains close to the initial representation. $$\mathcal{L}_{\text{rec}} = \mathbb{E}_{x \sim p}\|R_1(x)-x\|_2^2 + \mathbb{E}_{y \sim q}\|R_2(y)-y\|_2^2$$, where $x$ are embeddings from $M_1$ and $y$ are from $M_2$.
   • Cycle-Consistency Loss ($\mathcal{L}_{\text{CC}}$): Ensures that translating an embedding to the other space and back again results in minimal change. $$\mathcal{L}_{\text{CC}} = \mathbb{E}_{x \sim p}\|F_2(F_1(x))-x\|_2^2 + \mathbb{E}_{y \sim q}\|F_1(F_2(y))-y\|_2^2$$.
   • Vector Space Preservation Loss ($\mathcal{L}_{\text{VSP}}$): Encourages the preservation of pairwise relationships (dot products) between embeddings after translation. For a batch of $B$ embeddings $x_i$ from $M_1$ and $y_i$ from $M_2$: $$\mathcal{L}_{\text{VSP}} = \frac{1}{B} \sum_{i=1}^{B} \sum_{j=1}^{B} \left[ \|M_1(x_i) \cdot M_1(x_j) - F_1(M_1(x_i)) \cdot F_1(M_1(x_j))\|_2^2 + \|M_2(y_i) \cdot M_2(y_j) - F_2(M_2(y_i)) \cdot F_2(M_2(y_j))\|_2^2 \right]$$. The combined generator loss is $$\mathcal{L}_{\text{gen}} = \lambda_{\text{rec}}\mathcal{L}_{\text{rec}} + \lambda_{\text{CC}}\mathcal{L}_{\text{CC}} + \lambda_{\text{VSP}}\mathcal{L}_{\text{VSP}}$$.

Implementation and Training

• Network Architecture: MLPs with residual connections, LayerNorm, and SiLU activations. Discriminators are similar but without residual connections.
• Training Data: The method is trained using unpaired sets of embeddings. For example, 1 million embeddings from model $M_1$ (e.g., GTR) generated from one set of NQ documents, and 1 million embeddings from model $M_2$ (e.g., GTE) generated from a disjoint set of NQ documents. The paper shows that even 50,000 embeddings can yield good results (Table A.5).
• Training Process: Due to GAN instability, selecting the best model from multiple initializations is recommended.
• Computational Cost: Training a single vec2vec model pair took 1-7 days on GPUs like NVIDIA 2080Ti, L4, A40, or A100.

Practical Application: Implementing vec2vec

To implement vec2vec for translating embeddings from an unknown source model $M_1$ to a known target model $M_2$:

1. Collect Unpaired Embeddings:
   • Obtain a large set of embeddings $\{u_i\}$ from the source $M_1$. These could be from a compromised vector database.
   • Generate a large set of embeddings $\{v_j\}$ using your known model $M_2$ from a corpus that is distributionally similar (e.g., same language, general domain) to the source documents of $\{u_i\}$. Crucially, $\{u_i\}$ and $\{v_j\}$ do not need to correspond to the same documents.
2. Define vec2vec Model Components:
   • Implement $A_1, A_2, T, B_1, B_2$ as MLPs. The paper suggests MLPs with residual connections, LayerNorm, and SiLU. The latent dimension $Z$ is a hyperparameter.
   • Implement discriminators $D_1, D_2, D_1^\ell, D_2^\ell$ as MLPs.
3. Train the Model:
   • Initialize the networks.
   • Iteratively update the generator parameters ($\theta = \{A_1, A_2, T, B_1, B_2\}$) and discriminator parameters using the adversarial and generator losses described above.
   • You'll need to choose hyperparameters $$\lambda_{\text{gen}}, \lambda_{\text{rec}}, \lambda_{\text{CC}}, \lambda_{\text{VSP}}$$.
   • The paper used Adam optimizer.
   • Save model checkpoints regularly and select the best performing one based on a validation set or heuristics related to loss convergence, as GAN training can be unstable.

# Pseudocode for a training step # x_m1: batch of embeddings from M1 # x_m2: batch of embeddings from M2 # --- Train Discriminators --- # Latent discriminators z_m1_real = T(A1(x_m1)) z_m2_real = T(A2(x_m2)) # (No explicit fake latent samples for D_latent, it operates on generator's latent output) # Output discriminators x_m1_to_m2_fake = F1(x_m1) # B2(T(A1(x_m1))) x_m2_to_m1_fake = F2(x_m2) # B1(T(A2(x_m2))) # Calculate discriminator losses (e.g., using BCEWithLogitsLoss) # d_loss_latent_m1 = loss_gan_D(D_latent_m1, z_m1_real.detach(), T(A2(x_m2)).detach()) # Comparing T(A1(x)) vs T(A2(y)) # d_loss_latent_m2 = loss_gan_D(D_latent_m2, z_m2_real.detach(), T(A1(x_m1)).detach()) # d_loss_output_m1 = loss_gan_D(D1, x_m1_real, x_m2_to_m1_fake.detach()) # Comparing M1 real vs M2->M1 fake # d_loss_output_m2 = loss_gan_D(D2, x_m2_real, x_m1_to_m2_fake.detach()) # Comparing M2 real vs M1->M2 fake # d_loss = d_loss_latent_m1 + d_loss_latent_m2 + d_loss_output_m1 + d_loss_output_m2 # Optimize discriminators # --- Train Generators --- # Adversarial losses for generator # g_loss_adv_latent_m1 = loss_gan_G(D_latent_m1, T(A2(x_m2))) # Fool D_latent_m1 with T(A2(y)) # g_loss_adv_latent_m2 = loss_gan_G(D_latent_m2, T(A1(x_m1))) # Fool D_latent_m2 with T(A1(x)) # g_loss_adv_output_m1 = loss_gan_G(D1, x_m2_to_m1_fake) # Fool D1 with M2->M1 fake # g_loss_adv_output_m2 = loss_gan_G(D2, x_m1_to_m2_fake) # Fool D2 with M1->M2 fake # g_loss_adv = g_loss_adv_latent_m1 + g_loss_adv_latent_m2 + g_loss_adv_output_m1 + g_loss_adv_output_m2 # Reconstruction losses x_m1_reconstructed = R1(x_m1) # B1(T(A1(x_m1))) x_m2_reconstructed = R2(x_m2) # B2(T(A2(x_m2))) loss_rec = mse_loss(x_m1_reconstructed, x_m1) + mse_loss(x_m2_reconstructed, x_m2) # Cycle-consistency losses x_m1_cycled = F2(F1(x_m1)) # B1(T(A2( B2(T(A1(x_m1))) ))) x_m2_cycled = F1(F2(x_m2)) # B2(T(A1( B1(T(A2(x_m2))) ))) loss_cc = mse_loss(x_m1_cycled, x_m1) + mse_loss(x_m2_cycled, x_m2) # Vector Space Preservation Loss (simplified conceptual calculation) # pairwise_dots_m1_orig = calculate_pairwise_dots(x_m1) # pairwise_dots_m1_translated = calculate_pairwise_dots(F1(x_m1)) # loss_vsp_m1 = mse_loss(pairwise_dots_m1_translated, pairwise_dots_m1_orig) # (Similarly for x_m2 and F2) # loss_vsp = loss_vsp_m1 + loss_vsp_m2 # g_loss_total = g_loss_adv + lambda_rec * loss_rec + lambda_cc * loss_cc + lambda_vsp * loss_vsp # Optimize generators

1. Translate and Utilize Embeddings:
   • Once trained, use the $F_1 = B_2 \circ T \circ A_1$ function to translate the source embeddings $\{u_i\}$ into $M_2$'s space: $v'_i = F_1(u_i)$.
   • These translated embeddings $v'_i$ can then be used with tools designed for $M_2$, for similarity search within $M_2$'s space, or for information extraction attacks.

Experimental Results and Performance

• Translation Quality: vec2vec achieves high cosine similarity (up to 0.92) between translated embeddings and ground-truth target embeddings (Table 2). Top-1 accuracy (correctly matching an embedding to its true counterpart in a large set) reaches up to 100% for some model pairs. It significantly outperforms naïve baselines (no translation) and oracle-aided optimal assignment (which assumes candidate targets are known) in cross-backbone model pairs.
• Robustness: The method shows good performance on out-of-distribution datasets (e.g., trained on NQ from Wikipedia, tested on tweets from TweetTopic or medical records from MIMIC, see Table 3, Table A.2, A.3).
• Cross-Modality: vec2vec can translate between unimodal text embeddings and multimodal embeddings like CLIP (Table 4), albeit with slightly lower performance than text-to-text, but still outperforming baselines.
• Information Extraction:
  • Attribute Inference: Translated embeddings allow for effective zero-shot attribute inference (e.g., inferring topics of tweets or diseases from medical records). Performance often surpasses naïve baselines and sometimes even an oracle baseline that uses true embeddings in the target space (Table 5).
  • Embedding Inversion: Translated embeddings retain enough semantic information for off-the-shelf zero-shot inversion models to reconstruct parts of the original text (e.g., names, topics from Enron emails, see Figure 5, Figure 6). Accuracy varied, reaching up to 80% for some pairs.

Limitations and Considerations

• GAN Training Instability: A key limitation is the inherent instability of GAN training. The authors mitigate this by training multiple initializations and selecting the best. More robust training procedures are future work.
• Computational Resources: Training is computationally intensive.
• Hyperparameter Tuning: The various loss weights ($$\lambda_{\text{rec}}, \lambda_{\text{CC}}, \lambda_{\text{VSP}}$$) and architectural choices (MLP depth/width, latent dimension $Z$) are hyperparameters that may require tuning for optimal performance on new pairs of embedding models. The ablation paper (Table A.4) shows that removing any of the loss components (VSP, CC, latent GAN) or skipping hyperparameter tuning degrades performance.

Conclusion

vec2vec demonstrates a practical method for unsupervised translation of text embeddings between different, incompatible spaces. This has dual implications: it enables greater interoperability between embedding-based systems and poses a security risk for vector databases by allowing attackers to translate and then analyze embeddings generated by unknown models. The success of vec2vec lends strong support to the Strong Platonic Representation Hypothesis, suggesting a universal underlying geometric structure in text embeddings.