Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data

Abstract: Multimodal representations enable zero-shot classification and retrieval, but aligning independently trained models usually requires large amounts of paired data. Yet, the Platonic Representation Hypothesis suggests that models trained on different modalities may converge spontaneously toward a shared representation geometry. But then, do we even need paired examples for cross-modal alignment? Remarkably, we show that paired examples are unnecessary for coarse cross-modal alignment. Our simple Wasserstein Procrustes method with a coarse geometric initialization aligns two disjoint embedding sets by estimating a single orthogonal map without seeing any pairs. Across datasets, modalities, and unimodal models, we show that we can consistently align independently trained representations without pairs, and standard geometric alignment metrics accurately predict when this is possible. Nevertheless, we can naturally benefit from paired examples. In the very few-pair regime, our method substantially outperforms existing ones, while staying competitive with pair-based methods with more added examples. Finally, we demonstrate that the resulting alignments can enable text-to-image generation without paired examples. These results show that independently trained models often share enough geometry to establish cross-modal correspondence with little or no paired data.

• The paper introduces a method, Wasserstein Procrustes with geometric initialization, for aligning independently trained embedding spaces without paired examples, achieving a mean FOSCTTM of 0.154 on the MS COCO dataset, demonstrating significant semantic structure recovery.
• This method optimizes both a semi-orthogonal map and a transport plan through iterative clustering and matching steps, avoiding the need to materialize a full cross-similarity or transport matrix, and scales linearly with the number of samples.
• The alignment quality across various domains is highly correlated with geometric similarity measures, such as Centered Kernel Alignment (CKA), with high CKA values indicating better alignment and downstream success in applications like zero-shot classification and language-based image retrieval.

Problem formulation and central claim

“Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data” (2610.09411) studies whether independently trained embedding spaces from different modalities can be aligned without observing any cross-modal correspondences. The paper challenges the conventional assumption that even minimal paired supervision is necessary to establish a connection between modalities. Its central claim is deliberately limited but substantial: paired examples are unnecessary for recovering coarse cross-modal semantics when the two embedding spaces share sufficient geometric structure.

The setting is more difficult than standard few-shot alignment. Let the two embedding collections be independently centered and normalized matrices $X \in \mathbb{R}^{n \times d_X}$ and $Y \in \mathbb{R}^{m \times d_Y}$. The samples have no known correspondence, the feature coordinates have no shared basis, the datasets may differ, and the dimensions may be unequal. The method must therefore infer both a sample correspondence and a cross-modal transformation from the geometry of the two point clouds alone.

The empirical premise is related to the Platonic Representation Hypothesis: independently trained models may organize semantically related observations in partially convergent representational geometries. The paper’s contribution is to convert this representational agreement into an explicit unpaired alignment procedure and to characterize when the procedure succeeds. The resulting view is not that the modalities become identically represented, but that their shared, coarse organization can provide enough information to infer a useful orthogonal map.

Figure 1: Shared coarse geometry provides a cross-modal correspondence without paired examples.

Wasserstein Procrustes with geometric initialization

The alignment objective is a Wasserstein Procrustes problem. The method jointly estimates a semi-orthogonal map $W$ and a transport plan $P$ between the two embedding sets. Given one of these variables, the other has a tractable update: Procrustes yields the optimal orthogonal map for a fixed correspondence, while linear assignment yields the optimal correspondence for a fixed map. Alternating these updates is computationally simple, but the joint objective is non-convex and uninformative random or uniform initialization generally fails.

The paper’s main algorithmic contribution is therefore the initialization rather than the final Procrustes refinement. It repeatedly samples batches from both embedding spaces, clusters each batch into $C$ groups, and matches cluster centers by maximizing linear CKA under a permutation. This center-matching step is a quadratic assignment problem and is closely related to a low-rank Gromov-Wasserstein objective: it seeks a correspondence preserving within-space similarity structure rather than relying on pointwise distances being exactly isometric.

The resulting cluster-level transport plans are averaged across restarts. A batched assignment step then converts this coarse transport into sample-level pseudo-pairs, from which an initial Procrustes map is computed. Refinement alternates batched Hungarian assignment and Procrustes updates. The implementation uses $C=30$ clusters, $S=30$ initialization restarts, batch size $10{,}000$, and $R=100$ refinement iterations as a common default.

This design has two important computational properties. First, it avoids materializing an $n \times m$ cross-similarity or transport matrix. Second, after the clustering and quadratic assignment stages, the read-out and refinement cost scales linearly with the number of samples through batched assignments. The additional memory requirement is $Y \in \mathbb{R}^{m \times d_Y}$0 beyond the embeddings. The method consequently applies to substantially larger embedding collections than a direct quadratic transport formulation would allow.

Known pairs are incorporated without changing the overall procedure. They enter the cluster-matching objective as a linear term and contribute to the Procrustes updates alongside pseudo-pairs. This gives a continuous interpolation between the zero-pair and few-pair regimes rather than requiring separate algorithms.

Unpaired vision-language alignment

The main evaluation combines seven vision models, three LLMs, and four image-caption datasets: MS COCO, Stanford Paragraph Captions, Densely Captioned Images, and DOCCI. The principal metric is FOSCTTM, where $Y \in \mathbb{R}^{m \times d_Y}$1 corresponds to random matching and $Y \in \mathbb{R}^{m \times d_Y}$2 to perfect alignment. The paper also evaluates zero-shot image classification by mapping image embeddings into the language space and retrieving class prompts.

On MS COCO, the proposed method outperforms mini-vec2vec on 17 of 21 vision-LLM pairs. Its mean FOSCTTM is $Y \in \mathbb{R}^{m \times d_Y}$3, compared with $Y \in \mathbb{R}^{m \times d_Y}$4 for mini-vec2vec and approximately $Y \in \mathbb{R}^{m \times d_Y}$5 for random matching. This performance remains far from the $Y \in \mathbb{R}^{m \times d_Y}$6 FOSCTTM reported for CLIP ViT-L/14, which was trained on approximately 400 million paired image-text examples. The comparison therefore supports a precise interpretation: unpaired geometry recovers substantial semantic structure but does not generally reconstruct exact instance-level correspondence.

The semantic transfer results are stronger than the raw retrieval scores might suggest. Without using classification data during alignment, the method obtains average zero-shot accuracies of 47.4% Top-1 on CIFAR-10, 30.1% Top-5 on CIFAR-100, and 26.5% Top-5 on ImageNet-100. For the model combination studied in prior work, the method increases CIFAR-10 Top-1 accuracy from 37.3% to 69.1%, nearly doubling performance. The implication is that coarse alignment can be operationally useful even when FOSCTTM remains far from exact matching: downstream classification depends primarily on semantic regions and class-level organization, not necessarily on one-to-one recovery of individual examples.

Figure 2: Unpaired alignment on MS COCO improves FOSCTTM and transfers semantic information to zero-shot classification.

The method also works when the two modalities are drawn from different datasets. In the cross-dataset experiment, images come from MS COCO while captions come from Stanford Paragraph Captions, so no hidden paired examples exist. Mean FOSCTTM degrades from $Y \in \mathbb{R}^{m \times d_Y}$7 to $0.236, but remains substantially below chance. This result weakens an alternative explanation in which the method merely reconstructs the latent permutation of two disjoint views of the same corpus. Instead, the alignment exploits common semantic organization across datasets.

Performance is not uniform across language representations. Generative token pooling is competitive on MS COCO but performs poorly on detailed captioning corpora. The paper attributes this to a mismatch between the pooling mechanism and caption length: generated-token pooling tends to improve geometric alignment for short captions, with CKA gains of $Y \in \mathbb{R}^{m \times d_Y}$8 on MS COCO, $Y \in \mathbb{R}^{m \times d_Y}$9 on CC12M, and $W$0 on WIT, but it decreases alignment by $W$1 on DOCCI and $W$2 on DCI. The implication is that representation quality for unpaired alignment depends not only on model scale but also on whether the pooling operation preserves the information shared with the other modality.

Geometry predicts when alignment succeeds

A major empirical result is that geometric similarity predicts the quality of unpaired alignment. Across the vision-language experiments, CKA, mutual $W$3-nearest-neighbor agreement, triplet similarity, and quadruplet similarity all correlate with FOSCTTM, with CKA providing the strongest relationship. Pooled over the four image-caption corpora, CKA explains $W$4 of the variance in alignment quality, compared with $W$5 for mutual $W$6-NN, $W$7 for triplet similarity, and $W$8 for quadruplet similarity.

The relationship extends beyond vision and language. Across independently trained representations from medicine, biology, chemistry, materials science, astronomy, neuroscience, and language, CKA explains 69% of the variance in unpaired alignment performance, with Spearman correlation $W$$9</strong> between CKA and FOSCTTM. High-CKA pairs, such as independently trained text encoders, fMRI subject representations, and some single-cell assays, are often alignable well below chance. Low-CKA pairs, including several tissue-imaging, mass-spectrometry, and multimodal neuroscience settings, remain close to chance.</p> <p><img src="https://images.emergentmind.com/paper-images/2610-09411/other_domains-domain_scatter.png" alt="Figure 3" title="" class="markdown-image" loading="lazy"></p> <p><p class="figure-caption">Figure 3: Across scientific domains, higher shared geometry is associated with lower FOSCTTM.</p></p> <p>The <a href="https://www.emergentmind.com/topics/domain-specific-benchmarks" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">domain-specific benchmarks</a> further demonstrate that the method is not dependent on image-text pretraining conventions. On Natural Questions, the method reaches FOSCTTM of approximately <strong>$$P$0 on nine of ten encoder pairs and $P$1 on the remaining pair. On PBMC RNA-to-ATAC alignment, it achieves $P$2, compared with $P$3 for SCOT+, while label-transfer accuracy is 94.2% versus 91.9%. On NSD fMRI subject alignment, it obtains $P$4, compared with $P$5 for Platonic Brain. Thus, a generic geometry-based aligner is competitive with, and in one case improves upon, domain-specific methods, although the fMRI difference lies within the reported variability.

The use of geometry as a predictor has a practical limitation. Standard CKA and related metrics require matched samples to compare corresponding kernel matrices. Consequently, the paper can establish that geometry predicts success when paired evaluation data are available, but it cannot yet determine reliably from two completely unpaired datasets whether alignment will work. An unsupervised alignment-prediction score remains unresolved.

Alignment is predominantly coarse

The paper provides several analyses showing that the recovered correspondence is concentrated at coarse semantic scales. When both modalities are jointly clustered, alignment between cluster centers is consistently stronger than alignment among points within individual clusters or among random points. This indicates that the representations agree more reliably on broad regions of the semantic space than on local neighborhoods.

Figure 4: Cluster centers exhibit stronger cross-modal alignment than within-cluster or random samples.

The shared structure is also concentrated in dominant directions. Projection onto the top ten principal components captures most of the measured alignment and sometimes exceeds the alignment obtained in the full spaces. Random subspaces require substantially more dimensions to obtain comparable scores. Mutual $P$6-NN is an exception: approximately 100 principal components are needed to approximate full-space neighborhood alignment, indicating that local structure is distributed more broadly than global similarity structure.

Figure 5: A small number of principal components capture most global cross-modal alignment, while local neighborhood agreement requires more dimensions.

A complementary similarity-kernel analysis finds that cosine similarities between approximately $P$7 and $P$8 contribute most to CKA alignment. Clipping similarities above $P$9 has limited effect, as does replacing negative similarities with zero. Hence, the shared geometry is not primarily the exact arrangement of very close neighbors or very distant points; it lies in intermediate-scale relationships. This explains why unpaired alignment can recover scene types, object categories, cell populations, or broad scientific states without reliably identifying individual samples.

Few-pair alignment and sample efficiency

The unpaired solution serves as an informed initialization when a small number of paired examples becomes available. On MS COCO with DINOv2 and Qwen3 generative pooling, the proposed method outperforms all evaluated few-pair baselines at every tested budget from zero to 1,000 pairs.

The gains are largest in the low-pair regime. With at most 20 pairs, its FOSCTTM is 14 to 28 times lower than that of the strongest baseline. The advantage remains 5.4 times at 50 pairs and 3.2 times at 100 pairs. Baselines that match the method’s zero-pair FOSCTTM generally require 100–500 paired examples, while SUE does not reach the same level in the tested range.

The downstream result is similarly pronounced. The zero-pair model reaches 79.6% CIFAR-10 Top-1 accuracy, a level reached by STRUCTURE and the orthogonal-map baseline only with 500 pairs. Conversely, the proposed method needs only 20 pairs to match or outperform five of seven baselines trained with 1,000 pairs. The implication is not merely that geometry can replace supervision in a narrow zero-pair setting; it can also substantially improve the sample efficiency of conventional paired alignment.

Figure 6: Geometric initialization provides the largest advantage when fewer than 100 paired examples are available.

The ablations identify cluster matching and QAP quality as the critical components. PCA, sorting, random Gromov-Wasserstein, and uniform initialization remain close to chance on difficult benchmarks, whereas cluster matching with MPOpt produces strong initial correspondences. Increasing the number of restarts reduces both error and variance; on MS COCO, FOSCTTM decreases from $C$0 with one restart to $C$1 with 30 restarts and $C$2 with 100 restarts. The method is therefore simple at the level of its objective, but its empirical performance depends materially on obtaining a high-quality coarse assignment.

Cross-modal retrieval and text-to-image generation

Qualitative retrieval and PCA visualizations show that the aligned spaces share broad scene and object categories. Text can retrieve images with compatible semantic content, and images can retrieve captions from a different corpus. The aligned PCA space also organizes MS COCO supercategories into approximately corresponding regions, although exact image-caption identity is not recovered consistently.

The paper extends the alignment to text-to-image generation through a deliberately modular pipeline. A diffusion model is trained only on images and conditioned on DINOv2 embeddings. Text embeddings are then mapped into the image embedding space using either the proposed unpaired aligner or a linear map trained on known pairs. The diffusion model and decoder never see text, so the cross-modal aligner is the only component connecting language to image generation.

With MPNet and zero paired examples, the proposed map produces images with the correct broad semantic class and scene. On CyclePrefDB, the zero-pair system achieves CLIPScore 0.466, VQAScore 0.458, and TIFA 0.552, compared with 0.245, 0.207, and 0.267 for a zero-pair linear map. It also exceeds the linear map trained with 100 pairs on these three metrics, whose corresponding scores are $C$3, $C$4, and $C$5. With 100 pairs, the proposed method improves to 0.510 CLIPScore, 0.511 VQAScore, and 0.615 TIFA.

The result is substantially weaker than paired text-to-image systems: Scale-RAE and real images exceed $C$6 on VQAScore and TIFA. This gap is expected because the proposed pipeline uses at most 100 paired examples and relies on a diffusion model trained without text. The experiment should therefore be interpreted as evidence that coarse unpaired alignment can transmit usable semantic conditioning, not as a competitive text-to-image generation method.

Figure 7: Unpaired alignment enables broad text-to-image semantic transfer, while additional pairs improve fine-grained faithfulness.

The same ordering holds with Contriever, which has not been trained on MS COCO captions. This reduces concern that the MPNet result is primarily caused by memorization of the evaluation caption distribution. Nevertheless, the Contriever results are lower overall, and on MS COCO its zero-pair TIFA of $C$7 is below the linear map with 100 pairs at $C$8. Thus, the generation experiment confirms robustness of the qualitative mechanism but also exposes sensitivity to the geometry of the language encoder.

Limitations and open questions

The strongest assumption is that the two representations share enough global geometry for a low-rank correspondence to be informative. When that assumption fails, the method remains near chance. The paper reports chance-level or near-chance results for several pairs, including galaxy images versus spectra, tissue versus MRI or CT, MEG versus text, and mass spectra versus molecular fingerprints. Similar semantic content alone is therefore insufficient.

The orthogonal-map restriction is another substantive modeling assumption. It is appropriate when the shared information is approximately related by a rotation, reflection, or dimensional embedding, but it cannot represent arbitrary nonlinear or strongly anisotropic cross-modal relationships. The method also assumes that coarse cluster structure can be matched through CKA-based QAP optimization. If cluster geometry is unstable, highly multimodal, or dominated by modality-specific nuisance factors, the initialization may produce incorrect pseudo-pairs that refinement cannot recover from.

The evaluation metric itself creates a distinction between coarse semantic success and exact correspondence. FOSCTTM measures instance-level retrieval on held-out paired samples, while zero-shot classification and qualitative generation can benefit from category-level alignment. The paper appropriately reports both, but the results should not be interpreted as evidence that zero-pair alignment generally recovers sample identity.

Several experiments also rely on relatively favorable representation and data conditions. The image-caption corpora contain descriptions with substantial visual relevance, while the difficult detailed-caption settings reveal that representation pooling can substantially alter alignment quality. Geometry scores for some scientific modality pairs are estimated from only 123–168 matched samples, which increases uncertainty. Finally, the paper demonstrates that CKA predicts alignment when matched validation data are available, but a reliable predictor based solely on unpaired observations remains an open methodological problem.

Conclusion

The paper establishes that independently trained cross-modal representations can often be aligned without paired examples when their embedding geometries share sufficiently strong coarse structure. Its Wasserstein Procrustes method combines cluster-level CKA/QAP matching, batched assignment, and orthogonal refinement to recover semantic correspondences without labels, reference points, or hidden paired training examples.

The main empirical findings are consistent across image-text, language, biological, and neuroscientific settings: unpaired alignment can substantially outperform chance, CKA strongly predicts success, and the resulting initialization dramatically improves few-pair sample efficiency. The method does not recover fine-grained correspondence universally, and its success depends on a geometric compatibility that cannot yet be diagnosed from fully unpaired data alone. Within those conditions, however, the paper shows that paired supervision is not always the primary source of cross-modal alignment; in many representations, the requisite coarse correspondence is already encoded in their shared geometry.