Canonicalizing Multimodal Contrastive Representation Learning

Abstract: As models and data scale, independently trained networks often induce analogous notions of similarity. But, matching similarities is weaker than establishing an explicit correspondence between the representation spaces, especially for multimodal models, where consistency must hold not only within each modality, but also for the learned image-text coupling. We therefore ask: given two independently trained multimodal contrastive models (with encoders $(f, g)$ and $(\widetilde{f},\widetilde{g})$) -- trained on different distributions and with different architectures -- does a systematic geometric relationship exist between their embedding spaces? If so, what form does it take, and does it hold uniformly across modalities? In this work, we show that across model families such as CLIP, SigLIP, and FLAVA, this geometric relationship is well approximated by an orthogonal map (up to a global mean shift), i.e., there exists an orthogonal map $Q$ where $Q^\top Q = I$ such that $\widetilde{f}(x)\approx Q f(x)$ for paired images $x$. Strikingly, the same $Q$ simultaneously aligns the text encoders i.e., $\widetilde{g}(y)\approx Q g(y)$ for texts $y$. Theoretically, we prove that if the multimodal kernel agrees across models on a small anchor set i.e. $\langle f(x), g(y)\rangle \approx \langle \widetilde{f}(x), \widetilde{g}(y)\rangle$, then the two models must be related by a single orthogonal map $Q$ and the same $Q$ maps images and text across models. More broadly, this finding enables backward-compatible model upgrades, avoiding costly re-embedding, and has implications for the privacy of learned representations. Our project page: https://canonical-multimodal.github.io/

• The paper demonstrates that a rigid, modality-invariant orthogonal map can align independently trained image and text encoders effectively.
• It establishes theoretical guarantees through multimodal kernel agreement, proving that alignment holds under both perfect and approximate conditions.
• Empirical validation shows that the alignment method improves cosine similarity and retrieval accuracy, enabling efficient backward-compatible model upgrades.

Canonicalizing Multimodal Contrastive Representation Spaces

Motivation and Problem Formulation

Contrastive multimodal representation learning, exemplified by architectures such as CLIP, SigLIP, and FLAVA, anchors the semantic correspondence between modalities (e.g., image and text) at the embedding level. Historically, prior works have focused on representational convergence in terms of similarity kernels, as described by CKA or SVCCA, which are agnostic to affine transformations and do not provide explicit geometric correspondence. This work addresses a higher-fidelity question: given two independently trained multimodal contrastive models, does a systematic geometric relationship exist between their embedding spaces, and is this relationship modality-invariant?

The authors formalize the question using pairs of dual-encoder models $(f, g)$ and $(\tilde f, \tilde g)$, potentially trained on different distributions and architectures. The central focus is the cross-modal kernel $\langle f(x), g(y)\rangle$, which is hypothesized to be preserved (up to a constant) across independently trained models. The problem is non-trivial due to the modality gap, a geometrically stable but intrinsically disjoint embedding of image and text subspaces (see Figure 1).

Theoretical Contributions

The paper establishes several strong claims:

1. Existence of a Rigid, Modality-Invariant Orthogonal Map: It is shown empirically and theoretically that a single orthogonal transformation $Q$ (i.e., $Q^\top Q = I$) learned using the image encoders suffices to align both image and text embeddings across independently trained models. This relationship holds even under substantial architectural, dimensional, and data distribution disparities.
2. Theoretical Guarantees via Multimodal Kernel Agreement: The authors provide rigorous proofs that under population-level InfoNCE loss, agreement of the multimodal similarity kernel on a small anchor set forces alignment up to an isometry. If $$\langle f(x), g(y)\rangle \approx \langle \tilde f(x), \tilde g(y)\rangle$$ for $(x, y)$ in a modest anchor set, then an orthogonal map $Q$ relates the embedding spaces across both modalities. This result is established even under dataset curation and distributional shift assumptions.
3. Identifiability and Stability Bounds: The work further extends theoretical guarantees to the approximate regime, quantifying robustness of alignment when kernels are only approximately preserved. Under finite-sample estimation and imperfect anchor sets, the induced map is still approximately orthogonal, and semantic retrieval remains stable.

   Figure 2: A high-level schematic of the theoretical results, progressing from multimodal kernel alignment to explicit isometry and task-level correspondence.

Algorithmic Instantiation

The principal alignment method is the orthogonal Procrustes algorithm, solvable via SVD on paired mean-centered embeddings. A critical insight is that the orthogonality constraint preserves angles, norms, and neighbor relations in the embedding space, which is essential for downstream semantic transfer. Alignment is accomplished with paired samples from a single modality, reducing supervision requirements and providing effective transfer to the other modality.

Empirical Validation

Extensive experiments support the theoretical claims across a wide suite of models and datasets. Key findings include:

• Near-Oracle Cross-Model Alignment: Application of the orthogonal map boosts instance-level cosine similarity for image-image and text-text pairs from near zero to $0.8$--$0.9$, without degrading task accuracy.

  Figure 3: A single orthogonal map aligns image and text embeddings from two independent CLIP models, learned only from images.

• Modality-Invariant Transfer: Fitting $Q$ on images enables robust alignment of text embeddings, as measured by text-text cosine similarity and cross-model class retrieval accuracy.

  Figure 4: Alignment on Oxford-Pets; $Q$ learned from images generalizes to text, preserving retrieval accuracy and semantic structure across modalities.

• Data-Efficient and Transferable Alignment: Alignment map $Q$ is reliably estimated from only $\sim$30% of anchor data and generalizes to unseen classes and new datasets without re-fitting.

  Figure 5: The orthogonal map generalizes robustly under limited anchor supervision and to novel classes.

• Superiority of Orthogonal Maps: Linear and non-linear alignment maps can improve pointwise similarity but fail to preserve semantic geometry and cross-modal transfer, as evidenced by degraded retrieval accuracy.

  Figure 6: Comparison between orthogonal, linear, and non-linear aligners showing only orthogonal maps transfer successfully across modalities.

• Commutative Alignment Paths: Both direct and text-mediated alignment routes recover consistent semantic neighborhoods in k-NN retrieval, indicating that $Q$ commutes with cross-modal operators.

  Figure 7: Direct vs. text-mediated alignment paths yield consistent neighborhoods in retrieval.

Analysis of the Modality Gap

Contrary to naïve intuition, image and text embeddings in contrastive models do not converge to a unified space but are separated by a modality gap. Attempts to forcibly collapse this gap degrade task performance. The multimodal kernel, however, remains invariant across models, enabling effective cross-model and cross-modal alignment through global isometries.

Figure 1: PCA visualization of CIFAR-100 embeddings illustrates the modality gap: image and text reside in disjoint cones.

Figure 8: Multimodal kernel preservation is robust across CLIP variants, unlike unimodal kernel alignment.

Practical Implications

• Backward-Compatible Model Upgrades: The findings support efficient model upgrades in production settings. Existing ANN indices and embedding archives need not be re-computed; a small calibration set suffices to derive the orthogonal map and restore compatibility.
• Mix-and-Match Modular Pipelines: Practitioners can combine vision and text towers from disparate models without loss of geometric or semantic structure.
• Data Privacy and Governance: The ease of cross-model transfer raises implications for privacy: embeddings may encode more transferable semantic information than anticipated, and alignment can be accomplished without direct access to sensitive modality-specific data.

Limitations and Future Research

The evaluation focuses primarily on classification-level retrieval; fine-grained and dense ranking remain open tasks. Extension to other modalities (audio/video) and further exploration of semantic preservation, cycle consistency, and composability in complex pipelines are promising directions.

Conclusion

This work rigorously establishes that independently trained multimodal contrastive models, despite differences in data distribution, architecture, and embedding geometry, are related by a single, rigid orthogonal map that is shared across modalities. This geometric convergence is both theoretically precise and empirically robust, yielding practical strategies for backward-compatible model evolution and cross-modal interoperability in large-scale embedding systems.

Figure 9: Multimodal kernel structure and CKA measures are highly aligned across CLIP variants, confirming geometric convergence.

(Figure 10–16, 19, 20, 23, 24, 25 are not referenced due to lack of specific captions/detail.)