SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport

Abstract: The Platonic Representation Hypothesis posits that neural networks trained on different modalities converge toward a shared statistical model of the world. Recent work exploits this convergence by aligning frozen pretrained vision and LLMs with lightweight alignment layers, but typically relies on contrastive losses and millions of paired samples. In this work, we ask whether meaningful alignment can be achieved with substantially less supervision. We introduce a semi-supervised setting in which pretrained unimodal encoders are aligned using a small number of image-text pairs together with large amounts of unpaired data. To address this challenge, we propose SOTAlign, a two-stage framework that first recovers a coarse shared geometry from limited paired data using a linear teacher, then refines the alignment on unpaired samples via an optimal-transport-based divergence that transfers relational structure without overconstraining the target space. Unlike existing semi-supervised methods, SOTAlign effectively leverages unpaired images and text, learning robust joint embeddings across datasets and encoder pairs, and significantly outperforming supervised and semi-supervised baselines.

• The paper introduces SOTAlign, which combines a small paired SigLIP objective with unpaired image-text data and a linear teacher to align frozen vision and language encoders efficiently.
• The proposed KLOT optimal-transport divergence matches full affinity transport plans, while its explicit gradient is up to 100× more memory-efficient and 50× faster than competing differentiation methods at large batch sizes.
• SOTAlign improves COCO retrieval to 26.5 T2I and 34.1 I2T R@1 and reaches 46.1% ImageNet top-1 accuracy, while matching supervised baselines with about four times less paired supervision.

SOTAlign addresses the problem of aligning pretrained unimodal vision and language encoders when paired image–text supervision is scarce but unpaired unimodal data is abundant. The work is grounded in the Platonic Representation Hypothesis (Huh et al., 2024), which posits that neural networks trained on different modalities converge toward statistically compatible representations of the world; if this hypothesis holds, only lightweight alignment layers should be needed to reconcile frozen encoders, even without large-scale paired corpora. The paper makes three contributions: a demonstration that linear alignment from as few as 10k pairs already yields meaningful cross-modal alignment, the SOTAlign framework that refines this alignment using up to 1M unpaired images and texts, and KLOT, an optimal-transport-based divergence whose explicit gradient removes the memory bottlenecks of prior OT methods.

Problem setting and method

The authors formulate semi-supervised alignment as follows: given $n_p$ paired image–text embeddings $(A,B)$ with $n_p \ll n_x, n_y$, together with unpaired image embeddings $X \in \mathbb{R}^{n_x \times d_x}$ and text embeddings $Y \in \mathbb{R}^{n_y \times d_y}$, learn alignment layers $f_{\theta_1}$ and $g_{\theta_2}$ mapping both modalities into a shared $d$-dimensional space. The training objective combines a supervised SigLIP loss on the paired samples with a regularization term that constrains the affinity matrix in the learned joint space to remain close to a target geometry induced by a "linear teacher":

$$\mathcal{L}_\alpha(\theta) = \mathcal{L}(\theta; A,B) + \alpha\,\Omega(\theta; X,Y).$$

The teacher consists of two closed-form or lightly trained linear projections $W_x, W_y$ fit on the small set of pairs, using one of three candidates: Orthogonal Procrustes alignment, Canonical Correlation Analysis (CCA), or contrastive training with a SigLIP loss. The paper provides closed-form solutions for Procrustes and CCA in its appendix.

The central methodological question is how to define the divergence between the learned affinity matrix and the teacher's target when no ground-truth correspondences exist for unpaired data. The paper evaluates three choices — CKA, generalized InfoNCE, and the proposed KLOT divergence:

$(A,B)$0

where $(A,B)$1 denotes entropic optimal transport via the Sinkhorn algorithm. Unlike CKA, which enforces $(A,B)$2 exactly, and InfoNCE, which preserves only nearest-neighbor structure, KLOT transfers the entire transport plan while leaving sufficient flexibility to avoid overfitting to an imperfect linear teacher.

Explicit gradient of KLOT

The key theoretical result is a closed-form gradient: $(A,B)$3. This is established by combining a duality identity for entropic OT plans with the envelope theorem, avoiding both Sinkhorn unrolling (whose memory cost scales with iteration count) and implicit differentiation. Empirically, at batch size $(A,B)$4k the explicit formula is reported as up to 100× more memory-efficient than unrolling and up to 50× faster than implicit differentiation. This is a general contribution applicable beyond SOTAlign to any method differentiating through Sinkhorn objectives, including OT-CLIP and S-CLIP.

Experimental results

Experiments use DINOv3 ViT-L and NV-Embed-v2 as default frozen encoders, trained for 2000 iterations on a single A100 with batches of up to 32k samples composed of paired and unpaired data. Evaluation covers zero-shot retrieval (COCO, Flickr30k) and classification (ImageNet, Food-101, CIFAR-10/100, DTD).

Linear teachers are surprisingly strong. With only 10k CC3M pairs, standalone CCA achieves 21.5 MeanR@1 on COCO retrieval and the linear contrastive model (the SAIL baseline) reaches 24.2%. Across all divergences, KLOT systematically outperforms CKA and InfoNCE; combining CCA with KLOT yields 30.3 MeanR@1 versus 24.2 for the best supervised baseline, a gain of over six points from unpaired data alone.

Benchmark against baselines. In the standard setting of 10k pairs plus 1M unpaired CC3M samples, adapted semi-supervised baselines (semi-supervised SAIL, STRUCTURE, NNCLR, S-CLIP) all fail to exploit unpaired data, performing at or below their supervised counterparts. SOTAlign reaches 26.5 T2I / 34.1 I2T R@1 on COCO and 51.7 / 60.8 on Flickr30k — gains of roughly +5 points over the strongest baselines. On zero-shot ImageNet classification it attains 46.1% top-1 accuracy, +7.9 over the best baseline, and +13.9 over supervised SAIL. Under the per-dataset setting of SUE, SOTAlign improves I2T R@5 by +14.3 on COCO, +40.0 on Flickr30k, and +32.5 on Polyvore.

Supervision efficiency. Varying the number of pairs from $(A,B)$5 to $(A,B)$6 while fixing 1M unpaired samples shows consistent gains over SAIL of up to +10 points in the $(A,B)$7–$(A,B)$8 regime; overall, SOTAlign matches SAIL's performance with approximately four times less paired supervision. Both methods fail at 100 pairs. Gains from unpaired data saturate around 500k samples.

Robustness analyses. Three findings deserve emphasis. First, SOTAlign tolerates severe distribution shift: unpaired images from ImageNet and texts from CC12M still improve performance despite being drawn from disjoint corpora, and the total spherical sliced Wasserstein distance between unimodal datasets and the paired distribution strongly correlates with downstream gains, providing a training-free proxy for selecting unpaired data sources. Second, caption quality matters: replacing raw web captions with synthetic captions yields +4.8 T2I R@1. Third, encoder choice interacts with the hypothesis under test: DINOv3 outperforms DINOv2 across all text encoders, and the authors report a Pearson correlation of $(A,B)$9 between representational similarity (mutual k-NN) of encoder pairs and downstream retrieval performance — evidence consistent with, though not proof of, representational convergence enabling low-supervision alignment.

Limitations and open questions

The paper concedes several constraints. Performance collapses under extremely sparse supervision (100 pairs), so the method still requires a minimum viable paired set whose size likely depends on the encoder pair. Gains from unpaired data diminish once paired supervision reaches $n_p \ll n_x, n_y$0 samples, and the benefit saturates near 500k unpaired samples, suggesting a bounded regime of applicability rather than unrestricted scaling. The correlation between mutual k-NN similarity and downstream performance is based on a limited number of encoder combinations, and the authors themselves note that additional data points are required before drawing firm conclusions. The distribution-shift analysis establishes correlation, not causation, between Wasserstein distance and performance gains. Finally, hyperparameter selection ($n_p \ll n_x, n_y$1) relies on validation retrieval on the source dataset, which may not transfer to domains where no such validation signal exists; the paper does not evaluate sensitivity to this choice.

Conclusion

SOTAlign demonstrates that meaningful vision–language alignment of frozen unimodal encoders can be achieved from roughly 10k paired samples augmented with large-scale unpaired data, using a two-stage design that distills a linear teacher's geometry through a scalable OT-based divergence. The explicit KLOT gradient is a broadly reusable technical advance, and the strong empirical gains over adapted semi-supervised baselines establish a new reference point for low-supervision multimodal alignment. Open questions include the minimum viable number of pairs across modality pairs and domains, causal characterization of when unpaired data helps, and extension of the framework beyond the vision–language testbed considered here.