Modulating Cross-Modal Convergence with Single-Stimulus, Intra-Modal Dispersion

Abstract: Neural networks exhibit a remarkable degree of representational convergence across diverse architectures, training objectives, and even data modalities. This convergence is predictive of alignment with brain representation. A recent hypothesis suggests this arises from learning the underlying structure in the environment in similar ways. However, it is unclear how individual stimuli elicit convergent representations across networks. An image can be perceived in multiple ways and expressed differently using words. Here, we introduce a methodology based on the Generalized Procrustes Algorithm to measure intra-modal representational convergence at the single-stimulus level. We applied this to vision models with distinct training objectives, selecting stimuli based on their degree of alignment (intra-modal dispersion). Crucially, we found that this intra-modal dispersion strongly modulates alignment between vision and LLMs (cross-modal convergence). Specifically, stimuli with low intra-modal dispersion (high agreement among vision models) elicited significantly higher cross-modal alignment than those with high dispersion, by up to a factor of two (e.g., in pairings of DINOv2 with LLMs). This effect was robust to stimulus selection criteria and generalized across different pairings of vision and LLMs. Measuring convergence at the single-stimulus level provides a path toward understanding the sources of convergence and divergence across modalities, and between neural networks and human neural representations.

• The paper demonstrates that low intra-modal dispersion significantly improves cross-modal (vision-language) alignment, often by a factor of up to two.
• The study introduces a GPA-based methodology that quantifies stimulus-specific dispersion using both mean residuals and principal component scores.
• Stimulus-level dispersion predicts deviations of held-out models, supporting the Platonic Representation Hypothesis and the universality of neural representations.

Modulating Cross-Modal Convergence with Single-Stimulus, Intra-Modal Dispersion

Introduction

The investigation centers on representational alignment among artificial neural networks, especially concerning the convergence of models trained on different modalities and objectives. While prior literature has established global convergence between model representations—and between artificial and biological systems—the degree to which this convergence is stimulus-specific remains an open question. This work applies a Generalized Procrustes Analysis (GPA)-based metric to resolve representational dispersion at the single-stimulus level and systematically examines how this intra-modal dispersion modulates cross-modal convergence, particularly between vision and LLMs. Results demonstrate substantial, robust, and numerically strong modulation of cross-modal alignment as a function of intra-modal dispersion, validating and refining hypotheses about representation universality and the hierarchical structure of model agreement.

Methodological Contributions

A central methodological innovation is the application of GPA—a technique for aligning multiple embedding spaces into a consensus frame with minimal geometric distortion—to vision model representations. GPA is operationalized by finding model-specific orthogonal transformations that minimize the aggregate Frobenius distance between each model's transformed space and a joint reference. Residuals from this projection quantify stimulus-level dispersion across model representations.

Dispersion for each stimulus is summarized using two approaches:

1. Mean Model Residual: Directly averaging deviations across models.
2. Principal Component Score: Taking the first principal component (PC1) of the residual matrix, to robustly capture shared axes of disagreement.

This intrinsic-per-stimulus metric reveals idiosyncratic and shared structure in representation spaces, enabling targeted hypothesis testing regarding both intra- and cross-modal convergence.

Figure 1: Schematic of the GPA procedure for yielding joint spaces, quantifying per-stimulus dispersion, identifying high- and low-dispersion stimuli, and linking dispersion to cross-modal convergence measurements.

Experimental Framework

Representations are drawn from four distinct ViTs—MAE, DINOv2, CLIP, and CLIP fine-tuned on ImageNet-12K—providing architectural and objective diversity. The Wikipedia-based Image Text dataset (WIT) supplies 1024 paired image/caption stimuli. For each vision model, representations are taken from the penultimate CLS-token layer. Language representations are obtained from mean-pooled token embeddings of corresponding captions, using blocks selected for maximal alignment from BLOOM, OpenLLaMA, and LLaMA model families.

Cross-modal alignment is evaluated using the CKNNA metric, a local, k-nearest neighbor extension of CKA that emphasizes neighborhood geometry within representation spaces. Stimulus sets for alignment assessment are constructed by selecting either low-dispersion, high-dispersion, or random subsets, ranked by the above dispersion metrics.

Results: Dispersion as a Determinant of Cross-Modal Alignment

The analyses yield several numerical and conceptual findings:

• Stimuli with low intra-modal dispersion consistently exhibit higher cross-modal alignment (CKNNA) between their vision and language representations than high-dispersion stimuli, frequently by a factor of up to two. This result is robust across both mean-residual and PC1-based selection criteria, and across all tested vision models and LLM pairings.

Figure 2: Vision-language alignment (CKNNA) for stimulus subsets binned by intra-modal dispersion, demonstrated across multiple vision and LLM pairings and selection criteria.

• Exclusion of high-dispersion stimuli systematically increases overall vision-to-language alignment, while exclusion of low-dispersion stimuli degrades it. This relation generalizes across four vision models and the full spectrum of LLMs. These trends are numerically large and statistically consistent.

Figure 3: Excluding high-dispersion stimuli (blue) improves CKNNA alignment across all vision-language pairings, while dropping low-dispersion stimuli (orange) degrades alignment relative to the full-set baseline (gray).

• Stimulus-level dispersion estimated from a subset of vision models predicts the held-out model's individual deviation from the GPA consensus. The partial-dispersion score for each stimulus robustly correlates (Pearson $r$ values 0.21–0.43, $p<0.001$) with the deviation of a withheld model from the joint space, indicating that dispersion is a stable, model-agnostic property.

Figure 4: Stimulus-level dispersion computed from three vision models predicts the deviation of a held-out model from the GPA consensus space, indicating cross-model relevance of the dispersion metric.

Implications and Theoretical Significance

These results supply stimulus-level evidence for the "Platonic Representation Hypothesis," which asserts that diverse neural networks trained on real-world data converge upon shared structures owing to inductive environmental priors. The finding that single-stimulus dispersion modulates cross-modal alignment suggests that convergence is not merely a global phenomenon but is sensitive to the specific representational characteristics of individual samples.

The practical implication is clear: representational alignment evaluations—and downstream tasks reliant on cross-modal transfer or grounding—should account for intra-modal dispersion at the stimulus level. Theoretically, the ability of dispersion measures to predict held-out model behavior indicates that dispersion is not simply an artifact of a particular architecture or objective, but reflects genuine structure relating to model universality and representational geometry. These properties are likely to generalize to broader classes of models and stimuli, potentially including biological neural systems.

Future Directions

The methodology outlined provides a principled route to understanding the sources of cross-modal divergence and convergence, and offers a stringent diagnostic tool for evaluating model-to-model and model-to-brain alignment at high granularity. Future work ought to focus on dissecting the factors—whether semantic, perceptual, or statistical—that render stimuli low- or high-dispersion, thereby opening paths to refined model training procedures, targeted dataset construction, or even alignment interpretability for neuroscience-grounded AI.

Expanding the GPA-based dispersion analysis to larger and more varied model pools, task domains, and real-world sensory data will be essential for uncovering general principles of representational universality. Investigating transfer to and from biological neural data stands as a particularly promising avenue.

Conclusion

The study demonstrates that single-stimulus, intra-modal representational dispersion, as quantified by Generalized Procrustes residuals, is a powerful, predictive determinant of cross-modal convergence between vision and LLMs. These findings refine current theoretical conceptions of representational universality and supply actionable insights for the design and assessment of both artificial and biological neural computation, with major ramifications for both practical cross-modal AI and the science of neural representation.