Cross-Modal Redundancy and the Geometry of Vision-Language Embeddings

Abstract: Vision-LLMs (VLMs) align images and text with remarkable success, yet the geometry of their shared embedding space remains poorly understood. To probe this geometry, we begin from the Iso-Energy Assumption, which exploits cross-modal redundancy: a concept that is truly shared should exhibit the same average energy across modalities. We operationalize this assumption with an Aligned Sparse Autoencoder (SAE) that encourages energy consistency during training while preserving reconstruction. We find that this inductive bias changes the SAE solution without harming reconstruction, giving us a representation that serves as a tool for geometric analysis. Sanity checks on controlled data with known ground truth confirm that alignment improves when Iso-Energy holds and remains neutral when it does not. Applied to foundational VLMs, our framework reveals a clear structure with practical consequences: (i) sparse bimodal atoms carry the entire cross-modal alignment signal; (ii) unimodal atoms act as modality-specific biases and fully explain the modality gap; (iii) removing unimodal atoms collapses the gap without harming performance; (iv) restricting vector arithmetic to the bimodal subspace yields in-distribution edits and improved retrieval. These findings suggest that the right inductive bias can both preserve model fidelity and render the latent geometry interpretable and actionable.

• The paper introduces the Iso-Energy Assumption and Aligned Sparse Autoencoder to recover shared bimodal features across modalities.
• It demonstrates that removing unimodal atoms closes the modality gap without impairing retrieval or zero-shot classification performance.
• Empirical validation across VLMs shows significant improvements in reconstruction, probing accuracy, and OOD detection through targeted geometric regularization.

Geometric Decomposition and Cross-Modal Redundancy in Vision–Language Embeddings

Introduction and Motivation

This paper addresses the geometric and conceptual structure of vision–LLM (VLM) embedding spaces, specifically in dual-encoder architectures trained contrastively on paired image–text data. While state-of-the-art VLMs (e.g., CLIP, SigLIP) achieve strong alignment between images and texts at the representation level, the interpretability of the latent geometry remains poorly understood. Prior empirical findings consistently identify a modality gap: image and text embeddings occupy distinct but linearly separable cones, with most extracted conceptual features being unimodal and activating predominantly in only one modality. This work provides a formal, generative perspective that explains this phenomenon and offers actionable approaches for representation disentanglement.

Iso-Energy Assumption and the Aligned Sparse Autoencoder

The core theoretical contribution is the Iso-Energy Assumption: truly cross-modal concepts, if present, must manifest with consistent activation energy (i.e., expected squared code activation) across modalities. The authors formalize a generative process in which underlying concept vectors are mapped through modality-dependent generators, and the observed embedding is a sparse linear combination of abstract conceptual atoms.

Figure 1: Multimodal data-generating process with latent concepts rendered through image or text generators and mapped to a shared embedding via a dual-encoder.

Recovering these concepts from nonlinear mappings is ill-posed without additional constraints. Classical sparse autoencoders (SAEs), commonly used for concept discovery in both vision and LLMs, identify overcomplete linear codes but lack a constraint tying corresponding conceptual directions between modalities. This often results in dictionaries with predominantly modality-specific (unimodal) atoms. To address this, the authors propose the Aligned Sparse Autoencoder (SAEA): a semi-supervised matching-pursuit SAE augmented with a soft penalty that encourages across-modality energy consistency for each feature. This induces a geometric bias toward recovering bimodal (shared) concepts.

Empirical Validation

The new inductive bias is tested on both synthetic data and large-scale VLM embeddings. In synthetic setups with known underlying concept structure, the alignment penalty enables recovery of ground-truth shared features only in scenarios where true cross-modal redundancy exists. Crucially, the penalty does not hallucinate unwarranted shared concepts if the underlying data is not redundant—neutrality is preserved.

Across six representative VLMs, the SAEA achieves similar or superior reconstruction (MSE, $R^2$) while producing dictionaries with systematically higher proportions of bimodal atoms, as assessed by geometric, functional, and distributional alignment metrics.

Figure 2: Most learned features in the SAEA are bimodal with medium activation energy, while only a handful of high-energy unimodal features dominate each modality.

Notable results:

• Functional alignment ($\rho$): increases by more than an order of magnitude after alignment regularization, indicating that almost the entire cross-modal alignment signal is carried by a compact set of bimodal atoms.
• Probing accuracy ($p_{\mathrm{acc}}$): improves systematically, showing increased distinguishability between modality-specific and shared directions.
• Ablation of unimodal atoms collapses the modality gap (as measured by mean separation, Wasserstein distance, and out-of-distribution KNN statistics) with no measurable impact on retrieval performance or zero-shot classification, directly contradicting prior results where mean-shifting or linear projection substantially degrade cross-modal performance.

  Figure 3: Modality gap arises from several unimodal concepts, while bimodal concepts are sufficient for cross-modal alignment; removing unimodal atoms via a mask closes the gap.

  

  Figure 4: Filtering unimodal atoms merges image and text distributions, outperforming the embedding shift method which only shifts means but does not align distributions fully.

Latent Concept Geometry and Interpretability

Low-dimensional projections of the dictionaries reveal three geometric clusters: image-only, text-only, and bimodal, respectively aligned with modality cones and their orthogonal complement.

Figure 5: Example activations for high and low-energy bimodal concepts—demonstrating semantic fidelity across both modalities.

(Figure 6, Figure 7, Figure 8, Figure 9)

Figure 6: Image-only features, activating for covers/devices/logos, with minor activation in text.

Figure 7: More image-only concepts (covers, sports, famous figures) with non-coincident text activation.

Figure 8: Text-only features (news/scam/news), inactive in image space.

Figure 9: Further text-only features (birthdays, holidays, name-dropping), semantically specific and absent in images.

High-energy unimodal atoms are shown to cluster, often forming 'modality biases' that are nearly global offsets, while residual, lower-energy unimodal atoms encode subtle, often uninterpretable modality-specific features. In contrast, bimodal atoms, isolated by the Iso-Energy regularization, consistently encode cross-modal, semantically aligned features (e.g., 'blue', 'cat', 'running').

Actionable Interventions and Downstream Implications

Modality Gap Elimination and Retrieval Fidelity

By constructing binary masks to remove all unimodal components from embeddings, the authors achieve full elimination of the modality gap. Unlike previous approaches which project out canonical bias directions or shift means—both of which degrade retrieval—this approach ensures that the shared (bimodal-only) embedding retains all cross-modal ranking structure:

• For any content-preserving retrieval query, the removal of modality-specific components that live in orthogonal subspaces does not affect the relative cosine similarities (see theoretical supplement on ranking preservation).
• The recombined bimodal-only embeddings support accurate semantic vector arithmetic, maintaining in-distribution queries and improved retrieval (e.g., color modification on FashionIQ).

  Figure 10: Classical edit vectors for concept arithmetic are corrupted by unimodal noise, while arithmetic restricted to bimodal features yields in-distribution queries and correct retrieval.

Out-of-Distribution Detection

K-nearest-neighbor distances for OOD detection confirm that classical arithmetic in the full embedding space often yields queries that fall between the image and text cones (off-manifold), whereas operations in the bimodal backbone remain within the support of valid visual embeddings.

Metric Invariance and Theoretical Guarantees

The orthogonality of modality and shared subspaces ensures that content-based ranking and classification scores are provably invariant to the ablation of varying unimodal atoms, provided the cross-modal (bimodal) content is the key information. The authors provide formal proofs and synthetic counterexamples clarifying when this invariance fails (i.e., highly adaptive, non-orthogonal bias structures).

Theoretical and Practical Implications

The findings articulate a strong, actionable claim: in large-scale dual-encoder VLMs, cross-modal alignment is supported exclusively by a compact, easily isolatable set of bimodal features; all other capacity is devoted to encoding modality-specific information. This geometric separation renders the latent space linearly and functionally disentanglable—a valuable property for post-hoc control, interpretability, and downstream safety guarantees.

From a practical perspective, this enables:

• Post-hoc debiasing or harmonization of visual and textual embeddings without retraining.
• Controlled semantic edits that preserve conceptual fidelity and manifold support.
• Improved OOD robustness for retrieval and compositional reasoning tasks.

Theoretically, by grounding interpretability in probabilistic generative models and energy-based criteria, the paper bridges conceptual extraction with geometric regularization, providing a basis for consistent feature recovery across modalities and seeds. This approach may generalize to other multimodal, contrastive, or even generative architectures as they scale in alignment coverage.

Conclusion

This work provides a rigorous geometric, statistical, and functional analysis of cross-modal representation learning in dual-encoder VLMs. Through the Iso-Energy Assumption and aligned sparse coding, the latent structure is decomposed into unimodal (modality-specific) and bimodal (shared, alignment-supporting) atoms, enabling direct interventions and interpretability without loss of downstream capability.

The results challenge prior beliefs about the necessity of retaining modality-specific biases for performance and demonstrate that targeted regularization can yield representations that are not only high-performing, but also amenable to fine-grained manipulation and mechanistic insight. Limitations include sensitivity to penalty tuning and the focus on retrieval-based architectures; further work is warranted to test whether similar decompositions exist in unified or generative multimodal models.

---

Reference:

"Cross-Modal Redundancy and the Geometry of Vision-Language Embeddings" (2602.06218)