Modality Gap Elimination

• Modality gap elimination is a set of techniques addressing systematic geometric and statistical discrepancies between multimodal embeddings.
• Strategies like explicit basis reallocation, cross-modal knowledge distillation, and embedding standardization improve fusion, retrieval, and classification performance.
• Applications in healthcare, multimodal retrieval, and safe VLMs demonstrate practical gains, including improved NDCG, Dice scores, and reduced modality collapse.

Modality gap elimination encompasses a family of methodologies and theoretical insights designed to bridge systematic representational discrepancies between different sensor or data modalities within multimodal machine learning systems. The persistence of the modality gap—notably between representations from, for example, vision and language encoders—has been shown to undermine downstream performance in fusion, retrieval, clustering, transfer, and robustness, and to be implicated in issues such as modality collapse and loss of safety in large multimodal LLMs. The literature identifies the root causes of the gap, quantifies it with formal metrics, and supplies both algorithmic and architectural remedies for its removal or minimization. The following sections synthesize the current state of knowledge from empirical, theoretical, and applied work across domains.

1. Theoretical Characterization and Measurement of the Modality Gap

The modality gap denotes the systematic geometric and statistical separation between the embedding distributions of distinct modalities in a supposedly shared latent space. In two-encoder architectures (such as CLIP), the gap is typically quantified as the $\ell_2$ (Euclidean) distance between the mean (centroid) vectors of the modalities, $\|\mu_{\rm image} - \mu_{\rm text}\|_2$, or as a “gap vector” $c_\perp$ orthogonal to the shared semantic subspace. Additional metrics include Fréchet Inception Distance (FID), cross-modal alignment scores, and discriminability of modality labels via a linear classifier on the joint space (Chaudhuri et al., 28 May 2025, Li et al., 25 Jul 2025, Yaras et al., 2024, An et al., 2024, Liang et al., 2022, Fahim et al., 2024).

Random-initialized encoders create narrow “cones” in the embedding hypersphere, and contrastive objectives reinforce a persistent gap contingent on the contrastive loss temperature. This geometric separation is robust under standard InfoNCE or NT-Xent losses, and is only marginally alleviated by increased training or naive parameter modifications (Yaras et al., 2024, Liang et al., 2022, Fahim et al., 2024).

In multimodal fusion networks, the modality gap can be traced to the interaction of modality collapse with low-rank simplicity bias in the shared fusion head: cross-modal polysemantic neurons entangle predictive and noisy features, entrenching the overshadowing of weaker modalities and exacerbating the modality gap relative to unimodal performance (Chaudhuri et al., 28 May 2025).

2. Mechanisms Underlying the Modality Gap and Its Persistence

Analyses converge on several root causes:

• Contrastive Objective Geometry: Symmetric contrastive losses and hard negative sampling forcibly separate modality clusters, yielding a residual gap even at optimization minima (Yaras et al., 2024, Liang et al., 2022, Fahim et al., 2024). Decreasing the temperature parameter in InfoNCE sharpens this effect.
• Model Initialization and Encoder Design: Separate encoders coupled with non-orthogonal parameter initializations produce non-overlapping manifolds (“cone effect”), further amplified by activation nonlinearities such as ReLU (Liang et al., 2022, Fahim et al., 2024).
• SGD and Low-rank Bottleneck: Stochastic gradient descent concentrates modality signals in shared low-rank subspaces, creating polysemantic entanglements that preclude the emergence of monosemantic dimensions for each modality and precipitate modality collapse (Chaudhuri et al., 28 May 2025).
• Misalignment in Conditional Distributions: Class-conditional distributions for the same semantic class remain shifted and dispersed across modalities, undermining category-level and instance-level fusion (Zhao et al., 3 Dec 2025, Xu et al., 2024).
• Residual Modality-specific Statistical Biases: After embedding normalization, certain dimensions exhibit modality-specific magnitude and bias, impeding cosine similarity maximization even for matched pairs (An et al., 2024).

3. Algorithmic and Architectural Strategies for Modality Gap Elimination

A diverse set of strategies has been distilled and validated, including both model-internal corrective losses and post-hoc transformations:

Method	Mechanism	Notable Impact/Domain
Explicit Basis Reallocation (EBR) (Chaudhuri et al., 28 May 2025)	Adversarially reallocates latent bases via a bottleneck and modality-discriminator, maximizing semantic content retention while hiding modality identity	Recovers predictive signals and prevents collapse in noisy/missing modalities
Cross-Modal Knowledge Distillation (KD) (Chaudhuri et al., 28 May 2025, Lu et al., 2024)	Denoises weaker modalities by distilling representations from dominant modalities, freeing rank bottlenecks	Fused representations retain high rank; enhances robustness to missing input
Linear or Residual Mapping (Yang et al., 2024)	Projects one modality into the other's space with linear transformation, combined with local triplet loss	Enables text embeddings to serve as accurate prototypes for image classification
Embedding Standardization (I0T) (An et al., 2024)	Post-hoc centering and per-modality batch normalization collapses the gap in zero-shot or frozen-encoder scenarios	DRAMATIC reduction in centroid distance (e.g., $\sim$0.76$\to$0.01), preserves classification/retrieval performance
Post-hoc Centroid or Spectral Alignment (Li et al., 25 Jul 2025, Role et al., 6 May 2025, Yu et al., 2 Feb 2026)	Global centering (subtracting mean vectors) and spectral/optimal transport warping aligns global modality distributions	>70–90% cross-modal recall in embedding-based search from zero
Contrastive Regularization (Alignment + Uniformity) (Fahim et al., 2024, Grassucci et al., 26 Jan 2026)	Explicitly penalizes modality centroid distances and encourages uniform sphere occupancy; sometimes includes cross-modal uniformity terms	Closes the gap while maintaining or improving zero-shot and multimodal arithmetic tasks
Modality Sampling and Single-Branch Networks (Ganhör et al., 23 Sep 2025)	Weight-sharing and random modality drop during training forces all modalities into a unified embedding space robust to missing data	Gains of up to 40% NDCG for missing modality recommendation
Missing Modality Token (MMT) (Ramazanova et al., 2024)	Add learnable placeholder tokens for missing modalities in sequence models, trained with stochastic modality dropout	Shrinks performance drop in partial-modal inference from $\sim$30% to $\sim$10%
Structure-Preserving Diffusion Mapping (Nam et al., 13 Oct 2025, Zhao et al., 3 Dec 2025)	Diffusion models or Brownian-bridge kernels project one modality into another, preserving geometric relationships	Substantial improvements in cross-modal retrieval/captioning
Regional/Prototype Alignment (Xu et al., 2024)	Learn class-specific prototypes in the vision space using divergence to text prototypes and region-level contrastive alignment	+4–5% mIoU on semantic segmentation benchmarks

Practitioners select methods based on tradeoff between retraining requirement, target domain, architectural freeze, and degree of invariance demanded.

4. Modality Gap Elimination in Key Application Domains

Multimodal Retrieval and Mixed Modality Search: Calibration methods (e.g., GR-CLIP) center embeddings by subtracting modality means, achieving up to +26 percentage point increase in NDCG@10 on real-world mixed-corpus retrieval datasets (Li et al., 25 Jul 2025).

Healthcare and Scientific Imaging: Explicit latent-feature alignment (via KL divergence or MSE to a learned anchor distribution) yields modality-invariant latent spaces for robust segmentation, with an average +1.75 Dice improvement in missing-modality brain tumor segmentation (Liu et al., 2024).

Safe and Robust VLMs: Directly regularizing the input-projection alignment in vision-LM pretraining (using MIR/FID and L2-matching) is highly correlated with safety performance; ReGap achieves a 16–18% reduction in unsafe outputs with no utility drop (Yang et al., 30 May 2025).

Recommendation Systems: Single-branch architectures with contrastive intra-item regularization massively narrow modality cluster separation, preserving recommendation accuracy under cold-start and missing-modality scenarios (Ganhör et al., 23 Sep 2025).

Weakly-supervised/Low-shot Vision: Prototype learning and cross-modal mapping enable text-based prototypes to serve as strong classifiers for image features, closing the gap and improving top-1 accuracy by up to 5% across datasets (Yang et al., 2024, Xu et al., 2024).

Speech–Text Reasoning: Dense representation and behavior alignment rewards in reinforcement learning eliminate the disparity in reasoning performance between speech and text input in LLMs, raising the Modality Recovery Rate to $\sim$100% (Wang et al., 9 Jan 2026).

5. Task-specific Implications and Broader Impact

The necessity of modality gap elimination is task-dependent. Standard instance-retrieval objectives (Recall@K) are insensitive to absolute gap (preserving only relative orderings), but group-wise tasks such as clustering, semantic grouping, fusion for complex reasoning, and continual learning exhibit vastly inflated within-class scatter and degraded inference when the gap persists (Grassucci et al., 26 Jan 2026, Huang et al., 12 Jul 2025). Preserving the “correct” amount of gap is also important in continual learning, where the negative similarity between modalities serves as a drift-monitor for retaining zero-shot generalization (Huang et al., 12 Jul 2025).

In multimodal LLM scaling, precise subspace alignment (as in ReAlign and ReVision) allows pretraining on unpaired data, reducing the necessity for costly aligned corpora and increasing efficiency (Yu et al., 2 Feb 2026). For large-scale multimodal models, direct regularization during pretraining is more effective and robust than post-hoc calibration.

6. Best Practices and Implementation Guidelines

The literature provides clear recommendations for minimizing or eliminating the modality gap:

• Monitor the centroid gap, inter-modal similarity, and within-class cosine (CosTP) metrics throughout training (Yaras et al., 2024, An et al., 2024, Li et al., 25 Jul 2025).
• Use temperature control or explicit regularization when training with contrastive objectives; do not let the temperature collapse to near zero (Yaras et al., 2024, Liang et al., 2022, Fahim et al., 2024).
• For fusion architectures, integrate adversarial or contrastive discriminators to force modality invariance within the bottleneck representation (Chaudhuri et al., 28 May 2025).
• For domain-agnostic fusion, apply post-hoc centering (subtracting modality means), spectral embedding, or optimal transport-based warping on batch or dataset-wide embeddings (Role et al., 6 May 2025, An et al., 2024).
• In missing-modality settings, combine stochastic dropout and missing-modality placeholders in training, ensuring the model never encounters missing scenarios for the first time at inference (Ramazanova et al., 2024).
• For VLM safety and robustness, regularize the token-input space during pretraining to enforce geometry alignment between modalities (Yang et al., 30 May 2025).
• In cross-modal generation or translation, enforce geometry-preserving diffusion or bridge modules to anchor both modalities within a unified distribution (Nam et al., 13 Oct 2025, Zhao et al., 3 Dec 2025).

7. Outlook and Open Problems

Despite substantial progress, challenges persist in scalability of gap-elimination methods to high-dimensional, many-modal, or open-world settings. Spectral and optimal-transport-based post-processing become computationally expensive beyond $n \sim 10^4$, and prototype-based alignment relies on high-quality class-specific seed prompts or masks (Role et al., 6 May 2025, Xu et al., 2024). Fully unified architectures that preserve cross-modal alignment under continual updates and substantial domain shift remain an open area of research. The role of the modality gap as a regularizer (or, conversely, an artifact to be eliminated) is also context-dependent, and its impact on model brittleness, fairness, and safety in high-stakes applications warrants continued investigation.

---

The modality gap constitutes both a geometric and a practical barrier to seamless multimodal integration. Its elimination—or precise control—is essential for the next generation of robust, equitable, and high-performing multimodal systems across retrieval, classification, segmentation, safety-critical deployment, and learning under missing or noisy modalities (Chaudhuri et al., 28 May 2025, Li et al., 25 Jul 2025, Yaras et al., 2024, Grassucci et al., 26 Jan 2026).