Modality Gap in Multimodal LLMs

• Modality gap is the systematic disparity in processing between text, image, and audio inputs in MLLMs, measured via cross-modal consistency metrics.
• It quantifies bias, representation misalignment, and performance differentials (e.g., up to >90 pp loss in image modality) due to imbalanced pretraining and architecture.
• Mitigation strategies such as prompt engineering, self-distillation, and multi-stage fusion offer actionable solutions to enhance cross-modal integration.

A modality gap in multimodal LLMs (MLLMs) is the systematic disparity in information processing, reasoning, and output quality between different input modalities—such as text, vision, audio, and speech—even when these inputs convey identical semantic content. This gap manifests across architectural, representation, and behavioral levels, and quantifying, analyzing, and mitigating it is a central challenge for unified multimodal intelligence.

1. Foundational Definitions and Taxonomy

A multimodal LLM accepts inputs in multiple modalities $M = \{ M_1, ..., M_n \}$, where each $M_i$ might represent text, image, audio, video, etc. The modality gap is observed as a failure of cross-modal consistency: for any information-preserving converter $K^{ab}_q: D_a \rightarrow D_b$, if $d_b = K^{ab}_q(d_a)$ then a cross-modally consistent model $M$ must satisfy $M(d_a, q) = M(d_b, q)$ for all $d_a \in D_a$, $q \in Q$. Systematic violations of this property—that is, output divergence across modalities for equivalent information—define the modality gap (Zhang et al., 2024).

The modality gap can also be formalized as bias toward a dominant modality, typically text, with under-utilization of others. This is quantified by the modality imbalance ratio: $$\Delta_{\mathrm{modality}} = \frac{C(M_{\mathrm{dominant}})}{C(M_{\mathrm{underutilized}})}$$ where $C(M_i)$ captures the dependence of the output on modality $M_i$ (Zheng et al., 24 May 2025).

Related notions include:

• Cross-modal skill composition gap: The deficit between model performance when skills are composed in sequence (e.g., OCR then reasoning) versus direct holistic inference, indicating failures in intra-model modularity (Ontalvilla et al., 11 Nov 2025).
• Representation gap: The persistent offset between modality-specific embeddings, often measured by cosine similarity or geometric separation in the unified feature space (Jiang et al., 2024, Yu et al., 2 Feb 2026).
• Neuron-level specialization: Distinct clusters of modality-specific neurons, with limited inter-modal information flow, indicating incomplete internal integration (Huang et al., 2024).

2. Quantitative Evaluation Frameworks

Comprehensive diagnosis of the modality gap uses controlled evaluation protocols:

• Cross-Modal Consistency Metric: For a set of task instances, measure the fraction $C_t = (1/n) \sum_{i=1}^n c_i$, where $c_i=1$ if $M(d^\text{text}_i, q) = M(d^\text{image}_i, q)$, else $0$ (Zhang et al., 2024).
• Performance Differential: $$\Delta_\text{acc} := \text{Acc}_\text{text} - \text{Acc}_\text{image}$$, giving absolute loss when content migrates from text to pixel modality (Sun et al., 10 Mar 2026).
• Modality Importance Score (MIS): For ablation, $$\mathrm{MIS}(i) = \frac{S_\text{full} - S_{-i}}{S_\text{full}}$$ measures performance drop when $M_i$ is removed (Zheng et al., 24 May 2025).
• Vision/Text Preference Ratio: In conflict scenarios, the VisionRatio is $S_\text{vision}/(S_\text{vision} + S_\text{text})$. Deviation from $0.5$ indicates bias (Zhang et al., 27 May 2025).
• Cross-layer CKA: Measures alignment between intermediate speech and text representations in speech-LLMs (Hsu et al., 2 Mar 2026).
• Neuron Attribution: Quantifies the fraction of modality-specific neurons in each layer and their impact on model accuracy (Huang et al., 2024).

3. Empirical Manifestations and Structural Origins

Modality gaps are widely reported in benchmark studies:

• GPT-4V shows up to $>90$ percentage point accuracy loss on Table Understanding when information is rendered as images instead of text, with cross-modal consistency as low as $0.10$. High OCR accuracy rules out perceptual failure; the loss is attributed to weaker reasoning in non-text modalities (Zhang et al., 2024).
• In Qwen2.5-VL, text-only consistency is $\sim56.5\%$, but image-only is $27\%$; removing text drops performance by $13-20$ p.p. (Zheng et al., 24 May 2025).
• Across speech-LLMs, reasoning from speech lags that from text by $4$–$14$ p.p.; CKA analysis reveals broad mid-layer misalignment due to speech redundancy, and naive geometric calibration does not close the gap (Hsu et al., 2 Mar 2026, Xiang et al., 14 Oct 2025).
• MLLMs struggle with cross-modal skill composition: cascaded inference (OCR → reasoning) can outperform end-to-end multimodal inference by $>20$ p.p. in open-source models (Ontalvilla et al., 11 Nov 2025).

Root causes include:

• Data imbalance: Text's semantic compactness and prevalence in pretraining cause dominance; visual/audio data is higher-dimensional and less frequent (Zheng et al., 24 May 2025).
• Imbalanced model capacity: Language sub-networks (typically transformers trained on huge corpora) can overpower less well-equipped vision/audio encoders.
• Component specialization: Mixture-of-experts and transformer blocks develop modality-specific subnetworks, with limited cross-modal neuron sharing (Huang et al., 2024).
• Objective mismatch: Pretraining tasks (contrastive, masked LM) often reinforce shortcuts in text; losses do not enforce equal weighting or integration (Zheng et al., 24 May 2025).
• Fusion limitations: Many models fuse modalities at only one layer (early or late), failing to provide the multi-stage alignment required for complex tasks (Zhang et al., 2024).

4. Bridging Strategies and Mitigation Techniques

Multiple families of techniques have emerged to diminish the modality gap:

• Prompt Engineering: Vision-Depicting-Prompting (VDP)—having the model first extract a textual summary from an image, then reason over both text and image—can raise image-mode accuracy by up to $90$ p.p. (Zhang et al., 2024). Carefully engineered prompts (e.g., "in one word" condensation) can collapse modality gaps in unified embedding models (Jiang et al., 2024).
• Data and Objective Engineering: Crafting benchmarks with high cross-modal dependency, using counterfactual samples, and balancing pretraining losses by modality; regularizers that penalize contribution imbalance (e.g., $$R_\text{modality} = \sum_i (C(M_i) - C_{\text{avg}})^2$$) (Zheng et al., 24 May 2025). Self-distillation—training on text-mode reasoning traces with image inputs—recover nearly all lost accuracy without catastrophic forgetting (Sun et al., 10 Mar 2026).
• Architectural Interventions:
  • Mixture-of-experts routing regularizers (SMAR) enforce balanced specialization across experts using KL-divergence between modality routing distributions, yielding high language retention and multimodal performance even with scarce text data (Xia et al., 6 Jun 2025).
  • Layer-wise, three-stage fusion in transformer architectures aligns coarse scene features early, object details mid-way, and aggregates for prediction late, supporting nuanced cross-modal alignment (Zhang et al., 2024).
• Unified Representation Learning: Text-centric pipelines transform all modalities into text using frozen bridge models and LLMs, training only the downstream text model—yielding zero-shot generalization across unseen modality combinations (Tsai et al., 2024).
• Representation Steering: Post-hoc representation engineering can explicitly shift latent activations along modality bias directions to induce vision or text preference, with applicability to downstream tasks such as hallucination mitigation and multimodal translation (Zhang et al., 27 May 2025).
• Skill Composition Training: Explicit chain-of-thought or pseudo-gold fine-tuning for sequential skill chaining narrows, though rarely closes, composition gaps for cross-modal tasks (Ontalvilla et al., 11 Nov 2025).

5. Structural Diagnosis, Limitations, and Open Questions

Mechanistic analyses reveal that cross-modal information transfer is largely confined to shallow or mid layers, with only a small subset of neurons mediating true cross-modal integration. Approximately $2\%$ of neurons can be modality-specific; deactivating these causes $>50\%$ drop in model accuracy (Huang et al., 2024). Cross-modal information flow is primarily hierarchical: coarse features are aligned early, object-level alignment follows, and late layers aggregate for answer synthesis (Zhang et al., 2024).

Limitations and open research areas include:

• Existing regularizers and tuning strategies often depend on static estimates of embedding distributions; adapting alignment dynamically during training remains unsolved (Yu et al., 2 Feb 2026).
• Extending advances beyond vision-text to more modalities (speech, audio, tactile) and more granular cross-modal tasks (dialogue, skill chaining) is an active direction (Zheng et al., 24 May 2025, Zhao et al., 2023).
• Many representation-steering and regularization techniques are decoder-only specific; generalization to encoder-decoder and autoregressive models remains open (Zhang et al., 27 May 2025).
• Causal and mechanistic interpretability tools (attention knockout, relevance propagation) are needed to ensure robust cross-modal reasoning corresponds to human understanding (Zhang et al., 2024, Chen et al., 28 Nov 2025).
• Most solutions address average-case performance, but models remain brittle under adversarial cross-modal association or distribution shifts (Sun et al., 10 Mar 2026, Cai et al., 26 May 2025).

6. Benchmarks and Comparative Results

Model/Strategy	Key Metric (example)	Observed Modality Gap (↓ is better)
GPT-4V (plain img ↔ txt)	Table understanding Cₜ	0.10 (consistency); image acc 3% vs. txt 93% (Zhang et al., 2024)
Qwen2.5-VL (MMMU-Pro)	Image-only consistency	~27%–28% (vs. text-only ~56%) (Zheng et al., 24 May 2025)
Self-Distillation (GSM8K img)	Accuracy increase (img mode)	30.71%→92.72% (closing Δ_acc to 1.4pp) (Sun et al., 10 Mar 2026)
E5-V (retrieval R@1, COCO)	CLIP baseline vs. E5-V	37.0%→52.0% (+15pp), single-modality training (Jiang et al., 2024)
SMAR (MoE)	Language retention ratio	86.6% (2.5% pure-text), much higher than baselines (Xia et al., 6 Jun 2025)
ChatBridge (multi-modal)	NoCaps captioning (CIDEr)	115.7 vs. BLIP-2 (FlanT5-XXL) 103.9 (Zhao et al., 2023)

7. Prospects for Bridging the Modality Gap

Closing the modality gap demands a blend of architectural flexibility, data diversity, systematic regularization, and mechanistic transparency. Unified embedding models with prompt-based alignment, dynamic mixture-of-experts assignment, explicit cross-modal regularizers, and rigorous cross-modal evaluation must become standard. Incorporation of adaptation strategies (representation steering, self-distillation, curriculum learning) and expansion to broader and more challenging benchmarks (e.g., multi-step reasoning, rare modality fusion, adversarial robustness) define current research frontiers (Zhang et al., 2024, Zheng et al., 24 May 2025, Zhang et al., 27 May 2025, Chen et al., 28 Nov 2025).

A principled solution likely requires integrating multi-stage fusion, dynamic alignment objectives, and cross-modal interpretability into both training and evaluation—paving the way toward genuinely unified multimodal reasoning systems.