---
title: Modality Bias in Multimodal Learning
url: https://www.emergentmind.com/topics/modality-bias
type: topic
---

# Modality Bias in Multimodal Learning

Searching arXiv for recent papers on modality bias in multimodal learning.
Modality bias is a recurrent failure mode in multimodal learning in which one modality exerts disproportionate influence on prediction, representation, or evaluation while other available modalities are underused, ignored, or exploited only weakly. Recent work uses the term across vision-language models, audio-visual systems, multimodal retrieval, federated learning, medical VQA, entity alignment, and person re-identification, but not in a single uniform sense: it may denote asymmetric reliance at inference time, benchmark-level skew in which a supposedly multimodal task is solvable from one modality, a causal shortcut from one modality directly to the output, or a decision-layer weighting problem that persists after balanced encoder training [2505.18657][2510.14411][2408.12763]. In large vision-language models, the term has also been used for a hallucination regime in which the model may ignore not only visual information but also textual modality, indicating difficulty in simultaneously attending to both visual and textual inputs [2508.02419]. Taken together, these formulations treat modality bias as a family of imbalance phenomena spanning data, optimization, fusion, inference, and benchmarking.

## 1. Conceptual definitions and scope

In the most general formulation, modality bias arises when “certain modalities dominate the learning process, while others are underutilized or contribute less effectively,” with the practical consequences of over-reliance on dominant modalities, underutilization of some modalities, and decreased performance when the dominant modality is missing [2505.18657]. In audio-visual captioning, the same idea is operationalized as asymmetric dependence on one stream over the other: a model is audio-biased if caption quality degrades far more under audio corruption than under visual corruption [2510.24024]. In video question answering, modality bias may instead describe a dataset property: many questions can be answered from subtitles or video alone, so benchmark performance reflects shortcut use rather than multimodal reasoning [2408.12763].

Several papers sharpen the concept by identifying where the bias enters the system. One line of work treats it as a causal shortcut. In medical VQA, modality preference bias is framed as direct paths such as $q \rightarrow a$ or $v \rightarrow a$ that bypass multimodal knowledge formation $q, v \rightarrow k \rightarrow a$ [2505.16209]. In multi-modal entity alignment, visual modality bias is the direct causal effect of image features on predictions, producing a shortcut image-matching task rather than alignment mediated by graph information [2504.19458]. Another line of work argues that even if representation learning is balanced, bias can remain at the decision layer because modalities produce uncalibrated logits and unequal decision-weight distributions [2510.14411].

This diversity of definitions is substantive rather than terminological. Some papers diagnose bias as a property of models; others diagnose it as a property of datasets; still others locate it in the interaction between the two. A plausible implication is that modality bias is best treated as a multi-level phenomenon rather than a single pathology.

## 2. Sources and mechanisms

A repeated explanation is spurious modality-label correlation. In the systematic study of multimodal classification, one modality affects prediction more because it has a spurious correlation with instance labels; colored digits can be predicted from color instead of shape, VQA from question type instead of vision, and video action from static appearance instead of motion [2202.12690]. Closely related arguments appear in TVQA, where subtitles alone are highly informative, and in multimodal intent detection, where over 90% of samples require textual input either alone or in combination with other modalities [2012.10210][2508.16122]. These results identify bias as an economizing response to training distributions that reward shortcut use.

A second mechanism is asymmetry in modality difficulty and backbone capability. In video retrieval, the query-video text relation is easier to learn than query-vision relevance because query and video text are in the same modality; the paper names this the modality gap and separates it from data bias, namely that most search-log training pairs can be solved by text matching alone [2204.08182]. In MLLMs, language data is described as compact and abstract, visual data as redundant and complex, while pretrained language backbones are typically stronger and more mature than visual backbones; current training objectives then fail to enforce balanced cross-modal alignment, reinforcing language overreliance [2505.18657]. This combination of data asymmetry, model asymmetry, and objective asymmetry recurs across tasks.

A third mechanism is fusion-stage or decision-stage miscalibration. In audio-visual classification, decision-layer heatmaps on CREMAD and Kinetic-Sounds show persistent preference for audio features even after separate unimodal pretraining and decision-layer fine-tuning, which the authors attribute to intrinsic feature-space disparity, unequal decision-weight distributions, and uncalibrated logits [2510.14411]. In weakly supervised audio-visual video parsing, the weak supervision loss nearly coincides with the audio loss, and label smoothing on only one modality can collapse attention toward the other, showing that auxiliary training heuristics can intensify bias rather than reduce it [2203.16860]. In audio-visual speech recognition, video dropout shifts the model from a multimodal joint distribution toward a unimodal audio distribution in latent space, producing robustness to missing video at the cost of excessive audio bias on complete inputs [2403.04245].

These mechanisms are not mutually exclusive. Bias can begin as a dataset prior, become amplified by backbone asymmetry, and persist through fusion and decision layers even after encoders improve.

## 3. Measurement and diagnosis

The dominant diagnostic family is modality ablation or perturbation. In video QA, the modality importance score (MIS) formalizes whether a modality adds information to a question relative to alternative modality subsets:
$$
\mathrm{MIS}^{i}_{m_j} = \mathit{perf}(q_i \mid M_j^+) - \mathit{perf}(q_i \mid M_j^-).
$$
Positive MIS indicates useful signal, negative MIS indicates interference, and zero indicates no additional information beyond other modalities [2408.12763]. The same paper estimates MIS with GPT-4 Turbo and uses it to show that complementary questions are rare across TVQA, LifeQA, and AVQA.

Perturbation-based robustness tests provide a more direct estimate of asymmetric reliance. In audio-visual captioning, LAVCap on AudioCaps has clean-input CIDEr $= 0.805$, which drops to $0.685$ or $0.674$ under visual blanking but to $0.333$ under audio silence and $0.226$ under audio Gaussian noise; shuffled visual gives CIDEr $= 0.566$, whereas shuffled audio gives $0.140$ [2510.24024]. These numbers make the audio bias explicit. In sound localization, the diagnosis is framed psychophysically through congruent, conflicting, absent-cue, audio-only, vision-only, and multi-instance conditions, with separate Audio Accuracy and Vision Accuracy to distinguish physical localization from semantic category alignment [2505.11217].

Benchmark-level bias is often exposed by unimodal baselines and subset analysis. In TVQA, vision-only models answer about 45% of questions, while subtitles-only achieve about 68%; the full multimodal-only subset identified by the authors’ inclusion-exclusion analysis is 3.79% of the validation set [2012.10210]. In multimodal intent detection, human validation confirms that text-only is sufficient for 82.46% of sampled MIntRec-1 items and 80.63% of sampled MIntRec2.0 items, while automated analysis reports $\Sigma T = 91.01\%$ and $\Sigma T = 94.00\%$ respectively [2508.16122]. In VidQA, MIS-based categorization finds complementary questions at only 2.1% on TVQA, 2.4% on LifeQA, and 0.6% on AVQA [2408.12763].

A further diagnostic class measures representation or fusion imbalance directly. Video retrieval introduces
$$
R_{vt} = \frac{\cos \langle \mathcal{F}_v(v), \mathcal{F}_m(m) \rangle}{\cos \langle \mathcal{F}_t(t), \mathcal{F}_m(m) \rangle},
$$
where small values indicate that the fused video embedding aligns much more strongly with text than with vision [2204.08182]. In AVSR, transcript similarity to ASR outputs and latent representation similarity to unimodal audio states are used to quantify dropout-induced audio bias [2403.04245]. These diagnostics move beyond accuracy and ask how predictions are formed.

## 4. Modality-specific manifestations across tasks

Language-dominant bias is especially prominent in vision-language settings. The position paper on MLLMs argues that models often behave as language-anchored reasoners, showing higher consistency between complete input and text-only input than between complete input and image-only input [2505.18657]. TVQA displays strong subtitle dominance [2012.10210], VidQA benchmarks contain mostly subtitle-biased, video-biased, or modality-agnostic questions rather than complementary ones [2408.12763], MedVQA exhibits question-to-answer shortcuts that can remain hidden unless priors are changed across train and test [2505.16209], and temporal action localization with VLMs can be dominated by linguistic priors unless vision is preserved as the dominant signal [2601.21078].

Audio-dominant bias appears in several audio-visual tasks. LAVCap trained on AudioCaps is strongly audio-biased [2510.24024]. Decision-layer analysis on CREMAD and Kinetic-Sounds likewise finds that audio has larger weights and logits than video, and the model cannot automatically allocate decision-layer weights to match per-category modality capability [2510.14411]. In AVSR, the dominant modality is again audio, and aggressive video dropout increases robustness to missing frames precisely by increasing audio dominance [2403.04245].

Vision-dominant bias appears in other domains. In sound localization, current multimodal AI models often default to visual input under cross-modal conflict, degrading performance to near chance levels, whereas humans are more resilient to misleading or absent visuals by relying on auditory information [2505.11217]. In multi-modal entity alignment, over-reliance on image similarity becomes a shortcut image-matching task, especially problematic because on FB-DB15K, 86.3% of aligned pairs have image similarity below 0.5 and 43.7% are below 0.3 [2504.19458]. These findings are a reminder that modality bias is not always text-dominant; the dominant modality depends on task, data, and architecture.

Hallucination is a special manifestation. In LVLMs, object hallucination has usually been attributed to linguistic prior, but the abstract of the attention-lens paper argues for a broader modality bias in which models may ignore textual modality as well as visual information, leading to fragmented understanding of user instructions [2508.02419]. Preference-optimization work on MLLMs further links single-modality overreliance to irrelevant or hallucinated responses and reports that debiasing also reduces hallucination metrics [2503.17928].

## 5. Mitigation strategies

Data-centric mitigation changes what the model is rewarded for using. AudioVisualCaps augments AudioCaps with captions that jointly describe audio and visual streams; training LAVCap on AudioVisualCaps makes performance drops under missing or shuffled audio and missing or shuffled visual input more symmetric [2510.24024]. In MedVQA, SLAKE-CP and RadVQA-CP are constructed so that question-answer priors change between train and test, exposing models that rely on question shortcuts [2505.16209]. VidQA work proposes MIS-guided curation of modality-balanced datasets, while multimodal intent detection removes textually biased samples and highly biased intent categories to create debiased versions of MIntRec-1 and MIntRec2.0 [2408.12763][2508.16122].

Optimization and fusion strategies often aim to prevent dominant modalities from monopolizing the objective. Multi-Modal De-Bias (MMDB) replaces standard cross-entropy with an adaptive-margin cosine loss whose class-specific margin is derived from biased-modality statistics, improving OoD performance across colored-digit recognition, VQA-CP, and Kinetics OoD benchmarks [2202.12690]. In video retrieval, MBVR combines manually generated modality-shuffled negatives with a dynamic margin based on visual relevance to force the encoder to reject text-matched but visually wrong videos [2204.08182]. In multimodal federated learning, Balanced Modality Selection combines a modal enhancement loss with modality-level selection so that local training and global aggregation do not amplify a dominant modality [2401.00403].

Causal and counterfactual methods remove direct shortcut effects at inference or training time. In VQA and entity alignment, the core logic is
$$
TIE = TE - NDE,
$$
so prediction is based on indirect multimodal effects after subtracting the direct biased effect of a modality [2305.19664][2504.19458]. MedCFVQA uses counterfactual inference to estimate and subtract question-driven bias during inference [2505.16209]. In visible-infrared person re-identification, DMDL combines a causality-inspired adjustment intervention with collaborative bias-free training over data, labels, and features [2512.03745].

Adaptive weighting methods treat balance as context dependent rather than uniform. ActionVLM estimates Language Advantage, the incremental benefit of language over vision-only predictions, and uses it to reweight language features in a residual aggregation scheme where vision remains the base representation [2601.21078]. NaPO constructs language-biased and vision-biased rejected responses and uses a noise-aware preference optimization objective to make MLLMs less dependent on a single modality [2503.17928]. “Freeze and Reveal” applies targeted debiasing to one encoder at a time and shows that the dominant source of gender bias can differ by architecture: CLIP’s vision encoder is more biased, whereas PaliGemma2’s text encoder is more biased [2508.07432]. A plausible implication is that debiasing should often be modality-specific and architecture-aware rather than uniformly applied.

## 6. Evaluation principles, controversies, and open problems

A central unresolved issue is what “balance” should mean. Decision-layer analysis argues that balanced multimodal learning should not force identical weights across modalities; rather, relative balance should be promoted at the task or category level according to each modality’s capability [2510.14411]. This position complicates simple prescriptions such as equal weighting or equalized attention, since some tasks are intrinsically modality-asymmetric.

Another issue is that many multimodal benchmarks do not actually require multimodal reasoning. VidQA analysis concludes that roughly 89.8% to 94.8% of questions across the studied datasets can be answered using a single modality or are effectively modality-agnostic, with complementary questions only 0.6% to 2% [2408.12763]. Multimodal intent detection finds a similar textual skew [2508.16122], and TVQA’s multimodal-only subset is small relative to the full benchmark [2012.10210]. These results suggest that benchmark success can coexist with weak cross-modal integration, making modality bias partly an evaluation problem.

The current literature also treats explanation as incomplete. The MLLM position paper explicitly calls for systematic metrics, extension beyond vision-language settings to audio, tactile, video, and 3D or robotics-related modalities, and greater use of XAI to reveal internal mechanisms of modality dominance [2505.18657]. The abstract on LVLM hallucination suggests that even the standard visual-neglect explanation may be too narrow if models can also fail to retain textual instructions [2508.02419]. The resulting controversy is not whether modality bias exists, but whether it is primarily caused by data priors, backbone asymmetry, fusion design, optimization dynamics, or evaluation protocol. The evidence assembled across tasks indicates that all of these can matter, and that robust multimodal systems must therefore be designed and assessed at multiple levels simultaneously.

Source: https://www.emergentmind.com/topics/modality-bias