Modality Bias in Multimodal Learning

• Modality bias is an imbalance phenomenon in multimodal learning where one modality disproportionately influences predictions while others are underutilized.
• It results from factors including data priors, backbone asymmetry, fusion miscalibration, and causal shortcuts that simplify decision-making.
• Empirical studies suggest that mitigating modality bias requires targeted data augmentation, counterfactual inference, and adaptive optimization strategies.

Searching arXiv for papers on modality bias in multimodal learning. Modality bias is a recurrent failure mode in multimodal learning in which one modality exerts disproportionate influence on prediction, representation, or evaluation while other available modalities are underused, ignored, or exploited only weakly. Recent work uses the term across vision-LLMs, audio-visual systems, multimodal retrieval, federated learning, medical VQA, entity alignment, and person re-identification, but not in a single uniform sense: it may denote asymmetric reliance at inference time, benchmark-level skew in which a supposedly multimodal task is solvable from one modality, a causal shortcut from one modality directly to the output, or a decision-layer weighting problem that persists after balanced encoder training (Zheng et al., 24 May 2025, Ma et al., 16 Oct 2025, Park et al., 2024). In large vision-LLMs, the term has also been used for a hallucination regime in which the model may ignore not only visual information but also textual modality, indicating difficulty in simultaneously attending to both visual and textual inputs (Zheng et al., 4 Aug 2025). Taken together, these formulations treat modality bias as a family of imbalance phenomena spanning data, optimization, fusion, inference, and benchmarking.

1. Conceptual definitions and scope

In the most general formulation, modality bias arises when “certain modalities dominate the learning process, while others are underutilized or contribute less effectively,” with the practical consequences of over-reliance on dominant modalities, underutilization of some modalities, and decreased performance when the dominant modality is missing (Zheng et al., 24 May 2025). In audio-visual captioning, the same idea is operationalized as asymmetric dependence on one stream over the other: a model is audio-biased if caption quality degrades far more under audio corruption than under visual corruption (Ishikawa et al., 28 Oct 2025). In video question answering, modality bias may instead describe a dataset property: many questions can be answered from subtitles or video alone, so benchmark performance reflects shortcut use rather than multimodal reasoning (Park et al., 2024).

Several papers sharpen the concept by identifying where the bias enters the system. One line of work treats it as a causal shortcut. In medical VQA, modality preference bias is framed as direct paths such as $q \rightarrow a$ or $v \rightarrow a$ that bypass multimodal knowledge formation $q, v \rightarrow k \rightarrow a$ (Ye et al., 22 May 2025). In multi-modal entity alignment, visual modality bias is the direct causal effect of image features on predictions, producing a shortcut image-matching task rather than alignment mediated by graph information (Su et al., 28 Apr 2025). Another line of work argues that even if representation learning is balanced, bias can remain at the decision layer because modalities produce uncalibrated logits and unequal decision-weight distributions (Ma et al., 16 Oct 2025).

This diversity of definitions is substantive rather than terminological. Some papers diagnose bias as a property of models; others diagnose it as a property of datasets; still others locate it in the interaction between the two. A plausible implication is that modality bias is best treated as a multi-level phenomenon rather than a single pathology.

2. Sources and mechanisms

A repeated explanation is spurious modality-label correlation. In the systematic study of multimodal classification, one modality affects prediction more because it has a spurious correlation with instance labels; colored digits can be predicted from color instead of shape, VQA from question type instead of vision, and video action from static appearance instead of motion (Guo et al., 2022). Closely related arguments appear in TVQA, where subtitles alone are highly informative, and in multimodal intent detection, where over 90% of samples require textual input either alone or in combination with other modalities (Winterbottom et al., 2020, Mullick et al., 22 Aug 2025). These results identify bias as an economizing response to training distributions that reward shortcut use.

A second mechanism is asymmetry in modality difficulty and backbone capability. In video retrieval, the query-video text relation is easier to learn than query-vision relevance because query and video text are in the same modality; the paper names this the modality gap and separates it from data bias, namely that most search-log training pairs can be solved by text matching alone (Wang et al., 2022). In MLLMs, language data is described as compact and abstract, visual data as redundant and complex, while pretrained language backbones are typically stronger and more mature than visual backbones; current training objectives then fail to enforce balanced cross-modal alignment, reinforcing language overreliance (Zheng et al., 24 May 2025). This combination of data asymmetry, model asymmetry, and objective asymmetry recurs across tasks.

A third mechanism is fusion-stage or decision-stage miscalibration. In audio-visual classification, decision-layer heatmaps on CREMAD and Kinetic-Sounds show persistent preference for audio features even after separate unimodal pretraining and decision-layer fine-tuning, which the authors attribute to intrinsic feature-space disparity, unequal decision-weight distributions, and uncalibrated logits (Ma et al., 16 Oct 2025). In weakly supervised audio-visual video parsing, the weak supervision loss nearly coincides with the audio loss, and label smoothing on only one modality can collapse attention toward the other, showing that auxiliary training heuristics can intensify bias rather than reduce it (Pasi et al., 2022). In audio-visual speech recognition, video dropout shifts the model from a multimodal joint distribution toward a unimodal audio distribution in latent space, producing robustness to missing video at the cost of excessive audio bias on complete inputs (Dai et al., 2024).

These mechanisms are not mutually exclusive. Bias can begin as a dataset prior, become amplified by backbone asymmetry, and persist through fusion and decision layers even after encoders improve.

3. Measurement and diagnosis

The dominant diagnostic family is modality ablation or perturbation. In video QA, the modality importance score (MIS) formalizes whether a modality adds information to a question relative to alternative modality subsets:

$$\mathrm{MIS}^{i}_{m_j} = \mathit{perf}(q_i \mid M_j^+) - \mathit{perf}(q_i \mid M_j^-).$$

Positive MIS indicates useful signal, negative MIS indicates interference, and zero indicates no additional information beyond other modalities (Park et al., 2024). The same paper estimates MIS with GPT-4 Turbo and uses it to show that complementary questions are rare across TVQA, LifeQA, and AVQA.

Perturbation-based robustness tests provide a more direct estimate of asymmetric reliance. In audio-visual captioning, LAVCap on AudioCaps has clean-input CIDEr $= 0.805$, which drops to $0.685$ or $0.674$ under visual blanking but to $0.333$ under audio silence and $0.226$ under audio Gaussian noise; shuffled visual gives CIDEr $= 0.566$, whereas shuffled audio gives $v \rightarrow a$0 (Ishikawa et al., 28 Oct 2025). These numbers make the audio bias explicit. In sound localization, the diagnosis is framed psychophysically through congruent, conflicting, absent-cue, audio-only, vision-only, and multi-instance conditions, with separate Audio Accuracy and Vision Accuracy to distinguish physical localization from semantic category alignment (Jia et al., 16 May 2025).

Benchmark-level bias is often exposed by unimodal baselines and subset analysis. In TVQA, vision-only models answer about 45% of questions, while subtitles-only achieve about 68%; the full multimodal-only subset identified by the authors’ inclusion-exclusion analysis is 3.79% of the validation set (Winterbottom et al., 2020). In multimodal intent detection, human validation confirms that text-only is sufficient for 82.46% of sampled MIntRec-1 items and 80.63% of sampled MIntRec2.0 items, while automated analysis reports $v \rightarrow a$1 and $v \rightarrow a$2 respectively (Mullick et al., 22 Aug 2025). In VidQA, MIS-based categorization finds complementary questions at only 2.1% on TVQA, 2.4% on LifeQA, and 0.6% on AVQA (Park et al., 2024).

A further diagnostic class measures representation or fusion imbalance directly. Video retrieval introduces

$v \rightarrow a$3

where small values indicate that the fused video embedding aligns much more strongly with text than with vision (Wang et al., 2022). In AVSR, transcript similarity to ASR outputs and latent representation similarity to unimodal audio states are used to quantify dropout-induced audio bias (Dai et al., 2024). These diagnostics move beyond accuracy and ask how predictions are formed.

4. Modality-specific manifestations across tasks

Language-dominant bias is especially prominent in vision-language settings. The position paper on MLLMs argues that models often behave as language-anchored reasoners, showing higher consistency between complete input and text-only input than between complete input and image-only input (Zheng et al., 24 May 2025). TVQA displays strong subtitle dominance (Winterbottom et al., 2020), VidQA benchmarks contain mostly subtitle-biased, video-biased, or modality-agnostic questions rather than complementary ones (Park et al., 2024), MedVQA exhibits question-to-answer shortcuts that can remain hidden unless priors are changed across train and test (Ye et al., 22 May 2025), and temporal action localization with VLMs can be dominated by linguistic priors unless vision is preserved as the dominant signal (Li et al., 28 Jan 2026).

Audio-dominant bias appears in several audio-visual tasks. LAVCap trained on AudioCaps is strongly audio-biased (Ishikawa et al., 28 Oct 2025). Decision-layer analysis on CREMAD and Kinetic-Sounds likewise finds that audio has larger weights and logits than video, and the model cannot automatically allocate decision-layer weights to match per-category modality capability (Ma et al., 16 Oct 2025). In AVSR, the dominant modality is again audio, and aggressive video dropout increases robustness to missing frames precisely by increasing audio dominance (Dai et al., 2024).

Vision-dominant bias appears in other domains. In sound localization, current multimodal AI models often default to visual input under cross-modal conflict, degrading performance to near chance levels, whereas humans are more resilient to misleading or absent visuals by relying on auditory information (Jia et al., 16 May 2025). In multi-modal entity alignment, over-reliance on image similarity becomes a shortcut image-matching task, especially problematic because on FB-DB15K, 86.3% of aligned pairs have image similarity below 0.5 and 43.7% are below 0.3 (Su et al., 28 Apr 2025). These findings are a reminder that modality bias is not always text-dominant; the dominant modality depends on task, data, and architecture.

Hallucination is a special manifestation. In LVLMs, object hallucination has usually been attributed to linguistic prior, but the abstract of the attention-lens paper argues for a broader modality bias in which models may ignore textual modality as well as visual information, leading to fragmented understanding of user instructions (Zheng et al., 4 Aug 2025). Preference-optimization work on MLLMs further links single-modality overreliance to irrelevant or hallucinated responses and reports that debiasing also reduces hallucination metrics (Zhang et al., 23 Mar 2025).

5. Mitigation strategies

Data-centric mitigation changes what the model is rewarded for using. AudioVisualCaps augments AudioCaps with captions that jointly describe audio and visual streams; training LAVCap on AudioVisualCaps makes performance drops under missing or shuffled audio and missing or shuffled visual input more symmetric (Ishikawa et al., 28 Oct 2025). In MedVQA, SLAKE-CP and RadVQA-CP are constructed so that question-answer priors change between train and test, exposing models that rely on question shortcuts (Ye et al., 22 May 2025). VidQA work proposes MIS-guided curation of modality-balanced datasets, while multimodal intent detection removes textually biased samples and highly biased intent categories to create debiased versions of MIntRec-1 and MIntRec2.0 (Park et al., 2024, Mullick et al., 22 Aug 2025).

Optimization and fusion strategies often aim to prevent dominant modalities from monopolizing the objective. Multi-Modal De-Bias (MMDB) replaces standard cross-entropy with an adaptive-margin cosine loss whose class-specific margin is derived from biased-modality statistics, improving OoD performance across colored-digit recognition, VQA-CP, and Kinetics OoD benchmarks (Guo et al., 2022). In video retrieval, MBVR combines manually generated modality-shuffled negatives with a dynamic margin based on visual relevance to force the encoder to reject text-matched but visually wrong videos (Wang et al., 2022). In multimodal federated learning, Balanced Modality Selection combines a modal enhancement loss with modality-level selection so that local training and global aggregation do not amplify a dominant modality (Fan et al., 2023).

Causal and counterfactual methods remove direct shortcut effects at inference or training time. In VQA and entity alignment, the core logic is

$v \rightarrow a$4

so prediction is based on indirect multimodal effects after subtracting the direct biased effect of a modality (Vosoughi et al., 2023, Su et al., 28 Apr 2025). MedCFVQA uses counterfactual inference to estimate and subtract question-driven bias during inference (Ye et al., 22 May 2025). In visible-infrared person re-identification, DMDL combines a causality-inspired adjustment intervention with collaborative bias-free training over data, labels, and features (Li et al., 3 Dec 2025).

Adaptive weighting methods treat balance as context dependent rather than uniform. ActionVLM estimates Language Advantage, the incremental benefit of language over vision-only predictions, and uses it to reweight language features in a residual aggregation scheme where vision remains the base representation (Li et al., 28 Jan 2026). NaPO constructs language-biased and vision-biased rejected responses and uses a noise-aware preference optimization objective to make MLLMs less dependent on a single modality (Zhang et al., 23 Mar 2025). “Freeze and Reveal” applies targeted debiasing to one encoder at a time and shows that the dominant source of gender bias can differ by architecture: CLIP’s vision encoder is more biased, whereas PaliGemma2’s text encoder is more biased (Kavuri et al., 10 Aug 2025). A plausible implication is that debiasing should often be modality-specific and architecture-aware rather than uniformly applied.

6. Evaluation principles, controversies, and open problems

A central unresolved issue is what “balance” should mean. Decision-layer analysis argues that balanced multimodal learning should not force identical weights across modalities; rather, relative balance should be promoted at the task or category level according to each modality’s capability (Ma et al., 16 Oct 2025). This position complicates simple prescriptions such as equal weighting or equalized attention, since some tasks are intrinsically modality-asymmetric.

Another issue is that many multimodal benchmarks do not actually require multimodal reasoning. VidQA analysis concludes that roughly 89.8% to 94.8% of questions across the studied datasets can be answered using a single modality or are effectively modality-agnostic, with complementary questions only 0.6% to 2% (Park et al., 2024). Multimodal intent detection finds a similar textual skew (Mullick et al., 22 Aug 2025), and TVQA’s multimodal-only subset is small relative to the full benchmark (Winterbottom et al., 2020). These results suggest that benchmark success can coexist with weak cross-modal integration, making modality bias partly an evaluation problem.

The current literature also treats explanation as incomplete. The MLLM position paper explicitly calls for systematic metrics, extension beyond vision-language settings to audio, tactile, video, and 3D or robotics-related modalities, and greater use of XAI to reveal internal mechanisms of modality dominance (Zheng et al., 24 May 2025). The abstract on LVLM hallucination suggests that even the standard visual-neglect explanation may be too narrow if models can also fail to retain textual instructions (Zheng et al., 4 Aug 2025). The resulting controversy is not whether modality bias exists, but whether it is primarily caused by data priors, backbone asymmetry, fusion design, optimization dynamics, or evaluation protocol. The evidence assembled across tasks indicates that all of these can matter, and that robust multimodal systems must therefore be designed and assessed at multiple levels simultaneously.