---
title: Lexical Bias in Multimodal Models
url: https://www.emergentmind.com/topics/lexical-bias-in-multimodal-models
type: topic
---

# Lexical Bias in Multimodal Models

Lexical bias in multimodal models—commonly referred to as language-modality bias or text dominance—denotes the systematic propensity of contemporary multimodal architectures to over-rely on linguistic information at the expense of other modalities, notably vision, audio, or structured input. This phenomenon is seen across vision-language models, multimodal large language models (MLLMs), and related systems that fuse diverse source signals for inference or generation. Although superficially beneficial for certain language-heavy benchmarks, lexical bias undermines genuine multimodal reasoning, impedes robustness when non-text modalities are crucial, and can lead to stereotypical or hallucinated outputs when textual priors override grounded perceptual data.

## 1. Formalizations and Core Metrics

The foundational formalization of modality and lexical bias in the MLLM literature is encapsulated by the modality imbalance ratio:
\[
\Delta_{\mathrm{modality}} = \frac{C(M_{\mathrm{dominant}})}{C(M_{\mathrm{underutilized}})}
\]
where \(C(M_i)\) is the estimated contribution of modality \(M_i\) to the model’s output. Lexical bias is the special case where the dominant modality is language ("text dominance") [2505.18657].

Further metrics used to operationalize this concept include:

- **Performance Deltas:** Comparison of model accuracy when full multimodal input (\(\mathrm{Acc}_{\mathrm{full}}\)) is presented vs. text-only (\(\mathrm{Acc}_{\mathrm{text}}\)) or image-only (\(\mathrm{Acc}_{\mathrm{image}}\)). The corresponding deltas,
  \[
  \Delta_{\mathrm{text}} = \mathrm{Acc}_{\mathrm{full}} - \mathrm{Acc}_{\mathrm{text}}, \qquad
  \Delta_{\mathrm{image}} = \mathrm{Acc}_{\mathrm{full}} - \mathrm{Acc}_{\mathrm{image}}
  \]
  quantify the dependence on each modality [2505.18657].

- **Modality Dominance Index (MDI):**
  \[
  \mathrm{MDI} = \frac{A_T/|\mathcal{T}|}{A_O/|\mathcal{O}|}
  \]
  with \(A_T\), \(A_O\) being total attention mass on text vs. non-text tokens and \(|\mathcal{T}|\), \(|\mathcal{O}|\) their counts. MDI ≫ 1 is strong text dominance [2508.10552].

- **Modality Importance Score (MIS):** Using ablation, MIS measures how much the addition of a modality boosts a model's performance on a given task (e.g., in VideoQA, MIS_text quantifies increase in accuracy when text is present) [2408.12763].

- **Faithfulness/Fidelity Metrics in Multilingual Multimodal Models:** Quantifies the drop in probability that a multimodal model's response remains in the query language when an image is included, highlighting language priors (e.g., English defaulting) [2407.02333].

## 2. Empirical Manifestations Across Tasks

Lexical bias is empirically robust and pervasive. For instance, on fine-grained VQA benchmarks such as MMMU-Pro, Qwen2.5-VL-32B exhibits \(\mathrm{Acc}_{\mathrm{full}}=57.80\%\), with text-only input (removing the image) retaining \(40.17\%\) (Δ_text = –17.63) compared to just \(27.23\%\) for image-only (Δ_image = –30.57). This demonstrates that "language alone reproduces over half of the multimodal answer set, whereas vision alone accounts for only about one quarter" [2505.18657].

On canonical VideoQA datasets (TVQA, LifeQA), only 2–3% of test cases truly require both modalities; most ("subtitle-biased") can be answered from the subtitle alone, with performance drops of 51.4 percentage points when subtitles are permuted (versus only 2.5 pp when video is permuted), highlighting extreme over-reliance on textual features [2408.12763].

Large-scale intent detection benchmarks such as MIntRec-1 and 2.0 are over 90% textually biased according to instance-level minimal modality set analysis: "more than nine out of ten test items can be solved by text alone" [2508.16122]. Even human annotators confirm that words alone suffice for intent classification in 80–90% of cases in these datasets.

## 3. Causal Mechanisms and Model-Intrinsic Factors

Three primary classes of causal factors underlie lexical bias [2505.18657, 2508.10552]:

1. **Data Characteristics:** Text data is semantically compact and information-dense; vision and other modalities are typically high-dimensional, redundant, and harder to encode. The resulting signal-to-noise ratio imbalance drives models to shortcut via language.

2. **Imbalanced Backbone Capabilities:** MLLMs are architecturally anchored in strong, heavily-pretrained LLMs. Vision or other modality encoders lag in both capacity and pretraining scale, predisposing models to default to language representations.

3. **Training Objectives and Architecture:** Prevalent multimodal training objectives (e.g., CLIP contrastive loss, image-text matching) frequently treat text as the semantic anchor, often lacking explicit loss components to enforce balanced information flow.

Lexical bias further intensifies under architectural factors (fusion bottlenecks, shallow bridging, attention concentration), and under task design biases where prompts or answer formats favor text [2508.10552].

Notably, detailed mechanistic analyses demonstrate that, in spatial reasoning, the bias operates downstream in the language model on the candidate label integration stage. The visual pathway may correctly encode and attend to the relevant information, but lexical priors in option wording (e.g., "behind," "left") hijack the model’s choice when more options are added, even if the true relation is retained internally [2606.01914].

## 4. Dataset Properties, Benchmarking, and Confounding Biases

Most current multimodal benchmarks systematically over-represent language-dominant items. MIS-based audits of VQA and VideoQA datasets show that samples where both modalities are necessary are exceptionally rare (<3%) [2408.12763]. For intent detection, statistical and human validation confirms that "audio-only" and "video-only" sufficient examples occur in under 5% of the corpus [2508.16122]. This skews evaluation, as models can perform well without integrating multiple modalities.

In compound attribute contexts (e.g., emotion recognition), unimodal text-only models deliver highest accuracy and lowest group-based fairness disparities (ΔF1 ≈ 1ppt, EOD ≈ 1.4ppt), while multimodal fusion often amplifies group disparities, suggesting that fusion can import or compound biases from less robust auxiliary modalities [2205.08383].

Biases may also be cross-modal: stereotypical associations (e.g., gender-labeling of objects) may be present independently in text and vision, and multimodal pretraining (as with VL-BERT) can compound these biases rather than suppress them [2104.08666].

## 5. Mechanistic and Quantitative Diagnostics

A range of analytic methodologies has emerged to dissect and quantify lexical bias:

- **Missing-Modality Evaluation:** Systematic masking or ablation (e.g., replacing images with blank/noise, dropping inputs), with performance drops precisely tracking bias [2505.18657, 2508.16122].
- **Attention and Representation Probing:** Layerwise analysis of cross-attention distributions (as in MDI/AEI), revealing that late-stage attention in MLLMs strongly favors text, sometimes by factors > 30–150 [2508.10552].
- **Mechanistic Interventions:** Activation patching, residual stream probing, and sparse component knockouts in LLM backbones have located and mitigated specific neuron or channel clusters responsible for lexical preference in decision readout [2606.01914].
- **MIS Auditing:** Direct estimation of which modality is essential for each sample via model or human-in-the-loop permutation tests [2408.12763].
- **Reporting Bias Analysis:** Comparing model predictions to true distributional facts in human experience, e.g., canonical object color, exposing how language-only pretraining inherits skew from textual under-reporting of obvious facts, while multimodal models acquire more faithful perceptual priors [2110.08182].

## 6. Mitigation and Debiasing Strategies

Research avenues for reducing lexical bias span dataset, model, and training levels:

- **Dataset-Level:** Careful curation of visually-dependent, modality-balanced benchmarks that minimize text-only shortcuts (e.g., MMStar, counterfactual augmentation, explicit annotation of per-instance modality necessity) [2505.18657, 2508.16122, 2408.12763].
- **Model-Level:** Visual-attention steering; dynamic or learnable token compression (CLS-guided pruning) to balance the semantic density of text and non-text tokens, which can drop MDI to near 1 [2508.10552].
- **Preference-Based Debiasing:** Methods such as Bootstrapped and Noise-Aware Preference Optimization build negative examples (with corrupted text/image), fine-tuning toward correct modality use [2505.18657].
- **Post-Hoc Regularization:** Calibration and contrastive sampling to correct output distributions under no-visual grounding, removing the language model prior (e.g., affine transformation for uniformity, visual debias decoding) [2403.05262].
- **Mechanistic Interventions:** Runtime interventions at critical transformer layers, e.g., adding a "language attribute" to restore response fidelity in non-English settings [2407.02333].
- **Backbone Expansion:** Training and deploying LLMs with intrinsic multilingual, multimodal, or balanced encoding capacities [2407.02333].

## 7. Research Outlook and Theoretical Implications

Persistent lexical bias threatens the development of robust, generalizable, and genuinely multimodal AI systems. Over-reliance on text shortcuts can erode resilience to perturbations (e.g., text noise or depletion) and narrow the range of tasks where multimodal models truly excel. Critical research priorities include establishing standardized, theory-grounded bias metrics (e.g., instance-level MIS, MDI), designing architectures that enforce cross-modal equilibrium, and constructing benchmarks that demand—and measure—contextual modality switching and fusion [2505.18657, 2508.10552, 2408.12763].

Moreover, purely textual grounding fundamentally limits model perceptual fidelity due to human linguistic reporting bias, with phenomena like color distributions irrecoverable from language alone [2110.08182]. Integrative approaches that fuse direct sensory input during pretraining and devise cross-modal debiasing objectives have the potential to produce models with representations that more closely reflect human perceptual and cognitive processes.

In sum, lexical bias is a pervasive, methodologically tractable challenge in multimodal modeling. Addressing it requires a multi-pronged strategy encompassing dataset design, model architecture, learning objectives, and interpretability—advancing the field toward robust, modality-agnostic artificial general intelligence [2505.18657].

Source: https://www.emergentmind.com/topics/lexical-bias-in-multimodal-models