Lexical Bias in Multimodal Models

• Lexical bias in multimodal models is the over-reliance on textual cues at the expense of visual, audio, or structured inputs.
• Metrics like modality imbalance ratio, MDI, and MIS quantify how text dominance skews performance in tasks such as VQA and intent detection.
• Mitigation strategies involve balanced dataset design, improved model architectures, and debiasing techniques to promote genuine multimodal fusion.

Lexical bias in multimodal models—commonly referred to as language-modality bias or text dominance—denotes the systematic propensity of contemporary multimodal architectures to over-rely on linguistic information at the expense of other modalities, notably vision, audio, or structured input. This phenomenon is seen across vision-LLMs, multimodal LLMs (MLLMs), and related systems that fuse diverse source signals for inference or generation. Although superficially beneficial for certain language-heavy benchmarks, lexical bias undermines genuine multimodal reasoning, impedes robustness when non-text modalities are crucial, and can lead to stereotypical or hallucinated outputs when textual priors override grounded perceptual data.

1. Formalizations and Core Metrics

The foundational formalization of modality and lexical bias in the MLLM literature is encapsulated by the modality imbalance ratio: $$\Delta_{\mathrm{modality}} = \frac{C(M_{\mathrm{dominant}})}{C(M_{\mathrm{underutilized}})}$$ where $C(M_i)$ is the estimated contribution of modality $M_i$ to the model’s output. Lexical bias is the special case where the dominant modality is language ("text dominance") (Zheng et al., 24 May 2025).

Further metrics used to operationalize this concept include:

• Performance Deltas: Comparison of model accuracy when full multimodal input ($\mathrm{Acc}_{\mathrm{full}}$) is presented vs. text-only ($\mathrm{Acc}_{\mathrm{text}}$) or image-only ($\mathrm{Acc}_{\mathrm{image}}$). The corresponding deltas,

$$\Delta_{\mathrm{text}} = \mathrm{Acc}_{\mathrm{full}} - \mathrm{Acc}_{\mathrm{text}}, \qquad \Delta_{\mathrm{image}} = \mathrm{Acc}_{\mathrm{full}} - \mathrm{Acc}_{\mathrm{image}}$$

quantify the dependence on each modality (Zheng et al., 24 May 2025).

• Modality Dominance Index (MDI):

$$\mathrm{MDI} = \frac{A_T/|\mathcal{T}|}{A_O/|\mathcal{O}|}$$

with $A_T$, $A_O$ being total attention mass on text vs. non-text tokens and $C(M_i)$0, $C(M_i)$1 their counts. MDI ≫ 1 is strong text dominance (Wu et al., 14 Aug 2025).

• Modality Importance Score (MIS): Using ablation, MIS measures how much the addition of a modality boosts a model's performance on a given task (e.g., in VideoQA, MIS_text quantifies increase in accuracy when text is present) (Park et al., 2024).
• Faithfulness/Fidelity Metrics in Multilingual Multimodal Models: Quantifies the drop in probability that a multimodal model's response remains in the query language when an image is included, highlighting language priors (e.g., English defaulting) (Hinck et al., 2024).

2. Empirical Manifestations Across Tasks

Lexical bias is empirically robust and pervasive. For instance, on fine-grained VQA benchmarks such as MMMU-Pro, Qwen2.5-VL-32B exhibits $C(M_i)$2, with text-only input (removing the image) retaining $C(M_i)$3 (Δ_text = –17.63) compared to just $C(M_i)$4 for image-only (Δ_image = –30.57). This demonstrates that "language alone reproduces over half of the multimodal answer set, whereas vision alone accounts for only about one quarter" (Zheng et al., 24 May 2025).

On canonical VideoQA datasets (TVQA, LifeQA), only 2–3% of test cases truly require both modalities; most ("subtitle-biased") can be answered from the subtitle alone, with performance drops of 51.4 percentage points when subtitles are permuted (versus only 2.5 pp when video is permuted), highlighting extreme over-reliance on textual features (Park et al., 2024).

Large-scale intent detection benchmarks such as MIntRec-1 and 2.0 are over 90% textually biased according to instance-level minimal modality set analysis: "more than nine out of ten test items can be solved by text alone" (Mullick et al., 22 Aug 2025). Even human annotators confirm that words alone suffice for intent classification in 80–90% of cases in these datasets.

3. Causal Mechanisms and Model-Intrinsic Factors

Three primary classes of causal factors underlie lexical bias (Zheng et al., 24 May 2025, Wu et al., 14 Aug 2025):

1. Data Characteristics: Text data is semantically compact and information-dense; vision and other modalities are typically high-dimensional, redundant, and harder to encode. The resulting signal-to-noise ratio imbalance drives models to shortcut via language.
2. Imbalanced Backbone Capabilities: MLLMs are architecturally anchored in strong, heavily-pretrained LLMs. Vision or other modality encoders lag in both capacity and pretraining scale, predisposing models to default to language representations.
3. Training Objectives and Architecture: Prevalent multimodal training objectives (e.g., CLIP contrastive loss, image-text matching) frequently treat text as the semantic anchor, often lacking explicit loss components to enforce balanced information flow.

Lexical bias further intensifies under architectural factors (fusion bottlenecks, shallow bridging, attention concentration), and under task design biases where prompts or answer formats favor text (Wu et al., 14 Aug 2025).

Notably, detailed mechanistic analyses demonstrate that, in spatial reasoning, the bias operates downstream in the LLM on the candidate label integration stage. The visual pathway may correctly encode and attend to the relevant information, but lexical priors in option wording (e.g., "behind," "left") hijack the model’s choice when more options are added, even if the true relation is retained internally (Ma et al., 1 Jun 2026).

4. Dataset Properties, Benchmarking, and Confounding Biases

Most current multimodal benchmarks systematically over-represent language-dominant items. MIS-based audits of VQA and VideoQA datasets show that samples where both modalities are necessary are exceptionally rare (<3%) (Park et al., 2024). For intent detection, statistical and human validation confirms that "audio-only" and "video-only" sufficient examples occur in under 5% of the corpus (Mullick et al., 22 Aug 2025). This skews evaluation, as models can perform well without integrating multiple modalities.

In compound attribute contexts (e.g., emotion recognition), unimodal text-only models deliver highest accuracy and lowest group-based fairness disparities (ΔF1 ≈ 1ppt, EOD ≈ 1.4ppt), while multimodal fusion often amplifies group disparities, suggesting that fusion can import or compound biases from less robust auxiliary modalities (Schmitz et al., 2022).

Biases may also be cross-modal: stereotypical associations (e.g., gender-labeling of objects) may be present independently in text and vision, and multimodal pretraining (as with VL-BERT) can compound these biases rather than suppress them (Srinivasan et al., 2021).

5. Mechanistic and Quantitative Diagnostics

A range of analytic methodologies has emerged to dissect and quantify lexical bias:

• Missing-Modality Evaluation: Systematic masking or ablation (e.g., replacing images with blank/noise, dropping inputs), with performance drops precisely tracking bias (Zheng et al., 24 May 2025, Mullick et al., 22 Aug 2025).
• Attention and Representation Probing: Layerwise analysis of cross-attention distributions (as in MDI/AEI), revealing that late-stage attention in MLLMs strongly favors text, sometimes by factors > 30–150 (Wu et al., 14 Aug 2025).
• Mechanistic Interventions: Activation patching, residual stream probing, and sparse component knockouts in LLM backbones have located and mitigated specific neuron or channel clusters responsible for lexical preference in decision readout (Ma et al., 1 Jun 2026).
• MIS Auditing: Direct estimation of which modality is essential for each sample via model or human-in-the-loop permutation tests (Park et al., 2024).
• Reporting Bias Analysis: Comparing model predictions to true distributional facts in human experience, e.g., canonical object color, exposing how language-only pretraining inherits skew from textual under-reporting of obvious facts, while multimodal models acquire more faithful perceptual priors (Paik et al., 2021).

6. Mitigation and Debiasing Strategies

Research avenues for reducing lexical bias span dataset, model, and training levels:

• Dataset-Level: Careful curation of visually-dependent, modality-balanced benchmarks that minimize text-only shortcuts (e.g., MMStar, counterfactual augmentation, explicit annotation of per-instance modality necessity) (Zheng et al., 24 May 2025, Mullick et al., 22 Aug 2025, Park et al., 2024).
• Model-Level: Visual-attention steering; dynamic or learnable token compression (CLS-guided pruning) to balance the semantic density of text and non-text tokens, which can drop MDI to near 1 (Wu et al., 14 Aug 2025).
• Preference-Based Debiasing: Methods such as Bootstrapped and Noise-Aware Preference Optimization build negative examples (with corrupted text/image), fine-tuning toward correct modality use (Zheng et al., 24 May 2025).
• Post-Hoc Regularization: Calibration and contrastive sampling to correct output distributions under no-visual grounding, removing the LLM prior (e.g., affine transformation for uniformity, visual debias decoding) (Zhang et al., 2024).
• Mechanistic Interventions: Runtime interventions at critical transformer layers, e.g., adding a "language attribute" to restore response fidelity in non-English settings (Hinck et al., 2024).
• Backbone Expansion: Training and deploying LLMs with intrinsic multilingual, multimodal, or balanced encoding capacities (Hinck et al., 2024).

7. Research Outlook and Theoretical Implications

Persistent lexical bias threatens the development of robust, generalizable, and genuinely multimodal AI systems. Over-reliance on text shortcuts can erode resilience to perturbations (e.g., text noise or depletion) and narrow the range of tasks where multimodal models truly excel. Critical research priorities include establishing standardized, theory-grounded bias metrics (e.g., instance-level MIS, MDI), designing architectures that enforce cross-modal equilibrium, and constructing benchmarks that demand—and measure—contextual modality switching and fusion (Zheng et al., 24 May 2025, Wu et al., 14 Aug 2025, Park et al., 2024).

Moreover, purely textual grounding fundamentally limits model perceptual fidelity due to human linguistic reporting bias, with phenomena like color distributions irrecoverable from language alone (Paik et al., 2021). Integrative approaches that fuse direct sensory input during pretraining and devise cross-modal debiasing objectives have the potential to produce models with representations that more closely reflect human perceptual and cognitive processes.

In sum, lexical bias is a pervasive, methodologically tractable challenge in multimodal modeling. Addressing it requires a multi-pronged strategy encompassing dataset design, model architecture, learning objectives, and interpretability—advancing the field toward robust, modality-agnostic artificial general intelligence (Zheng et al., 24 May 2025).