Cross-Modal Conflicts Overview

Updated 14 July 2026

Cross-modal conflicts are cases where differing modalities or internal knowledge yield incompatible interpretations of the same input.
They encompass regimes like direct contradiction, knowledge conflict, fusion conflict, and sample conflicts, each affecting system performance uniquely.
Approaches to address these conflicts include layerwise detection, attention-map analysis, and conflict-aware architectures that improve multimodal reasoning and robustness.

Cross-modal conflicts are cases in which multiple modalities, or a modality and a model’s internal parametric knowledge, support incompatible interpretations of the same scene, query, or action. In contemporary multimodal systems they arise when images contradict captions, audio contradicts visual location, video or audio evidence contradicts a textual premise, LiDAR and camera BEV features disagree, or dense MLLM-generated text conflicts with sparse captions. The literature increasingly treats these conflicts as a diagnostic lens on modality bias, joint reasoning, hallucination, and sensor fusion, because models may internally encode the mismatch yet still fail to act on it, may over-trust one modality, or may degrade under naive fusion (Nguyen et al., 2 Sep 2025, Zhu et al., 2024, Quang et al., 13 May 2026, Fu et al., 2024).

The term covers several distinct regimes rather than a single failure mode. Some works focus on direct contradiction in the input, such as an image of a dog paired with the caption “A photo of a cat,” or a misleading textual premise that contradicts what the model sees or hears (Hua et al., 2 Jul 2025, Quang et al., 13 May 2026). Others formalize conflict as disagreement between visual and textual or parametric knowledge, defining a cross-modality parametric knowledge conflict when the visual answer $y^v$ and the textual answer $y^t$ differ for the same query (Zhu et al., 2024). In sensor fusion and affective computing, conflict also includes mismatches induced by projection pipelines, missing cues, ambiguity, sarcasm, or intrinsically contradictory signals, yielding typologies such as extrinsic versus inherent conflicts and benign versus severe conflicts (Fu et al., 2024, Yu et al., 6 May 2026).

Conflict regime	Description in the literature	Representative formulation or example
Direct contradiction	Modalities provide incompatible evidence about the same referent	Conflicting image-caption pairs; contradiction-targeted QA (Hua et al., 2 Jul 2025, Popordanoska et al., 24 Nov 2025)
Knowledge conflict	External evidence disagrees with textual memory or internal knowledge	$y^v \neq y^t$ ; factual versus counterfactual completions (Zhu et al., 2024, Ortu et al., 18 Jul 2025)
Fusion conflict	Heterogeneous sensors or cues interfere during fusion	Extrinsic/inherent BEV conflicts; benign/severe MER conflicts (Fu et al., 2024, Yu et al., 6 May 2026)
Sample or representation conflict	Training samples or auxiliary representations encode misleading disagreement	False negatives in motion-text retrieval; dense-versus-sparse caption conflict (Yan et al., 2023, Mao et al., 3 Nov 2025)

This broad usage has an important methodological consequence. A system can fail because it never detects the contradiction, because it detects it but resolves it poorly, because it resolves it with a stable bias toward one modality, or because fusion itself destroys information that remained available in unimodal pathways. The recent literature increasingly decomposes these possibilities rather than treating multimodal disagreement as a unitary robustness failure.

2. Detection, resolution, and modality following

Mechanistic work on LLaVA-OV-7B explicitly separates conflict detection from conflict resolution. Using layerwise activations $h^l$ , LASSO logistic-regression linear probes predict a binary conflict label $y_{\text{conflict}} \in \{0,1\}$ and show that a linearly decodable conflict signal emerges in intermediate layers, peaking at intermediate layers ( $\sim$ layer 10 and after layer 15). Attention-map analysis then distinguishes detection-related divergence from resolution-related divergence: non-overlapping head clusters specialize for either detection or resolution, and the attention divergence associated with detection surfaces prior to resolution. The same work defines

$\text{resolution\_confidence} = p_{\text{image-aligned}} - p_{\text{text-aligned}},$

and reports a nonlinear and heteroscedastic relation between this output-space quantity and the internal detection signal, supporting the claim that detection and resolution are functionally distinct mechanisms (Nguyen et al., 2 Sep 2025).

A complementary behavioral formulation appears in work on modality following in MLLMs. There, the choice of which modality to follow is decomposed into relative reasoning uncertainty and inherent modality preference. Unimodal uncertainty is measured by output entropy,

$H(x) = - \sum_{y \in \mathcal{V}} p(y \mid x)\log p(y \mid x),$

and relative uncertainty is defined as

$\Delta H_{\mathrm{rel}}(x) = \frac{2(H^{(t)}(x) - H^{(v)}(x))}{H^{(t)}(x) + H^{(v)}(x)}.$

The reported regularity is monotonic: the probability of following a modality decreases as its relative uncertainty increases. The balance point, where text- and vision-following become equally likely, serves as an indicator of inherent preference. Near this balance point, layer-wise logit tracking reveals oscillation between modalities, whereas clear cases commit early and remain stable (Zhang et al., 4 Nov 2025).

Omnimodal evaluation extends the same decomposition from vision-LLMs to systems that jointly process video, audio, and text. IMAVB documents a “Representation-Action Gap”: hidden states reliably encode premise-perception mismatches even when the same models almost never reject the false claim in their outputs. The benchmark also distinguishes under-rejection, where models answer misleading questions as if the false premise were true, from over-rejection, where models reject more often but sacrifice ordinary comprehension accuracy. Audio grounding underperforms vision grounding, indicating that the detection-action dissociation is modality-asymmetric rather than purely generic (Quang et al., 13 May 2026).

3. Evaluation paradigms and empirical regularities

The recent benchmark literature replaces coarse dataset-level following ratios with controlled contradiction settings. CLASH pairs COCO images with contradictory captions containing controlled object-level or attribute-level contradictions, supplies targeted questions in both multiple-choice and open-ended formats, and includes approximately 30,000 raw samples, approximately 15,000 filtered training samples, and a 1,289-example human-verified diagnostic set. In multiple choice, GPT-5 reaches 86.78% conflict detection and Gemini 2.5 Pro 88.48%, whereas LLaVa-1.5-7b records 0.13% and InternVL 1.5 records 16.71%. The open-ended setting preserves the same stratification, with strong closed models detecting contradictions frequently and many open models failing to verbalize conflict at all (Popordanoska et al., 24 Nov 2025).

Synthetic cross-modal and cross-lingual benchmarks show that this difficulty is not explained by weak unimodal reasoning alone. On CMQA and CLQA, foundation models recognize conflicts in unimodal contexts 90% of the time, but the rate falls as low as 3% when evidence is split across modalities or languages. The same study attributes this collapse to cross-modal attention imbalance, and reports that manual attention reweighting boosts conflict detection by up to 43% absolute in multilingual settings and by a factor of 2 in multimodal settings. Instance-level modality mixing reduces cross-modal attention imbalance by up to 4× in cross-lingual and by 34% in cross-modal cases, with downstream gains on HardBLINK, SAT, and MMMU (Wu et al., 2 Oct 2025).

Related work on cross-modality parametric knowledge conflict reformulates ViQuAE and InfoSeek into multiple-choice QA and quantifies disagreement through flip rate and conflict rate. The flip rate is

$FR = \frac{1}{N}\sum_{i=1}^N \mathbb{1}(y_i^v \neq y_i^t),$

and the conflict rate is defined as $y^t$ 0, where $y^t$ 1 is the textual–visual accuracy gap. This formalization is designed to separate genuine knowledge conflict from differences in baseline modality performance (Zhu et al., 2024).

Perceptual protocols reveal analogous asymmetries. In sound localization across six audiovisual conditions—Congruent, Conflicting Visual Cue, Absent Visual Cue, Audio Only, Vision Only, and Multi-Instance Localization—humans consistently outperform multimodal AI. Humans remain well above chance under misleading or absent visuals by relying on auditory information, whereas most AI models default to visual input and degrade to near chance. Fine-tuning on the stereo AudioCOCO dataset improves conflict handling substantially, and the refined model also exhibits a human-like horizontal localization bias (Jia et al., 16 May 2025).

SegSub generalizes conflict evaluation through targeted image perturbations and three conflict types: parametric conflicts, counterfactual conflicts, and source conflicts. Baseline VLMs are relatively robust to parametric conflicts, with 20% adherence rates, but remain weak at counterfactual conditions with less than 30% accuracy and at source conflicts with less than 1% accuracy. The reported correlation between contextual richness and hallucination rate, $y^t$ 2 with $y^t$ 3, indicates that richer contextual support can make counterfactual hallucination more likely rather than less (Carragher et al., 19 Feb 2025).

4. Mechanistic circuits and intervention methods

Mechanistic analysis consistently finds that conflicting information remains represented deep inside the model. In experiments with inconsistent image-caption pairs drawn from CIFAR-10, CIFAR-100, ImageNet100, Pascal VOC, and CUB, linear probes show that both image and caption information remain decodable throughout most of the network, while consistency probes achieve near-perfect accuracy at later layers. K-means clustering and V-Measure further show that when a model behaviorally favors one modality, last-layer representation clusters align with that modality’s labels. The V-Measure gap predicts target-modality behavioral accuracy with $y^t$ 4 and $y^t$ 5. Head-scaling experiments identify modality-agnostic “router heads,” which route toward the modality requested in the instruction, and modality-specific promotion heads, which consistently favor image or caption regardless of instruction (Hua et al., 2 Jul 2025).

Conflict resolution between visual evidence and internal knowledge appears similarly localized. Using the Whoops-Aha! dataset of multimodal counterfactual queries, work on LLaVA-NeXT and Gemma3 identifies a small set of upper-layer attention heads that control factual-versus-counterfactual preference. Counterfactual heads assign about 60–61% of their attention to image tokens, compared with 28–29% for factual heads and 22% for the average head. Targeted intervention on these heads raises factual accuracy from 22% to 74% in LLaVA-NeXT and up to 83% in Gemma3, while attention from the same heads pinpoints localized image regions driving visual overrides and outperforms gradient-based attribution in precision (Ortu et al., 18 Jul 2025).

Several mitigation methods now target decoding rather than representation alone. Dynamic contrastive decoding compares answer confidence across visual and textual branches and subtracts the less confident modality’s logits; on LLaVA-34B it improves average accuracy by 2.24% on ViQuAE and InfoSeek (Zhu et al., 2024). Probe-guided logit adjustment on IMAVB learns a mismatch signal from hidden states and re-injects it into the logits of rejection options, producing a mean balanced-accuracy gain of +15pp across eight open-source models (Quang et al., 13 May 2026). Training-free Active Visual-Context Refinement attacks “textual inertia” by invoking active visual re-grounding and context folding; explicit reflection rises to 29% on KeyE-Preview-8B and 31% on Qwen2.5-VL-7B under entity perturbations, whereas baseline self-correction is reported below 10% (Zhu et al., 7 Jan 2026).

5. Conflict-aware architectures across application domains

Conflict-aware modeling has become an architectural principle in multimodal sentiment and emotion recognition. MCAN for multimodal sentiment analysis argues that interaction contains both alignment and conflict aspects, and therefore progressively segregates alignment and conflict constituents from unimodal and bimodal representations using Micro-MSIN, Macro-MSIN, and SVD-based decomposition. A dedicated conflict modeling branch imposes discrepancy constraints at both the representation and predicted-output levels without generated conflict labels. On CMU-MOSI and CMU-MOSEI, MCAN reports state-of-the-art performance, and ablations identify both the conflict modeling branch and the discrepancy losses as crucial (Gao et al., 13 Feb 2025).

In multimodal emotion recognition, DCR sharpens this distinction by introducing a conflict taxonomy based on resolvability. Benign conflicts stem from missing, weak, or ambiguous cues and are handled by the Affective Fusion Distiller, while severe conflicts arise from contradictory or misleading signals and are handled by the Affective Discernment Agent, formulated as a contextual bandit that chooses among fusion and unimodal predictions. On the CH-SIMS severe-conflict subset, DCR reaches 50.3% accuracy versus approximately 41–44% for the strongest baselines, and on benign conflicts it reaches 72.4% versus 59–62%. Modality-selection analysis shows that the agent still prefers the fusion path most often overall, but reduces fusion to 49% under severe conflicts (Yu et al., 6 May 2026).

In sensor fusion, ECFusion decomposes LiDAR-camera conflicts in BEV space into extrinsic conflicts, caused by BEV feature construction, and inherent conflicts, caused by heterogeneous sensor signals. Semantic-guided Flow-based Alignment corrects spatial mismatch before fusion, while Dissolved Query Recovering restores objectness clues lost in the fused BEV feature. On nuScenes, the resulting detector reaches 73.4% NDS and 70.7% mAP, exceeding the BEVFusion baseline by +2.0% NDS and +2.2% mAP (Fu et al., 2024). For joint audio-video generation, Cross-Modal Context Learning identifies conflicts arising from gating-induced manifold variations, background-region bias in cross-modal attention, and inconsistencies in multimodal classifier-free guidance, and introduces TARP, Learnable Context Tokens, Dynamic Context Routing, and Unconditional Context Guidance to improve alignment, train-inference consistency, and generation quality (Ma et al., 19 Mar 2026).

Other application areas expose more specialized forms of conflict. DropTriple Loss for motion-text retrieval treats overlapping atomic actions as semantic conflicts that create false negatives; by discarding those false negatives before hard-negative mining, it reaches R@10 of 62.9% for motion retrieval and 71.5% for text retrieval on HumanML3D (Yan et al., 2023). SEPS addresses dense-versus-sparse semantic conflict in fine-grained alignment by integrating dense MLLM-generated text with sparse captions for patch selection and relevance-aware patch-word correspondence, reporting improvements of 23%–86% in rSum across diverse model architectures (Mao et al., 3 Nov 2025). AsymLoRA formulates instruction-tuning conflicts as the joint problem of preserving task-specific low-rank pathways for conflicting objectives while sharing a common projection for transferable knowledge, using $y^t$ 6 with shared $y^t$ 7 and task-specific $y^t$ 8 (Wei et al., 27 Feb 2025). In multimodal deepfake detection, CAD combines cross-modal alignment with cross-modal distillation so that semantic misalignment and modality-specific forensic traces contribute jointly, reaching 99.6% AUC on FakeAVCeleb and 99.96% AUC on IDForge (Du et al., 21 May 2025).

6. Human studies, robustness, and open problems

Human-centered experiments provide an important counterpoint to model-centric studies. In an immersive audio-visual setup with four animated avatars, 33 subjects showed that visual bias depends both on spatial distance and on scene semantics: moving lips synchronized with vocalization induced the strongest bias, and beyond 20–25° separation the bias decreased. A multichannel deep neural network trained on human behavioral responses enabled an iCub robot to reproduce similar real-time conflict-resolution patterns, although the robot lacked the human central-versus-periphery asymmetry because its cameras did not model foveation (Parisi et al., 2018).

Several broader regularities now appear stable across benchmarks. Aggregate following ratios such as TFR and VFR are confounded by unimodal reasoning difficulty and dataset composition, which is why balance-point analyses and uncertainty-conditioned curves are more informative than macro-level ratios alone (Zhang et al., 4 Nov 2025). Blindly scaling multimodal or multilingual datasets does not remove attention imbalance when training instances do not explicitly require joint reasoning (Wu et al., 2 Oct 2025). Prompt engineering also appears insufficient as a general solution: IMAVB reports that the Representation-Action Gap is prompt-resistant across seven variants, and LogicGraph-based perturbation studies show that once textual hallucination enters a reasoning chain, models usually propagate it rather than explicitly reject it (Quang et al., 13 May 2026, Zhu et al., 7 Jan 2026).

Benchmark evidence also indicates that contradiction detection is not uniformly distributed across semantic categories. CLASH reports stronger conflict detection on animals and vehicles than on household and environmental attributes, and exploratory spatial contradictions are substantially harder even for strong closed models (Popordanoska et al., 24 Nov 2025). This suggests that “cross-modal conflict” is not only a modality-integration problem but also a fine-grained compositionality problem involving attribute types, spatial relations, and abstention behavior.

Taken together, the literature supports a shift from indiscriminate multimodal fusion toward conflict-aware systems that can detect contradiction, expose internal preference dynamics, abstain when evidence is irreconcilable, and intervene at the level of heads, logits, queries, or training instances. This suggests that the central problem is no longer whether multimodal models can combine inputs, but whether they can preserve the distinction between alignment, contradiction, uncertainty, and source reliability while doing so (Nguyen et al., 2 Sep 2025).