Cross-Modal Explanation Discrepancy (XMED)

• XMED is a diagnostic metric that quantifies the divergence in explanations between models using genuine compositional reasoning and those relying on modality-specific shortcuts.
• It employs mathematical formulations like Jensen-Shannon divergence and pointwise discrepancy measures to assess the fidelity of multimodal explainers.
• XMED has been applied in vision-language auditing, biomedical time-series analysis, and contestable AI to enhance transparent and reliable model interpretation.

Cross-Modal Explanation Discrepancy (XMED) is a diagnostic metric and conceptual framework for quantifying the mismatch, unreliability, or unfaithfulness in post-hoc explanations generated by multimodal or cross-modal models. XMED rigorously measures whether an explainer method reliably distinguishes between true synergistic interactions across modalities—such as vision and language—or merely attributes decisions to modality-specific shortcuts or superficial correlations. XMED has been utilized in diverse domains, including vision-LLM auditing, biomedical time-series contestable AI, and benchmarking explainable AI techniques for robust safety-critical deployment (Belsare et al., 2 Jun 2026, Nguyen et al., 21 Oct 2025, Ky et al., 21 May 2026).

1. Formal Definition and Motivations

Cross-Modal Explanation Discrepancy is defined as the degree to which an explanation method (E) produces divergent outputs when applied to models with intentionally different cross-modal reasoning strategies. XMED’s canonical instantiation in the GridVQA-X framework involves two paired models: one employing genuine compositional reasoning ($M_{\text{pure}}$) and another structurally constrained to utilize shallow attribute-based shortcuts ($M_{\text{spur}}$). A high XMED score is a necessary signal for an explainer’s faithfulness: only explainers truly sensitive to underlying model logic should produce large explanation discrepancies across these paired models. Low XMED indicates model-blind or hallucinated explanations, which risks practitioner over-trust and failure to flag shortcut-driven or unreliable inferences (Belsare et al., 2 Jun 2026).

The motivation for XMED extends to broader cross-modal settings. In clinical time-series AI, XMED quantifies the attention–attribution gap between fundamentally different explanation modalities (e.g., Grad-CAM versus LRP), surfacing unreliable or conflicted model behavior for human review (Nguyen et al., 21 Oct 2025). In vision–language architectures, XMED underpins recent findings that traditional unimodal perturbation metrics collapse when faced with cross-modal redundancy, failing to capture authentic fusion or synergy (Ky et al., 21 May 2026).

2. Mathematical Formulation

The formalism of XMED is context-dependent but uniformly quantifies explanation divergence across models or modalities.

2.1 Model-Contrast XMED (GridVQA-X)

Given an image–question pair $(V,Q)$ over pixels $I$, and for an explainer $E$ and model $M$, let $A^M_E(i)$ be the raw attribution map and $M_{\mathrm{true}}$ the synthesized binary ground-truth mask (anchor + target object pixels). The attribution maps are normalized over $M_{\mathrm{true}}$: $$p^M_E(i) = \frac{A^M_E(i)}{\sum_{j \in M_{\mathrm{true}}} A^M_E(j)}, \quad i \in M_{\mathrm{true}}$$

The core XMED score on a sample is the Jensen-Shannon divergence between the normalized attributions from the two paired models: $M_{\text{spur}}$0 where $M_{\text{spur}}$1. Mean XMED over a test set of $M_{\text{spur}}$2 samples yields

$M_{\text{spur}}$3

For global methods yielding synergy scores $M_{\text{spur}}$4, a global XMED is defined as the absolute score difference: $M_{\text{spur}}$5

2.2 Modality-Contrast XMED (Attention–Attribution Gap)

For a time-series $M_{\text{spur}}$6, e.g., vGRF signals in clinical gait modeling, two normalized explanation vectors are extracted (e.g., Grad-CAM and LRP): $M_{\text{spur}}$7 Compute the pointwise discrepancy $M_{\text{spur}}$8. With a threshold $M_{\text{spur}}$9 (commonly 0.5), mark high-discrepancy timesteps: $(V,Q)$0 Aggregate to obtain the XMED score: $(V,Q)$1 Reported as a percentage, XMED quantifies the fraction of input where explanation modalities starkly disagree, signaling internal model or attribution inconsistency (Nguyen et al., 21 Oct 2025).

3. Evaluation Protocols and Benchmarks

XMED is operationalized via rigorously controlled experimental setups. In GridVQA-X (Belsare et al., 2 Jun 2026), the paired models $(V,Q)$2 (robust spatial-relational reasoning via compositional grounding with bounding-box supervision and confuser region distractors) and $(V,Q)$3 (forced Bag-of-Words heuristics) are constructed. Visual–linguistic questions are framed such that only precise spatial or attribute-based reasoning suffices for correct prediction; confounders nullify alternative cues or easy priors.

Explainability methods are evaluated by applying the same post-hoc explainer E to both models, restricting attribution to the provable ground-truth regions, and computing the mean XMED (local or global). Reliable explainers produce large XMED values, faithfully distinguishing multi-hop, synergistic logic from shortcut use. Low XMED typifies hallucinated or model-blind attributions.

In clinical time-series explainability (Nguyen et al., 21 Oct 2025), XMED safeguards are directly integrated into clinician-centered workflows: high XMED triggers contestable review, highlighting high-disagreement regions for oversight. Thresholds and aggregation hyperparameters are selected empirically on validation data.

4. Representative Results and Methodological Insights

Empirical findings highlight the utility of XMED for both benchmarking and operational reliability:

• Local explainers in GridVQA-X: MultiViz-gradient (RMA = 0.44 on both $(V,Q)$4 and $(V,Q)$5) yields XMED ≈ 0 (model-blindness). DIME yields XMED ≈ 0.005 (accidental faithfulness). MultiSHAP paradoxically aligns explanations more with $(V,Q)$6 (XMED$(V,Q)$7 ≈ 0.07).
• Global synergy explainers: EMAP’s $(V,Q)$8 drops with complexity (0.821→0.673), but $(V,Q)$9 stays artificially high ($I$00.60), yielding $I$1. InterSHAP exhibits non-monotonic XMED (≈0.07–0.09), insufficiently distinguishing true synergy.
• Clinical XMED: In Motion2Meaning, mean XMED % is 1.45 for correct predictions and 7.56 for incorrect, a consistent fivefold discrepancy (Nguyen et al., 21 Oct 2025). This pattern is robust across random seeds and validation sweeps.

A summary table of GridVQA-X results:

Explainer	RMA$I$2	RMA$I$3	XMED (local/global)
MultiViz-gradient	≈ 0.44	≈ 0.44	≈ 0.00
DIME	≈ 0.29	≈ 0.28	≈ 0.005
MultiSHAP	≈ 0.62	≈ 0.69	≈ 0.07
EMAP (global)	varies	≈ 0.60	≈ 0.15
InterSHAP (global)	0.921→0.629	0.85→0.54	0.07–0.09

5. Broader Implications, Theoretical Factors, and Limitations

High XMED empirically aligns with model-aware, faithful explainability—classifying explainers that surface true cross-modal interaction logic from those that mask shortcut-driven or superficial behaviors. In interpretability pipelines, this allows practitioners to audit explanations critically, reduce over-trust, and focus human attention on potentially dangerous or unreliable cases.

Low XMED is diagnostic of hallucinated explanations, typically failing to reflect divergences in true underlying reasoning. This can result in practitioner overconfidence, regulatory risk, and failure to detect model errors. In high-stakes domains such as healthcare, contestable AI, and autonomous systems, maximizing XMED across known divergences constitutes an essential methodological goal.

A related phenomenon in VLMs is cross-modal redundancy, where unimodal perturbation metrics—such as visual-only deletion or insertion curves—fail to capture joint modality reasoning and may yield adversarial or paradoxical explanations. Empirical studies show that the correlation (Kendall’s $I$4) between visual and textual explanation rankings often collapses to near zero ($I$5 globally) (Ky et al., 21 May 2026), further motivating the necessity for XMED-style diagnostics.

6. Relationships to Synergistic Metrics and Future Directions

XMED occupies a crucial position in the landscape of explainability metrics: it is complementary to, but not strictly subsumed by, metrics like Synergistic Faithfulness ($I$6) (Ky et al., 21 May 2026). While XMED quantifies explanation change across model logic or explanation modality, $I$7 isolates the joint contribution of multimodal features using cooperative game theory (Harsanyi dividends, Shapley Interaction Index). The two approaches, while distinct, are both essential in benchmarking and deploying faithful multi-modal explainers.

A plausible implication is that future explainability research will integrate XMED maximization within method development pipelines, enforcing explainers to be both model-sensitive and robust to cross-modal shortcut phenomena. Controlled benchmarks leveraging ground-truth interventions—such as those in GridVQA-X—enable principled evaluation and robustification of explainability under adversarial, shortcut, or spurious reasoning regimes.

7. Application Domains and Operational Impact

XMED has demonstrated impact in three major lines of research:

• Multimodal Vision–LLM Auditing: As a core metric in the GridVQA-X and cross-modal synergy literatures, XMED underlies the evaluation of explainers for spatial-relational and compositional reasoning models, exposing vulnerabilities to shortcut behaviors (Belsare et al., 2 Jun 2026, Ky et al., 21 May 2026).
• Biomedical Time-Series and Contestable AI: XMED operates as an automatic model reliability safeguard, highlighting latent disagreement or confusion for human-in-the-loop review in clinical decision support (notably, Parkinson’s disease gait analysis) (Nguyen et al., 21 Oct 2025).
• Synergistic Explainability Benchmarking: XMED-driven protocol design refines the deployment criteria for researchers comparing model- and modality-sensitive explainers, guiding selection toward methods that surface true cross-modal fusion, and away from those over-indexed on unimodal salience.

The operationalization of XMED is foundational for the transparent, reliable, and fault-tolerant deployment of sophisticated multimodal models in safety-critical or scientifically demanding contexts.