Modality Discrepancy Metrics

• Modality Discrepancy Metric is a family of quantitative methods that define the gap between data modalities using measures like MIR, SAS, and DWMD.
• It encompasses techniques from latent token distribution comparisons to spectral asymmetry and sample-level Shapley contributions, each suited to specific evaluation tasks.
• These metrics have practical applications in pre-training diagnostics, active sample selection, domain adaptation, and robustness auditing, highlighting task-dependent interpretations.

A modality discrepancy metric is a quantitative device for characterizing the gap between modalities in multimodal learning, cross-modality transfer, and heterogeneous recognition. Recent work operationalizes this gap in several non-equivalent ways: as an inter-modal distribution distance in latent token space, as an asymmetric structural mismatch between embedding spectra, as a discrepancy between conditional distributions $P(Y\mid X)$, as a paired performance drop under controlled modality substitution, or as a sample-level divergence in how modalities contribute to prediction (Huang et al., 2024, Gambetti et al., 3 Jun 2026, Ma et al., 2024, Zhang et al., 7 Feb 2026, Wei et al., 2023). The resulting metrics serve different purposes—pre-training diagnosis, active sample selection, domain adaptation, retrieval, or robustness auditing—and their definitions are tightly coupled to the task and evaluation protocol.

1. Conceptual scope and main formulations

Formulation	Representative quantity	Setting
Inter-modal distribution distance	$$\mathrm{MIR}=\log \Bigl(\sum_{k=1}^K \mathrm{FID}_k\Bigr)$$	LVLM pre-training (Huang et al., 2024)
Directional structural mismatch	$\Delta_{SAS}=S_{X\to Y}-S_{Y\to X}$	Medical VLM diagnostics (Gambetti et al., 3 Jun 2026)
Conditional-distribution mismatch	$$D(\mathcal M^s,\mathcal M^t)=\inf_{\pi,\mathcal B} d(P(Y^s_{\pi,\mathcal B}\mid \hat X),P(Y^t\mid \hat X))$$	Cross-modality transfer (Ma et al., 2024)
Paired performance discrepancy	$$\Delta\mathrm{Dice}_{m,o,j}=|\mathrm{Dice}^{\rm structural}-\mathrm{Dice}^{\rm functional}|$$	3D medical segmentation (Zhang et al., 7 Feb 2026)
Sample-level modality contribution	Shapley-style $\phi^i$	Multimodal cooperation (Wei et al., 2023)
Logits-based sample categorization	MI-score and UN-score	VLM-based UDA/ADA (Li et al., 7 Aug 2025)

These formulations are not interchangeable. MIR and DWMD are explicit distribution metrics; SAS is an asymmetric diagnostic that exposes directionality; the UMD protocol measures robustness by holding anatomy fixed and changing only modality; MoNA formalizes discrepancy at the level of semantic knowledge; and sample-level modality valuation and MDI treat discrepancy as an instance-wise phenomenon rather than a single dataset-level scalar (Huang et al., 2024, Wei et al., 2020, Zhang et al., 7 Feb 2026, Ma et al., 2024, Wei et al., 2023, Li et al., 7 Aug 2025).

This suggests that “modality discrepancy metric” is best understood as a family of operational definitions rather than a single canonical formula.

2. Distributional and spectral metrics

The Modality Integration Rate (MIR) is defined for a pretrained LVLM $\mathcal M=(\mathcal E,\mathcal P,\mathcal D)$ with vision encoder $\mathcal E(\cdot)$, vision-language projector $\mathcal P(\cdot)$, and LLM $\mathcal D=(\mathcal D_t,\mathcal F)$. Given image-text pairs, token features from the first $$\mathrm{MIR}=\log \Bigl(\sum_{k=1}^K \mathrm{FID}_k\Bigr)$$0 layers of $$\mathrm{MIR}=\log \Bigl(\sum_{k=1}^K \mathrm{FID}_k\Bigr)$$1 are extracted, concatenated across samples, normalized by a text-centric scale

$$\mathrm{MIR}=\log \Bigl(\sum_{k=1}^K \mathrm{FID}_k\Bigr)$$2

and optionally passed through an outlier-removal function $$\mathrm{MIR}=\log \Bigl(\sum_{k=1}^K \mathrm{FID}_k\Bigr)$$3. At each layer, MIR computes a Fréchet Inception Distance between the vision-token and text-token clouds,

$$\mathrm{MIR}=\log \Bigl(\sum_{k=1}^K \mathrm{FID}_k\Bigr)$$4

and then aggregates across layers as

$$\mathrm{MIR}=\log \Bigl(\sum_{k=1}^K \mathrm{FID}_k\Bigr)$$5

The paper characterizes MIR as Effective, Robust, and Generalize, and reports that as MIR decreases, average downstream scores rise almost monotonically, with Pearson $$\mathrm{MIR}=\log \Bigl(\sum_{k=1}^K \mathrm{FID}_k\Bigr)$$6 in scaling and data-detailedness experiments. MIR varies $$\mathrm{MIR}=\log \Bigl(\sum_{k=1}^K \mathrm{FID}_k\Bigr)$$7 under changes in image domain, text domain, relevant versus irrelevant pairs, conversation template, and seen versus unseen samples (Huang et al., 2024).

A distinct line of work argues that symmetric alignment scores hide directional failure. The Spectral Alignment Score (SAS) begins with centered image and text embedding matrices $$\mathrm{MIR}=\log \Bigl(\sum_{k=1}^K \mathrm{FID}_k\Bigr)$$8, eigendecomposes an anchor covariance such as $$\mathrm{MIR}=\log \Bigl(\sum_{k=1}^K \mathrm{FID}_k\Bigr)$$9, projects both modalities onto the anchor eigenbasis, and computes eigenvalue-weighted per-eigenmode correlations. The directional score is

$\Delta_{SAS}=S_{X\to Y}-S_{Y\to X}$0

with $\Delta_{SAS}=S_{X\to Y}-S_{Y\to X}$1 defined by an eigenvalue quantile. Swapping the anchor yields $\Delta_{SAS}=S_{X\to Y}-S_{Y\to X}$2, and

$\Delta_{SAS}=S_{X\to Y}-S_{Y\to X}$3

quantifies which modality carries richer structure in the top eigenmodes. In a benchmark over 15 VLMs, $\Delta_{SAS}=S_{X\to Y}-S_{Y\to X}$4 is near zero on natural datasets and positive on medical datasets, a directional asymmetry that the paper states is invisible to symmetric metrics such as CKA, SVCCA, CORAL, MMD, and RMG (Gambetti et al., 3 Jun 2026).

DWMD provides a third distributional formulation. For source and target activations $\Delta_{SAS}=S_{X\to Y}-S_{Y\to X}$5, it defines an infinite-series distance on orderwise moment differences with dimension-wise weights:

$\Delta_{SAS}=S_{X\to Y}-S_{Y\to X}$6

The paper states that DWMD is “error-free,” strictly reflects distribution differences between domains, is valid without any feature distribution assumption, and explicitly aligns high-order moments while weighting dimension-wise differences. It also gives a truncation bound for the empirical estimator and contrasts DWMD with MMD, CMD, and Wasserstein distance (Wei et al., 2020).

Taken together, these methods show three distinct ways to encode discrepancy: layerwise Gaussian approximation and FID accumulation, anchor-dependent spectral asymmetry, and infinite-series moment comparison.

3. Sample-level and prediction-level discrepancy

A major theme in multimodal learning is that discrepancy may vary substantially across samples. The sample-level modality valuation framework models modalities as players in a cooperative game. For a sample $\Delta_{SAS}=S_{X\to Y}-S_{Y\to X}$7 with label $\Delta_{SAS}=S_{X\to Y}-S_{Y\to X}$8, the payoff for a modality subset $\Delta_{SAS}=S_{X\to Y}-S_{Y\to X}$9 is

$$D(\mathcal M^s,\mathcal M^t)=\inf_{\pi,\mathcal B} d(P(Y^s_{\pi,\mathcal B}\mid \hat X),P(Y^t\mid \hat X))$$0

and the sample-level contribution of modality $$D(\mathcal M^s,\mathcal M^t)=\inf_{\pi,\mathcal B} d(P(Y^s_{\pi,\mathcal B}\mid \hat X),P(Y^t\mid \hat X))$$1 is the Shapley-style average

$$D(\mathcal M^s,\mathcal M^t)=\inf_{\pi,\mathcal B} d(P(Y^s_{\pi,\mathcal B}\mid \hat X),P(Y^t\mid \hat X))$$2

The paper emphasizes efficiency, symmetry, dummy, and linearity, and argues that dataset-level averages can hide substantial sample-level shifts. It further reports that modality discrepancy indeed could be different at sample-level beyond the global contribution discrepancy at dataset-level (Wei et al., 2023).

A more operational logits-based formulation appears in the MDI metric of the unified modality separation framework. Let $$D(\mathcal M^s,\mathcal M^t)=\inf_{\pi,\mathcal B} d(P(Y^s_{\pi,\mathcal B}\mid \hat X),P(Y^t\mid \hat X))$$3 be vision logits and $$D(\mathcal M^s,\mathcal M^t)=\inf_{\pi,\mathcal B} d(P(Y^s_{\pi,\mathcal B}\mid \hat X),P(Y^t\mid \hat X))$$4 be text logits. A sample is modality-invariant (MI) when

$$D(\mathcal M^s,\mathcal M^t)=\inf_{\pi,\mathcal B} d(P(Y^s_{\pi,\mathcal B}\mid \hat X),P(Y^t\mid \hat X))$$5

with MI-score

$$D(\mathcal M^s,\mathcal M^t)=\inf_{\pi,\mathcal B} d(P(Y^s_{\pi,\mathcal B}\mid \hat X),P(Y^t\mid \hat X))$$6

A sample is modality-specific (MS) when the classifiers cross their top-2 predictions,

$$D(\mathcal M^s,\mathcal M^t)=\inf_{\pi,\mathcal B} d(P(Y^s_{\pi,\mathcal B}\mid \hat X),P(Y^t\mid \hat X))$$7

and all remaining samples are uncertain (UN), with

$$D(\mathcal M^s,\mathcal M^t)=\inf_{\pi,\mathcal B} d(P(Y^s_{\pi,\mathcal B}\mid \hat X),P(Y^t\mid \hat X))$$8

Trusted MI points are selected by an $$D(\mathcal M^s,\mathcal M^t)=\inf_{\pi,\mathcal B} d(P(Y^s_{\pi,\mathcal B}\mid \hat X),P(Y^t\mid \hat X))$$9th-percentile cutoff on MI-score, while the lowest UN-score samples are queried for annotation under a budgeted active adaptation scheme. The reported ablation on OfficeHome gives 82.4% accuracy for full MDI, versus 81.0% for energy-based active sampling and 79.7% for random replacement (Li et al., 7 Aug 2025).

These two approaches share a core premise: discrepancy is not only a global property of two modalities, but also an instance-level property that can be exploited for resampling, pseudo-label selection, and ensemble control.

4. Controlled performance-gap metrics

In general-purpose 3D medical segmentation, modality discrepancy is operationalized directly as a within-subject performance gap. For each organ $$\Delta\mathrm{Dice}_{m,o,j}=|\mathrm{Dice}^{\rm structural}-\mathrm{Dice}^{\rm functional}|$$0, model $$\Delta\mathrm{Dice}_{m,o,j}=|\mathrm{Dice}^{\rm structural}-\mathrm{Dice}^{\rm functional}|$$1, and paired subject $$\Delta\mathrm{Dice}_{m,o,j}=|\mathrm{Dice}^{\rm structural}-\mathrm{Dice}^{\rm functional}|$$2, the discrepancy is

$$\Delta\mathrm{Dice}_{m,o,j}=|\mathrm{Dice}^{\rm structural}-\mathrm{Dice}^{\rm functional}|$$3

where Dice is

$$\Delta\mathrm{Dice}_{m,o,j}=|\mathrm{Dice}^{\rm structural}-\mathrm{Dice}^{\rm functional}|$$4

Per-organ means are averaged over subjects, and an overall discrepancy is obtained by averaging across organs. The protocol uses 490 PET/CT pairs and 464 PET/MRI pairs, with each pair coming from a single subject in a single session, so that anatomy, scale, pose, and related factors are held fixed while modality is varied. The paper summary states that because the only thing that changes between the two runs is the modality, any difference in Dice can be ascribed to “modality discrepancy” (Zhang et al., 7 Feb 2026).

The reporting protocol further uses paired significance tests on structural versus functional Dice scores, with light-gray shading for $$\Delta\mathrm{Dice}_{m,o,j}=|\mathrm{Dice}^{\rm structural}-\mathrm{Dice}^{\rm functional}|$$5 and dark-gray shading for $$\Delta\mathrm{Dice}_{m,o,j}=|\mathrm{Dice}^{\rm structural}-\mathrm{Dice}^{\rm functional}|$$6. No absolute $$\Delta\mathrm{Dice}_{m,o,j}=|\mathrm{Dice}^{\rm structural}-\mathrm{Dice}^{\rm functional}|$$7 threshold is stated. The provided interpretation is that a small $$\Delta\mathrm{Dice}_{m,o,j}=|\mathrm{Dice}^{\rm structural}-\mathrm{Dice}^{\rm functional}|$$8 would indicate near-modality-agnostic performance for organ $$\Delta\mathrm{Dice}_{m,o,j}=|\mathrm{Dice}^{\rm structural}-\mathrm{Dice}^{\rm functional}|$$9 under model $\phi^i$0, while large values such as $\phi^i$1 expose failure to transfer from structural to functional domains (Zhang et al., 7 Feb 2026).

A retrieval-oriented counterpart appears in RGB-Infrared person re-identification. The Similarity Inference Metric (SIM) is presented as exploiting intra-modality sample similarities to circumvent the cross-modality discrepancy targeting optimal cross-modality image matching. According to the abstract, SIM works by successive similarity graph reasoning and mutual nearest-neighbor reasoning that mine cross-modality sample similarities by leveraging intra-modality sample similarities from two different perspectives, and it reports significant accuracy improvement with little extra training on SYSU-MM01 and RegDB (Jia et al., 2020).

This family of metrics does not estimate discrepancy from feature geometry alone. Instead, it measures the effect of modality substitution on the final task under a controlled protocol.

5. Discrepancy as an optimization target

Several methods do not merely diagnose discrepancy; they incorporate it directly into the training objective. In sketch-based 3D shape retrieval, the class-aware cross-modality mean discrepancy is

$\phi^i$2

with mini-batch estimation via per-class feature means. The transformation network is trained adversarially, and the CMD term is added to the generator loss:

$\phi^i$3

The paper states that global adversarial alignment may not respect class-conditional structure, whereas CMD forces per-class feature means to match and thereby promotes local alignment. Reported retrieval performance rises from approximately 0.037 to 0.795 on SHREC 2013 and from approximately 0.028 to 0.782 on SHREC 2014 when adaptation with CMD and adversarial learning is introduced (Chen et al., 2018).

In heterogeneous face recognition, the Adaptive Penalty Metric (APM) focuses on hard cross-modal pairs. With squared Euclidean distance $\phi^i$4, pair label $\phi^i$5, margins $\phi^i$6, hard-pair indicator $\phi^i$7, selector $\phi^i$8, and penalties $\phi^i$9 and $\mathcal M=(\mathcal E,\mathcal P,\mathcal D)$0, APM is

$\mathcal M=(\mathcal E,\mathcal P,\mathcal D)$1

The summary states that APM guarantees the cross-modality intra-class compactness and inter-class separation, and that replacing LMM with APM improves VR@FAR=0.1% on CASIA NIR-VIS 2.0, Oulu-CASIA NIR-VIS, and BUAA-VisNir (Xu et al., 2020).

MoNA introduces a more abstract view: semantic knowledge of a modality is its conditional distribution $\mathcal M=(\mathcal E,\mathcal P,\mathcal D)$2. The modality semantic knowledge discrepancy is defined as

$\mathcal M=(\mathcal E,\mathcal P,\mathcal D)$3

where the source label space is matched to the target through subset selection and permutation. The appendix gives a randomized approximation by source-logit subset selection and a 0–1 mismatch indicator averaged over target examples. MoNA then aims to implicitly minimize this discrepancy by meta-learning a target data transformation that preserves source discriminability after simulated target finetuning (Ma et al., 2024).

These methods illustrate a common design principle: modality discrepancy can be optimized away either through class-conditional alignment, adaptive hard-pair penalization, or meta-learned knowledge alignment.

6. Interpretation, empirical behavior, and limitations

Across the literature, the interpretation of a discrepancy metric depends on what is being measured. In MIR, lower values indicate better integration, and the paper gives practical ranges for a 7B model: approximately 3.6 for poor alignment, approximately 3.0 for near-saturation in a frozen encoder/projector-only scenario, and below 2.8 for strong alignment. It also recommends monitoring $\mathcal M=(\mathcal E,\mathcal P,\mathcal D)$4 during pre-training and treating $\mathcal M=(\mathcal E,\mathcal P,\mathcal D)$5 per epoch $\mathcal M=(\mathcal E,\mathcal P,\mathcal D)$6 as a diminishing-return regime for further compute (Huang et al., 2024).

SAS introduces a limitation of symmetric alignment analysis. The paper explicitly argues that standard alignment metrics produce a single bidirectional score and cannot tell which modality is losing information. By contrast, positive $\mathcal M=(\mathcal E,\mathcal P,\mathcal D)$7 indicates that one anchor modality’s dominant structural directions are better recovered than the reverse. In the reported medical-domain experiments, this asymmetry is consistent and strongly correlated with retrieval performance (Gambetti et al., 3 Jun 2026).

Sample-level valuation introduces a corresponding limitation of dataset-level averages. The paper states that a dataset-level measure would average $\mathcal M=(\mathcal E,\mathcal P,\mathcal D)$8 across samples, but this hides the fact that one modality may dominate on some examples while another is critical on others. That observation underlies the proposed sample-level and modality-level resampling strategies (Wei et al., 2023).

The paired $\mathcal M=(\mathcal E,\mathcal P,\mathcal D)$9 protocol exposes a different limitation: literature-reported benchmarks may not reflect real-world efficacy when evaluation remains confined to structural imaging. By isolating imaging modality as the primary independent variable through intra-subject comparisons, the UMD study reports a stark discrepancy between literature-reported benchmarks and real-world efficacy, particularly when transitioning from structural to functional domains (Zhang et al., 7 Feb 2026).

A plausible implication is that no single modality discrepancy metric can serve all purposes. Metrics based on token distributions are well suited to pre-training diagnostics; asymmetric spectral scores are suited to modality imbalance auditing; logits-based metrics are suited to pseudo-label selection and active querying; paired task-performance gaps are suited to robustness evaluation; and loss-based discrepancy terms are suited to optimization within retrieval or recognition systems. The research record therefore supports a task-dependent interpretation of modality discrepancy, with the metric definition chosen according to whether the objective is diagnosis, selection, alignment, or controlled robustness assessment.