---
title: 'MIS: Assessing Modality Contributions in Multimodal Systems'
url: https://www.emergentmind.com/topics/modality-importance-score-mis
type: topic
---

# MIS: Assessing Modality Contributions in Multimodal Systems

The Modality Importance Score (MIS) is a class of quantitative metrics designed to assess the relative influence, utility, or necessity of each input modality in multimodal systems. MIS quantifies, in a rigorous and reproducible manner, how much each constituent modality (such as text, image, audio, sensor) contributes to downstream performance, information representation, inference, or model interpretability. Empirical MIS variants have proliferated in deep learning, medical imaging, human language understanding, video QA, and explainable AI. Despite diverse methodology, all MIS frameworks aim to allocate responsibility for either predictive performance or representational content to specific modalities within otherwise opaque multimodal pipelines.

## 1. Formal Definitions Across Methodological Families

MIS computation frameworks differ by modeling paradigm, system architecture, and the precise information of interest.

### Information-Theoretic (MI-Based) MIS

"Mutual Information Analysis in Multimodal Learning Systems" defines MIS from pairwise mutual information among modalities [2405.12456]:

For $M$ modalities $\{M_1,\ldots,M_M\}$, with pairwise mutual information $I(M_m; M_n)$ (measured via compression-based entropy estimation), the MIS for modality $m$ is
\[
MIS_m = \frac{\sum_{n\neq m} I(M_m; M_n)}{\sum_{k=1}^M \sum_{l\neq k} I(M_k; M_l)}
\]
Here, a high $MIS_m$ indicates that modality $m$ shares substantial information with others; normalization ensures $\sum_{m=1}^M MIS_m = 1$.

### Ablation and Model Sensitivity-Based MIS

In deep learning medical applications, modality importance is measured by the proportional effect of selective input occlusion [2503.01904]:
\[
m_i = \frac{\mathbf{1}^\top \mathbf{d}_i}{\sum_{j=1}^n \mathbf{1}^\top \mathbf{d}_j}
\]
where $\mathbf{d}_{i,l}^k = |\mathbf{p}_0^k - \mathbf{p}_{i,l}^k|$ is the model output change upon patch-wise masking of modality $i$ in sample $k$, and $m_i$ measures that modality's share of total model sensitivity, enforcing $\sum_{i=1}^n m_i = 1$.

### Fusion-Weight and Attention-Based MIS

In interpretable fusion architectures (e.g., self-attention over modalities [2309.14398]), for $N$ samples, $d$-dimensional fusion coordinates, and $M$ modalities:
\[
MIS_m = \frac{1}{Nd} \sum_{s=1}^N \sum_{i=1}^d \mathbf{1}[m^*_{s,i}=m]
\]
where $m^*_{s,i}$ is the modality with maximal fusion weight at position $i$, in sample $s$.

### Test-Subset Performance Differential MIS

In video QA and related evaluation, MIS is defined as the performance gain when a modality is included in the set of provided modalities, relative to when it is excluded [2408.12763]:
\[
MIS_{i, m_j} = perf(q_i | M_j^+) - perf(q_i | M_j^-)
\]
where $perf(q_i | M')$ is the accuracy on sample $q_i$ for all modality subsets in $M'$, $M_j^+$ includes $m_j$ (with $|S| \geq 2$), and $M_j^-$ excludes $m_j$.

### KL-Divergence/Deviance-Based MIS

Statistical modeling approaches, e.g., high-dimensional GLMs, define MIS as the expected gain in relative entropy when adding a modality [2601.16196]:
\[
MIS_j = \mathbb{E}_X \big[ D_{KL}(p(y|X_{-j}, X_j) \| p(y|X_{-j})) \big]
\]
with sample estimator $\widehat{MIS}_j = \frac{1}{n} [\ell(\hat{\beta}) - \ell(\hat{\beta}_0)]$.

## 2. Algorithmic Estimation Procedures

A variety of estimation and inference procedures are used to compute MIS, tailored to modality type, dataset, and model class.

**Compression-Based MI (InfoMeter):**
- Feature maps are quantized and mapped to suitable latent spaces via invertible transforms.
- Neural entropy estimators (e.g., autoregressive iWave++ models) are trained to minimize empirical entropy $h_X$, $h_Y$, $h_{X,Y}$.
- Post-hoc MI estimation yields $MIS_m$ by pairwise MI summation and normalization [2405.12456].

**Occlusion/Perturbation:**
- For each modality, segment input into patches; mask each in turn, measure change in output, sum changes per modality, and normalize [2503.01904].
- Pseudocode is as follows:
  ```
  for modality i in 1..n:
      for sample k in 1..N:
          for patch l in 1..h_i:
              x_occ = masked version of modality i, patch l
              d_i_k += | f(x^k) - f(x_occ) |
          D[i] += d_i_k
  m_i = D[i] / sum(D)
  ```

**Attention-Weight Averaging:**
- Stack modal embeddings; for each fusion dimension, identify argmax over learned importance weights; count per-modality selections and normalize [2309.14398].

**Performance Differential via Modality Subset Tests:**
- Evaluate the model (or an oracle MLLM) on all combinations of modalities per sample; compute differences in accuracy when a modality is included vs. excluded [2408.12763].

**Likelihood/Deviance-Based Inference:**
- Fit full and reduced GLMs under penalization; compute log-likelihood difference scaled by sample size; MIS is normalized deviance.
- For $p \gg n$, two-step Sure Independence Screening plus penalized likelihood (e.g., SCAD) is used [2601.16196].

## 3. Interpretations and Theoretical Rationale

MIS serves different conceptual goals depending on its formal basis:

- **Redundancy vs. Uniqueness:** MI-based MIS quantifies shared information, with high scores suggesting redundancy. Alternatives subtract pairwise MI from individual entropies to extract modality "uniqueness" [2405.12456].
- **Sensitivity:** Occlusion-based MIS quantifies how model outputs shift when a modality is perturbed, measuring the causal responsibility a modality bears for performance [2503.01904].
- **Explanatory Power:** Attention-based MIS interprets attention or fusion weights as attributions of decision influence, linking weight allocation to classifier behavior [2309.14398].
- **Data-Driven Necessity:** Performance-differential MIS explicitly quantifies whether a task truly depends on a given modality for specific samples, supporting the audit of dataset biases [2408.12763].
- **Statistical Significance:** KL/Deviance-based MIS is justified by information-theoretic gain and is equipped with confidence intervals and $p$-values, supporting hypothesis tests on modality relevance [2601.16196].

## 4. Empirical Outcomes and Use Cases

MIS metrics yield actionable insight into the construction, evaluation, and optimization of multimodal models.

| Application Domain                    | MIS Estimator  | Main Outcomes                                               |
|----------------------------------------|----------------|-------------------------------------------------------------|
| Autonomous vehicle 3D detection        | MI-sum         | Lower MI $\to$ higher detection accuracy                    |
| Medical multimodal diagnosis           | Occlusion      | High-$m_i$ modalities match strong unimodal performance     |
| Motivational interviewing (counseling) | Attention      | Text $\approx$ face $>$ audio/context; clusters in usage    |
| Video QA dataset audit                 | Perf diff      | Most questions are unimodal-biased or modality-agnostic     |
| High-dimensional neuroimaging GLM      | KL/Deviance    | MIS with CI, FDG-PET $>$ Amyloid-PET for EF and DX         |

Empirically, MIS often reveals "unimodal collapse"—the tendency for certain models or datasets to over-rely on a single modality, even when superficial fusion architectures are used [2503.01904, 2408.12763]. Normalized MIS values support cross-architecture, cross-dataset comparison.

## 5. Implementation Considerations and Best Practices

Best practices for reliable MIS estimation include:

- Use invertible transforms and suitable quantization to harmonize continuous-valued modality representations for entropy-based methods [2405.12456].
- For occlusion, set patch granularity to maximize disruption without excessive compute; balancing resolution vs. attribution accuracy [2503.01904].
- When using attention-weighted fusion, ensure embeddings are dimensionally matched and weights are properly regularized [2309.14398].
- For statistical models, penalized estimation (e.g., SIS + SCAD) is critical under $p \gg n$; CIs and $p$-values require proper asymptotic calibration [2601.16196].
- Public codebases are available for several methods, e.g., MC_MMD (PyTorch+MONAI) for occlusion-based MIS [2503.01904]. Independence of ground-truth allows universal applicability and black-box deployment.

## 6. Limitations, Alternatives, and Guidance

Each MIS formalism has caveats:

- MI-based scores may conflate redundancy and informativeness; low shared MI can represent complementarity but also lack of informative signal [2405.12456].
- Occlusion scores are computationally intensive (scaling with $N \sum h_i$ forward passes), and sensitive to patch size [2503.01904].
- Attention-based MIS is interpretable only to the extent that fusion weights reflect causal influence, which is not generally assured [2309.14398].
- Performance-difference MIS is discrete, and may be uninformative where models already succeed via modality-agnostic shortcuts [2408.12763].
- KL/deviance-based MIS requires that the reduced model (excluding a modality) is well-specified and regularized for accurate log-likelihood estimation [2601.16196].
- All global MIS variants summarize at the dataset or corpus level unless extended with per-sample procedures.

No current methodology guarantees causal attribution of modality utility absent strong interventional experiments (e.g., randomized masking). Nevertheless, cross-validation with single-modality models, permutation studies, and human annotation confirm that properly applied MIS tracks the influence and necessity of modalities with practical fidelity.

## 7. Summary Table of Core MIS Formalisms

| Method Family      | Key Equation(s) | Data/Model Prerequisites                                      |
|--------------------|-----------------|---------------------------------------------------------------|
| MI/Entropy-Based   | $MIS_m = \frac{\sum_{n\neq m} I(M_m; M_n)}{\sum_{k=1}^M \sum_{l\neq k} I(M_k; M_l)}$ | Feature map quantization, learned invertible transforms, entropy estimators [2405.12456]         |
| Occlusion-Based    | $m_i = \frac{1^\top D_i}{\sum_j 1^\top D_j}$         | Model w/black-box access, patch masking over inputs [2503.01904]                 |
| Attention-Based    | $MIS_m = \frac{1}{Nd} \sum_{s,i} \mathbf{1}[m^*_{s,i}=m]$ | Fusion model w/attention over modal embeddings [2309.14398]    |
| Performance Differential | $MIS_{i,m_j} = perf(q_i | M_j^+) - perf(q_i | M_j^-)$   | Multimodal model testable on all modal input subsets [2408.12763]|
| KL/Deviance-Based  | $MIS_j = \mathbb{E}_X D_{KL}(p(y|X_{-j}, X_j) \| p(y|X_{-j}))$; $\widehat{MIS}_j = [\ell(\hat{\beta}) - \ell(\hat{\beta}_0)] / n$ | Penalized GLM fits, log-likelihood computation [2601.16196]   |

MIS has become a foundational tool for quantifying the influence of individual modalities in increasingly complex multimodal pipelines, supporting model diagnostics, dataset audit, architecture design, and statistical inference across domains.

Source: https://www.emergentmind.com/topics/modality-importance-score-mis