---
title: 'Dr. SHAP-AV: Shapley Analysis for AVSR'
url: https://www.emergentmind.com/topics/dr-shap-av
type: topic
---

# Dr. SHAP-AV: Shapley Analysis for AVSR

Dr. SHAP-AV is a framework for analyzing relative modality contributions in audio-visual speech recognition (AVSR) with Shapley values. It treats audio and visual inputs as cooperative-game players and attributes the model’s token-level decoding behavior to them through Shapley-based feature attribution. Applied to six AVSR models across LRS2 and LRS3 under clean and noisy conditions, it introduces three complementary analyses—Global SHAP, Generative SHAP, and Temporal Alignment SHAP—and reports that AVSR systems shift toward visual reliance as acoustic quality degrades, yet often retain substantial audio contribution even at severe noise levels, thereby exposing a persistent audio bias [2603.12046].

## 1. Problem setting and motivation

AVSR combines acoustic and visual speech cues, typically to improve robustness under noise. Dr. SHAP-AV is motivated by a specific interpretability gap: AVSR work has usually emphasized word error rate rather than a principled account of how models balance modalities. In this setting, modality use is not a secondary concern. AVSR is a multimodal reliability-weighting problem, and a system that remains overly audio-driven under severe corruption may fail to exploit the visual stream when it is comparatively more reliable [2603.12046].

The framework is introduced against a preliminary observation from drop-modality experiments in clean conditions: removing video often leaves word error rate close to full AVSR, whereas removing audio causes catastrophic degradation for most models. This indicates a strong audio bias, especially in models trained only for AVSR rather than explicitly for visual-only recognition. Dr. SHAP-AV therefore asks how much audio and video contribute, how that balance evolves during autoregressive generation, and whether input-output temporal alignment is preserved across acoustic conditions [2603.12046].

Its target is not sentence-level correctness and not word error rate directly. Instead, it attributes the model’s token-by-token decoding behavior. This choice matters because modality balance can vary across output positions even within a single utterance, and because a coarse ablation analysis cannot recover the contribution structure of the generated sequence.

## 2. Shapley formulation for AVSR decoding

Let a pretrained AVSR model \(f\) generate a token sequence \(\mathbf y=(y_1,\dots,y_T)\) from audio features \(\mathbf x^A=(x_1^A,\dots,x_{N_A}^A)\) and visual features \(\mathbf x^V=(x_1^V,\dots,x_{N_V}^V)\). Dr. SHAP-AV defines the full feature set as
\[
\mathcal F=\mathcal A\cup\mathcal V,
\]
with \(\mathcal A\) indexing audio features and \(\mathcal V\) visual features. For each output token \(y_t\), the characteristic function on a coalition \(\mathcal C\subseteq\mathcal F\) is
\[
f_{\mathbf x}^t(\mathcal C)=\mathbb E\left[\log p(y_t \mid \mathbf X_{\mathcal C}, y_{<t}) \mid \mathbf X_{\mathcal C}=\mathbf x_{\mathcal C}\right].
\]
Thus the payoff is the expected log-probability of the generated token conditioned on preceding tokens and a subset of observed input features [2603.12046].

The token-level Shapley value for feature \(i\) is
\[
\phi_{i,t} = \sum_{\mathcal{C}\subseteq \mathcal{F}\setminus\{i\}} \frac{|\mathcal{C}|!\cdot(N-|\mathcal{C}|-1)!}{N!} \left[ f_{\mathbf{x}}^t(\mathcal{C}\cup\{i\}) - f_{\mathbf{x}}^t(\mathcal{C}) \right].
\]
Collecting \(\phi_{i,t}\) across features and output positions yields a Shapley matrix
\[
\mathbf{\Phi}\in\mathbb{R}^{N\times T}.
\]
Rows represent input features; columns represent generated tokens [2603.12046].

Exact computation is intractable, so the paper uses Permutation SHAP:
\[
\phi_i(f_{\mathbf x}) \approx \frac{1}{M}\sum_{m=1}^{M} \left[ f_{\mathbf{x}}(\mathcal{P}_i^{\pi_m}\cup\{i\}) - f_{\mathbf{x}}(\mathcal{P}_i^{\pi_m}) \right],
\]
where \(M\) is the number of sampled coalitions or permutations. The reported setting is
\[
M=2000,
\]
chosen because lower \(M\) produced higher variance, especially at low SNR. Sampling SHAP was also tested and gave similar results, but Permutation SHAP was used for subsequent analyses because it was faster [2603.12046].

Although the characteristic function is written as a conditional expectation, the implementation approximates it by masking absent features and performing a forward pass rather than by true marginalization. The masking scheme is architecture-specific. For LLM-based AVSR, masking is applied after modality-specific projection layers; for AV-HuBERT and Auto-AVSR, before fusion; for Whisper-Flamingo, before decoder cross-attention consumes the modality-specific features [2603.12046].

## 3. The three analyses

Dr. SHAP-AV organizes token-level feature attributions into three analyses.

| Analysis | Unit of aggregation | Purpose |
|---|---|---|
| Global SHAP | All features, all output tokens | Overall audio-vs-video balance |
| Generative SHAP | Output-token windows | Contribution dynamics during decoding |
| Temporal Alignment SHAP | Input bins \(\times\) output bins | Input-output correspondence |

Global SHAP defines modality-level proportions by aggregating absolute Shapley mass:
\[
\text{A-SHAP} = \frac{\sum_{j\in\mathcal A}\sum_{t=1}^{T} |\phi_{j,t}|}{\sum_{j\in\mathcal F}\sum_{t=1}^{T} |\phi_{j,t}|}, \qquad
\text{V-SHAP} = \frac{\sum_{j\in\mathcal V}\sum_{t=1}^{T} |\phi_{j,t}|}{\sum_{j\in\mathcal F}\sum_{t=1}^{T} |\phi_{j,t}|}.
\]
By construction,
\[
\text{V-SHAP}=1-\text{A-SHAP}.
\]
A value of \(0.5\) indicates balance; values above \(0.5\) indicate audio dominance [2603.12046].

Generative SHAP partitions the output sequence into windows \(\{\mathcal T_1,\dots,\mathcal T_W\}\) and computes per-window modality ratios:
\[
\text{A-SHAP}^{(w)} = \frac{\sum_{j\in\mathcal A}\sum_{t\in\mathcal T_w} |\phi_{j,t}|}{\sum_{j\in\mathcal F}\sum_{t\in\mathcal T_w} |\phi_{j,t}|}, \qquad
\text{V-SHAP}^{(w)} = 1-\text{A-SHAP}^{(w)}.
\]
This produces decoding-time trajectories rather than a single utterance-level summary [2603.12046].

Temporal Alignment SHAP normalizes both input positions and output positions by percentage of utterance length. For modality \(m\in\{A,V\}\), input features are partitioned into \(K\) bins \(\{\mathcal F_1^{(m)},\dots,\mathcal F_K^{(m)}\}\), output tokens into \(W\) bins \(\{\mathcal T_1,\dots,\mathcal T_W\}\), and the alignment matrix is
\[
H_{k,w}^{(m)} = \frac{\sum_{j\in\mathcal F_k^{(m)}}\sum_{t\in\mathcal T_w} |\phi_{j,t}|}{\sum_{w'=1}^{W}\sum_{j\in\mathcal F_k^{(m)}}\sum_{t\in\mathcal T_{w'}} |\phi_{j,t}|}.
\]
Each row sums to one. Diagonal structure in \(\mathbf H^{(m)}\) is interpreted as temporal alignment between input and output progression [2603.12046].

## 4. Models, benchmarks, and implementation choices

The framework is evaluated on six AVSR systems across two model families and two benchmarks.

| Family | Models | Notes |
|---|---|---|
| LLM-based AVSR | Llama-AVSR, Llama-SMoP, Omni-AVSR | Whisper medium audio, AV-HuBERT Large video, Llama 3.2-1B backbone |
| Cross-attention / encoder-decoder AVSR | AV-HuBERT, Auto-AVSR, Whisper-Flamingo | Architecturally distinct fusion and decoding schemes |

The benchmarks are LRS2 and LRS3. LRS2 contains 225 hours of BBC program footage; LRS3 contains 433 hours of TED-talk video. Most detailed analyses are reported on LRS3, with LRS2 results given for models with available checkpoints [2603.12046].

Noise is introduced at
\[
\infty,\ 10,\ 5,\ 0,\ -5,\ -10\ \text{dB}.
\]
The main corruption is babble noise, drawn from NOISEX for LLM-based systems and Auto-AVSR, and from MUSAN for Whisper-Flamingo and AV-HuBERT. Additional MUSAN categories—music, environmental sound, and speech—are also tested [2603.12046].

Several implementation choices are specific to fair multimodal attribution. For LLM-based models, audio features are downsampled by \(4\times\) and video features by \(2\times\), making both token streams operate at 12.5 Hz. For Whisper-Flamingo, because Whisper audio is at 50 Hz and AV-HuBERT video at 25 fps, grouped audio masking is used: one SHAP mask element controls two consecutive audio features, so one masked audio element and one video element both span \(0.04\) seconds. This is intended to avoid a spurious bias toward the higher-resolution audio stream [2603.12046].

Generative SHAP uses \(W=5\) windows and is evaluated on the 20 longest utterances, about 6 seconds each, to stabilize the windowed estimates. Temporal alignment visualizations use \(K=W=10\) for fine-grained heatmaps and \(K=3\), \(W=10\) for grouped early/middle/late analyses [2603.12046].

## 5. Empirical findings

The central empirical result is that AVSR models shift toward visual reliance as SNR decreases, but retain a persistent audio contribution even under severe degradation. On LRS3 at \(-10\) dB, most models still assign roughly \(39\%\) to \(46\%\) contribution to audio, whereas in clean conditions audio contribution is roughly \(63\%\) to \(73\%\). The paper interprets this as adaptation under noise, but not full reliability-aware reweighting [2603.12046].

Architectural differences are pronounced. Whisper-Flamingo and AV-HuBERT exhibit the largest adaptation ranges, about 30 to 34 percentage points between clean conditions and \(-10\) dB. Their dynamics differ: Whisper-Flamingo adapts rapidly in negative SNRs and then plateaus, while AV-HuBERT shifts more gradually across the full SNR range. Among LLM-based systems, Llama-SMoP shows the greatest visual reliance at extreme noise, with about \(39\%\) audio at \(-10\) dB, while Omni-AVSR remains comparatively audio-reliant across conditions. Auto-AVSR is the main outlier, remaining around \(57\%\) audio across all SNR levels and showing almost no adaptation; the paper attributes this to its MLP-based fusion [2603.12046].

Generative SHAP shows that modality balance need not be stationary during decoding. In clean conditions, Whisper-Flamingo and Omni-AVSR increase audio reliance over generation, from about \(65\%\) to \(71\%\) audio and from about \(63\%\) to \(72\%\) audio, respectively. Under noisy conditions, both show a U-shaped pattern: more balanced early decoding, a mid-sequence shift toward vision, and later recovery of audio reliance. AV-HuBERT is distinctive in keeping modality balance nearly stable, with less than 3 percentage points variation across generation in both clean and noisy conditions [2603.12046].

Temporal Alignment SHAP indicates that both modalities preserve input-output temporal structure. For AV-HuBERT, the reported diagonal alignment score for the audio heatmap is 2.90 in clean conditions and 1.70 under severe noise. Although the diagonal weakens under noise, grouped early/middle/late analyses still show that early input regions contribute most to early output regions, middle to middle, and late to late, for both audio and video [2603.12046].

Additional factor analyses show that SNR is the dominant driver of modality weighting. When utterances are binned by word error rate at fixed SNR, audio contribution changes only modestly within each SNR bin—typically by around 3 to 7 percentage points—whereas the shift across SNR conditions can reach 27 points for Whisper-Flamingo. The paper therefore treats acoustic condition, especially SNR, as the primary control variable for modality balance rather than recognition difficulty per se [2603.12046].

## 6. Significance, methodological position, and limitations

Dr. SHAP-AV’s immediate significance is diagnostic rather than architectural. It does not propose a new AVSR model; it proposes a way to measure how existing AVSR systems use modalities. The paper’s principal claim is that AVSR research should not stop at word error rate, because performance alone does not disclose whether audio and video are being balanced appropriately. On that basis it explicitly motivates ad-hoc modality-weighting mechanisms and presents Shapley-based attribution as a standard AVSR diagnostic [2603.12046].

Methodologically, Dr. SHAP-AV sits within the branch of SHAP research that applies approximate coalition sampling to complex black-box models rather than exploiting exact structure. Other SHAP work has shown that exact computation can become polynomial when low interaction order or model decomposition is available [2309.02417], and that some additive-model formulations can make Shapley values available in a single forward pass when the explanation target is representable by a suitable additive architecture [2502.14177]. Dr. SHAP-AV assumes no such tractable structure for AVSR and therefore relies on sampled coalition estimators over masked multimodal features [2603.12046].

Its limitations are explicit. Zero-masking absent features is only an approximation to the conditional expectation written in the characteristic function. The method is approximate rather than exact because it depends on Monte Carlo SHAP with \(M=2000\). It is also performance-agnostic: SHAP does not directly indicate whether a modality helped correctness, only how much it influenced predictions. The temporal-alignment section reports a diagonal alignment score, but the formula for that score is not given in the provided text. Generative SHAP is reported on the 20 longest utterances rather than the full test set, and several architecture-level interpretations—such as decoder attention remaining too audio-focused, or pretraining shaping balance—are interpretive rather than directly causal demonstrations [2603.12046].

A broader implication is that Dr. SHAP-AV analyzes multimodal influence at the token level but does not by itself supply inferential calibration. Separate work on SHAP has developed asymptotically normal estimators and confidence intervals for global aggregates such as mean absolute SHAP and mean squared SHAP [2602.10532]. This suggests, though the paper does not pursue it, that Dr. SHAP-AV’s modality-level aggregates could be embedded in a more formal uncertainty-aware analysis.

In the AVSR literature, the framework’s enduring contribution is its recasting of modality attribution as a structured, token-resolved Shapley problem. It provides a common explanatory language for LLM-based AVSR and cross-attention encoder-decoder AVSR, demonstrates that models do adapt under noise without fully relinquishing audio dominance, and formalizes three complementary views—global balance, decoding-time dynamics, and temporal alignment—through which modality use can be inspected [2603.12046].

Source: https://www.emergentmind.com/topics/dr-shap-av