Dr. SHAP-AV: Shapley Analysis for AVSR

• Dr. SHAP-AV is a framework that employs Shapley-based feature attribution to analyze the relative contributions of audio and visual modalities in speech recognition.
• It introduces three complementary analyses—Global SHAP, Generative SHAP, and Temporal Alignment SHAP—to quantify modality dynamics and temporal alignment during decoding.
• Empirical findings show that while AVSR models shift toward visual reliance under noise, a persistent audio bias remains, highlighting the need for balanced multimodal integration.

Dr. SHAP-AV is a framework for analyzing relative modality contributions in audio-visual speech recognition (AVSR) with Shapley values. It treats audio and visual inputs as cooperative-game players and attributes the model’s token-level decoding behavior to them through Shapley-based feature attribution. Applied to six AVSR models across LRS2 and LRS3 under clean and noisy conditions, it introduces three complementary analyses—Global SHAP, Generative SHAP, and Temporal Alignment SHAP—and reports that AVSR systems shift toward visual reliance as acoustic quality degrades, yet often retain substantial audio contribution even at severe noise levels, thereby exposing a persistent audio bias (Cappellazzo et al., 12 Mar 2026).

1. Problem setting and motivation

AVSR combines acoustic and visual speech cues, typically to improve robustness under noise. Dr. SHAP-AV is motivated by a specific interpretability gap: AVSR work has usually emphasized word error rate rather than a principled account of how models balance modalities. In this setting, modality use is not a secondary concern. AVSR is a multimodal reliability-weighting problem, and a system that remains overly audio-driven under severe corruption may fail to exploit the visual stream when it is comparatively more reliable (Cappellazzo et al., 12 Mar 2026).

The framework is introduced against a preliminary observation from drop-modality experiments in clean conditions: removing video often leaves word error rate close to full AVSR, whereas removing audio causes catastrophic degradation for most models. This indicates a strong audio bias, especially in models trained only for AVSR rather than explicitly for visual-only recognition. Dr. SHAP-AV therefore asks how much audio and video contribute, how that balance evolves during autoregressive generation, and whether input-output temporal alignment is preserved across acoustic conditions (Cappellazzo et al., 12 Mar 2026).

Its target is not sentence-level correctness and not word error rate directly. Instead, it attributes the model’s token-by-token decoding behavior. This choice matters because modality balance can vary across output positions even within a single utterance, and because a coarse ablation analysis cannot recover the contribution structure of the generated sequence.

2. Shapley formulation for AVSR decoding

Let a pretrained AVSR model $f$ generate a token sequence $\mathbf y=(y_1,\dots,y_T)$ from audio features $\mathbf x^A=(x_1^A,\dots,x_{N_A}^A)$ and visual features $\mathbf x^V=(x_1^V,\dots,x_{N_V}^V)$. Dr. SHAP-AV defines the full feature set as

$\mathcal F=\mathcal A\cup\mathcal V,$

with $\mathcal A$ indexing audio features and $\mathcal V$ visual features. For each output token $y_t$, the characteristic function on a coalition $\mathcal C\subseteq\mathcal F$ is

$$f_{\mathbf x}^t(\mathcal C)=\mathbb E\left[\log p(y_t \mid \mathbf X_{\mathcal C}, y_{<t}) \mid \mathbf X_{\mathcal C}=\mathbf x_{\mathcal C}\right].$$

Thus the payoff is the expected log-probability of the generated token conditioned on preceding tokens and a subset of observed input features (Cappellazzo et al., 12 Mar 2026).

The token-level Shapley value for feature $\mathbf y=(y_1,\dots,y_T)$0 is

$\mathbf y=(y_1,\dots,y_T)$1

Collecting $\mathbf y=(y_1,\dots,y_T)$2 across features and output positions yields a Shapley matrix

$\mathbf y=(y_1,\dots,y_T)$3

Rows represent input features; columns represent generated tokens (Cappellazzo et al., 12 Mar 2026).

Exact computation is intractable, so the paper uses Permutation SHAP: $\mathbf y=(y_1,\dots,y_T)$4 where $\mathbf y=(y_1,\dots,y_T)$5 is the number of sampled coalitions or permutations. The reported setting is

$\mathbf y=(y_1,\dots,y_T)$6

chosen because lower $\mathbf y=(y_1,\dots,y_T)$7 produced higher variance, especially at low SNR. Sampling SHAP was also tested and gave similar results, but Permutation SHAP was used for subsequent analyses because it was faster (Cappellazzo et al., 12 Mar 2026).

Although the characteristic function is written as a conditional expectation, the implementation approximates it by masking absent features and performing a forward pass rather than by true marginalization. The masking scheme is architecture-specific. For LLM-based AVSR, masking is applied after modality-specific projection layers; for AV-HuBERT and Auto-AVSR, before fusion; for Whisper-Flamingo, before decoder cross-attention consumes the modality-specific features (Cappellazzo et al., 12 Mar 2026).

3. The three analyses

Dr. SHAP-AV organizes token-level feature attributions into three analyses.

Analysis	Unit of aggregation	Purpose
Global SHAP	All features, all output tokens	Overall audio-vs-video balance
Generative SHAP	Output-token windows	Contribution dynamics during decoding
Temporal Alignment SHAP	Input bins $\mathbf y=(y_1,\dots,y_T)$8 output bins	Input-output correspondence

Global SHAP defines modality-level proportions by aggregating absolute Shapley mass: $\mathbf y=(y_1,\dots,y_T)$9 By construction,

$\mathbf x^A=(x_1^A,\dots,x_{N_A}^A)$0

A value of $\mathbf x^A=(x_1^A,\dots,x_{N_A}^A)$1 indicates balance; values above $\mathbf x^A=(x_1^A,\dots,x_{N_A}^A)$2 indicate audio dominance (Cappellazzo et al., 12 Mar 2026).

Generative SHAP partitions the output sequence into windows $\mathbf x^A=(x_1^A,\dots,x_{N_A}^A)$3 and computes per-window modality ratios: $\mathbf x^A=(x_1^A,\dots,x_{N_A}^A)$4 This produces decoding-time trajectories rather than a single utterance-level summary (Cappellazzo et al., 12 Mar 2026).

Temporal Alignment SHAP normalizes both input positions and output positions by percentage of utterance length. For modality $\mathbf x^A=(x_1^A,\dots,x_{N_A}^A)$5, input features are partitioned into $\mathbf x^A=(x_1^A,\dots,x_{N_A}^A)$6 bins $\mathbf x^A=(x_1^A,\dots,x_{N_A}^A)$7, output tokens into $\mathbf x^A=(x_1^A,\dots,x_{N_A}^A)$8 bins $\mathbf x^A=(x_1^A,\dots,x_{N_A}^A)$9, and the alignment matrix is

$\mathbf x^V=(x_1^V,\dots,x_{N_V}^V)$0

Each row sums to one. Diagonal structure in $\mathbf x^V=(x_1^V,\dots,x_{N_V}^V)$1 is interpreted as temporal alignment between input and output progression (Cappellazzo et al., 12 Mar 2026).

4. Models, benchmarks, and implementation choices

The framework is evaluated on six AVSR systems across two model families and two benchmarks.

Family	Models	Notes
LLM-based AVSR	Llama-AVSR, Llama-SMoP, Omni-AVSR	Whisper medium audio, AV-HuBERT Large video, Llama 3.2-1B backbone
Cross-attention / encoder-decoder AVSR	AV-HuBERT, Auto-AVSR, Whisper-Flamingo	Architecturally distinct fusion and decoding schemes

The benchmarks are LRS2 and LRS3. LRS2 contains 225 hours of BBC program footage; LRS3 contains 433 hours of TED-talk video. Most detailed analyses are reported on LRS3, with LRS2 results given for models with available checkpoints (Cappellazzo et al., 12 Mar 2026).

Noise is introduced at

$\mathbf x^V=(x_1^V,\dots,x_{N_V}^V)$2

The main corruption is babble noise, drawn from NOISEX for LLM-based systems and Auto-AVSR, and from MUSAN for Whisper-Flamingo and AV-HuBERT. Additional MUSAN categories—music, environmental sound, and speech—are also tested (Cappellazzo et al., 12 Mar 2026).

Several implementation choices are specific to fair multimodal attribution. For LLM-based models, audio features are downsampled by $\mathbf x^V=(x_1^V,\dots,x_{N_V}^V)$3 and video features by $\mathbf x^V=(x_1^V,\dots,x_{N_V}^V)$4, making both token streams operate at 12.5 Hz. For Whisper-Flamingo, because Whisper audio is at 50 Hz and AV-HuBERT video at 25 fps, grouped audio masking is used: one SHAP mask element controls two consecutive audio features, so one masked audio element and one video element both span $\mathbf x^V=(x_1^V,\dots,x_{N_V}^V)$5 seconds. This is intended to avoid a spurious bias toward the higher-resolution audio stream (Cappellazzo et al., 12 Mar 2026).

Generative SHAP uses $\mathbf x^V=(x_1^V,\dots,x_{N_V}^V)$6 windows and is evaluated on the 20 longest utterances, about 6 seconds each, to stabilize the windowed estimates. Temporal alignment visualizations use $\mathbf x^V=(x_1^V,\dots,x_{N_V}^V)$7 for fine-grained heatmaps and $\mathbf x^V=(x_1^V,\dots,x_{N_V}^V)$8, $\mathbf x^V=(x_1^V,\dots,x_{N_V}^V)$9 for grouped early/middle/late analyses (Cappellazzo et al., 12 Mar 2026).

5. Empirical findings

The central empirical result is that AVSR models shift toward visual reliance as SNR decreases, but retain a persistent audio contribution even under severe degradation. On LRS3 at $\mathcal F=\mathcal A\cup\mathcal V,$0 dB, most models still assign roughly $\mathcal F=\mathcal A\cup\mathcal V,$1 to $\mathcal F=\mathcal A\cup\mathcal V,$2 contribution to audio, whereas in clean conditions audio contribution is roughly $\mathcal F=\mathcal A\cup\mathcal V,$3 to $\mathcal F=\mathcal A\cup\mathcal V,$4. The paper interprets this as adaptation under noise, but not full reliability-aware reweighting (Cappellazzo et al., 12 Mar 2026).

Architectural differences are pronounced. Whisper-Flamingo and AV-HuBERT exhibit the largest adaptation ranges, about 30 to 34 percentage points between clean conditions and $\mathcal F=\mathcal A\cup\mathcal V,$5 dB. Their dynamics differ: Whisper-Flamingo adapts rapidly in negative SNRs and then plateaus, while AV-HuBERT shifts more gradually across the full SNR range. Among LLM-based systems, Llama-SMoP shows the greatest visual reliance at extreme noise, with about $\mathcal F=\mathcal A\cup\mathcal V,$6 audio at $\mathcal F=\mathcal A\cup\mathcal V,$7 dB, while Omni-AVSR remains comparatively audio-reliant across conditions. Auto-AVSR is the main outlier, remaining around $\mathcal F=\mathcal A\cup\mathcal V,$8 audio across all SNR levels and showing almost no adaptation; the paper attributes this to its MLP-based fusion (Cappellazzo et al., 12 Mar 2026).

Generative SHAP shows that modality balance need not be stationary during decoding. In clean conditions, Whisper-Flamingo and Omni-AVSR increase audio reliance over generation, from about $\mathcal F=\mathcal A\cup\mathcal V,$9 to $\mathcal A$0 audio and from about $\mathcal A$1 to $\mathcal A$2 audio, respectively. Under noisy conditions, both show a U-shaped pattern: more balanced early decoding, a mid-sequence shift toward vision, and later recovery of audio reliance. AV-HuBERT is distinctive in keeping modality balance nearly stable, with less than 3 percentage points variation across generation in both clean and noisy conditions (Cappellazzo et al., 12 Mar 2026).

Temporal Alignment SHAP indicates that both modalities preserve input-output temporal structure. For AV-HuBERT, the reported diagonal alignment score for the audio heatmap is 2.90 in clean conditions and 1.70 under severe noise. Although the diagonal weakens under noise, grouped early/middle/late analyses still show that early input regions contribute most to early output regions, middle to middle, and late to late, for both audio and video (Cappellazzo et al., 12 Mar 2026).

Additional factor analyses show that SNR is the dominant driver of modality weighting. When utterances are binned by word error rate at fixed SNR, audio contribution changes only modestly within each SNR bin—typically by around 3 to 7 percentage points—whereas the shift across SNR conditions can reach 27 points for Whisper-Flamingo. The paper therefore treats acoustic condition, especially SNR, as the primary control variable for modality balance rather than recognition difficulty per se (Cappellazzo et al., 12 Mar 2026).

6. Significance, methodological position, and limitations

Dr. SHAP-AV’s immediate significance is diagnostic rather than architectural. It does not propose a new AVSR model; it proposes a way to measure how existing AVSR systems use modalities. The paper’s principal claim is that AVSR research should not stop at word error rate, because performance alone does not disclose whether audio and video are being balanced appropriately. On that basis it explicitly motivates ad-hoc modality-weighting mechanisms and presents Shapley-based attribution as a standard AVSR diagnostic (Cappellazzo et al., 12 Mar 2026).

Methodologically, Dr. SHAP-AV sits within the branch of SHAP research that applies approximate coalition sampling to complex black-box models rather than exploiting exact structure. Other SHAP work has shown that exact computation can become polynomial when low interaction order or model decomposition is available (Hu et al., 2023), and that some additive-model formulations can make Shapley values available in a single forward pass when the explanation target is representable by a suitable additive architecture (Enouen et al., 20 Feb 2025). Dr. SHAP-AV assumes no such tractable structure for AVSR and therefore relies on sampled coalition estimators over masked multimodal features (Cappellazzo et al., 12 Mar 2026).

Its limitations are explicit. Zero-masking absent features is only an approximation to the conditional expectation written in the characteristic function. The method is approximate rather than exact because it depends on Monte Carlo SHAP with $\mathcal A$3. It is also performance-agnostic: SHAP does not directly indicate whether a modality helped correctness, only how much it influenced predictions. The temporal-alignment section reports a diagonal alignment score, but the formula for that score is not given in the provided text. Generative SHAP is reported on the 20 longest utterances rather than the full test set, and several architecture-level interpretations—such as decoder attention remaining too audio-focused, or pretraining shaping balance—are interpretive rather than directly causal demonstrations (Cappellazzo et al., 12 Mar 2026).

A broader implication is that Dr. SHAP-AV analyzes multimodal influence at the token level but does not by itself supply inferential calibration. Separate work on SHAP has developed asymptotically normal estimators and confidence intervals for global aggregates such as mean absolute SHAP and mean squared SHAP (Whitehouse et al., 11 Feb 2026). This suggests, though the paper does not pursue it, that Dr. SHAP-AV’s modality-level aggregates could be embedded in a more formal uncertainty-aware analysis.

In the AVSR literature, the framework’s enduring contribution is its recasting of modality attribution as a structured, token-resolved Shapley problem. It provides a common explanatory language for LLM-based AVSR and cross-attention encoder-decoder AVSR, demonstrates that models do adapt under noise without fully relinquishing audio dominance, and formalizes three complementary views—global balance, decoding-time dynamics, and temporal alignment—through which modality use can be inspected (Cappellazzo et al., 12 Mar 2026).