AD-AVSR: Robust AV Speech Recognition

• AD-AVSR is an asymmetric audiovisual speech recognition framework that redefines cross-modal fusion by treating audio and visual streams with differing temporal and information densities.
• It employs dual-stream audio encoding, an audio-aware visual refinement module, and a cross-modal noise suppression masking module, complemented by a threshold-based selection mechanism to filter weak pairings.
• Empirical evaluations on LRS2 and LRS3 demonstrate significant WER improvements under various noise conditions, emphasizing its robustness and suitability for real-world deployment.

AD-AVSR most specifically denotes “Asymmetric Dual-stream Enhancement for Robust Audio-Visual Speech Recognition”, an AVSR framework designed for conditions in which audio and visual streams are heterogeneous, asymmetric, noisy, or misaligned (Xue et al., 11 Aug 2025). In that formulation, AVSR is treated not as a symmetric fusion problem but as a setting in which the modalities differ in temporal density, reliability, and cross-modal usefulness: a single video frame may correspond to many audio samples, not every audio frame aligns well with every visual frame, and weak or irrelevant audio-visual pairings can degrade recognition (Xue et al., 11 Aug 2025). A plausible implication of the surrounding literature is that AD-AVSR also names a broader research direction concerned with robust audiovisual recognition under adverse acoustic conditions, scarce labels, modality imbalance, overlap, and deployment mismatch (Ma et al., 2023, Cappellazzo et al., 12 Mar 2026, Huang et al., 24 Mar 2026).

1. Conceptual basis and problem setting

The AD-AVSR framework in its strict sense begins from two critiques of much earlier AVSR design. First, many prior systems use unidirectional enhancement, in which one modality is refined by the other but the interaction is not closed-loop. Second, many systems use symmetric fusion, effectively treating audio and video as if they carried comparable structure and information density. The 2025 AD-AVSR paper argues that both assumptions are suboptimal because audio and video are structurally asymmetric and because real AV data contain many weakly correlated or irrelevant cross-modal pairs (Xue et al., 11 Aug 2025).

This emphasis on asymmetry places AD-AVSR within a longer sequence of robustness-oriented AVSR studies. Earlier work on overlapped speech for LRS2 argued that the overlapped audio and mouth ROI video of the target speaker create a setting in which the visual stream remains informative while the audio is corrupted by another talker, and it showed that gated audiovisual integration can outperform audio-only recognition and compete with more complex separation pipelines (Yu et al., 2020). A different expansion of the problem appeared in AVATAR, which targeted unconstrained videos and argued that full-frame RGB context—objects, actions, and background—not only lip crops, can contribute to AV-ASR, especially when the speaker’s mouth is not reliably visible (Gabeur et al., 2022).

Taken together, these lines of work suggest that AD-AVSR is best understood not merely as multimodal fusion, but as a family of methods for deciding how, when, and in which direction information should move between modalities under adverse conditions. In the specific 2025 formulation, that decision is encoded explicitly through asymmetric audio representations, bidirectional enhancement, and thresholded pair selection (Xue et al., 11 Aug 2025).

2. Asymmetric Dual-stream Enhancement architecture

The core AD-AVSR architecture contains three main components: Audio Dual-Stream Encoding, the Audio-aware Visual Refinement Module (AVRM), and the Cross-modal Noise Suppression Masking Module (CMNSM), followed by a Threshold-Based Selection Mechanism (TBSM) (Xue et al., 11 Aug 2025).

The audio dual-stream encoding strategy intentionally constructs two different audio representations from the same source signal. The first stream is a time-domain audio encoding used primarily for visual-enhanced audio in CMNSM; the waveform is processed by 1D CNN + ResNet18 and produces

$f_{a1} \in \mathbb{R}^{T_1 \times C_1}.$

The second stream is a frequency-domain audio encoding used primarily for audio-enhanced visual in AVRM; the audio is transformed into a Mel spectrogram via STFT, processed by a 1D convolution, and then temporally averaged so that every adjacent 25 audio frames are summed or averaged and repeated 25 times, yielding

$f_{a2} \in \mathbb{R}^{T_1 \times C_1}.$

The paper’s interpretation is that the time-domain stream preserves short-term temporal detail, whereas the frequency-domain stream carries structured spectral and semantic cues better suited to refining visual features (Xue et al., 11 Aug 2025).

The Audio-aware Visual Refinement Module is the audio-to-visual branch. Visual features

$$f_v \in \mathbb{R}^{T_1 \times C_1 \times H_1 \times W_1}$$

are extracted, aligned to the same temporal length as the audio representation, and divided into $k$ spatial regions per frame. Audio and visual features are then fused through attention. The paper gives the visual context computation as

$$\text{Context}_v = W_3 \sigma \left( W_1 \cdot \text{flatten}_{H,W}(f_v) + (W_2 \cdot f_{a2}) \mathbf{1}^T \right),$$

followed by

$w_t = \text{Softmax}(\text{Context}_v).$

In effect, the audio stream guides the allocation of attention across spatial visual regions, with the stated goal of focusing on speech-relevant visual regions, especially the lips, and reducing the effect of occlusion and synchronization problems (Xue et al., 11 Aug 2025).

The Cross-modal Noise Suppression Masking Module is the visual-to-audio branch. It first performs cross-modal attention in which audio provides the query and visual features provide key and value:

$$\text{Context}_a = \text{softmax}\left(\frac{Q^t \cdot K^T}{\sqrt{D_1}}\right) \cdot V,$$

with

$$Q^t = f_{a1}^t \cdot W_q, \quad K = f_v \cdot W_k, \quad V = f_v \cdot W_v.$$

From this context, the module generates a learnable mask:

$$m_0 = \text{ReLU}\Big( \text{BN}\big( \text{Conv}(\text{Context}_{a}) \big) \Big),$$

$$m = \text{Sigmoid}\Big( \text{BN}\Big( \text{Conv}\big( \text{ReLU}\big( \text{BN}\big( \text{Conv}(m_0) \big) \big) \big) \Big) \Big),$$

and applies it residually:

$f_{a2} \in \mathbb{R}^{T_1 \times C_1}.$0

The intended behavior is noise-aware refinement: visual cues indicate which parts of the audio representation are more reliable, while the residual connection preserves speech content (Xue et al., 11 Aug 2025).

The Threshold-Based Selection Mechanism addresses the claim that many audio-visual pairs are weak or irrelevant. After AVRM and CMNSM, the features are modeled with a BiLSTM to obtain

$f_{a2} \in \mathbb{R}^{T_1 \times C_1}.$1

Pairwise similarity matrices are then computed as

$f_{a2} \in \mathbb{R}^{T_1 \times C_1}.$2

Threshold pruning retains only sufficiently strong connections:

$f_{a2} \in \mathbb{R}^{T_1 \times C_1}.$3

with

$f_{a2} \in \mathbb{R}^{T_1 \times C_1}.$4

The paper describes this as keeping only positive pairs and discarding weak or negative pairings before feature aggregation (Xue et al., 11 Aug 2025).

3. Objective, datasets, and empirical profile

AD-AVSR is trained end-to-end with a joint CTC/attention objective,

$f_{a2} \in \mathbb{R}^{T_1 \times C_1}.$5

where $f_{a2} \in \mathbb{R}^{T_1 \times C_1}.$6 is the attention-based likelihood, $f_{a2} \in \mathbb{R}^{T_1 \times C_1}.$7 is the CTC-based likelihood, and the implementation uses

$f_{a2} \in \mathbb{R}^{T_1 \times C_1}.$8

(Xue et al., 11 Aug 2025).

The reported evaluation uses the standard English AVSR benchmarks LRS2 and LRS3. For LRS2, the paper reports 144,482 English utterances, 224 hours total, extraction from 482 BBC videos, with about 1,000 validation clips and about 1,200 test clips. For LRS3, it reports 439 hours total, about 120,000 pretraining samples (408 hours), 32,000 train/validation samples (30 hours), and 1,321 test samples (0.9 hours) (Xue et al., 11 Aug 2025).

Under clean and noisy audio corruption at clean, 10, 5, 0, and -5 dB, the paper reports average WERs of 4.8 on LRS2 and 4.0 on LRS3, with clean-condition WERs of 2.4 on LRS2 and 2.0 on LRS3 (Xue et al., 11 Aug 2025). It also reports robustness beyond audio noise alone. On LRS2 under visual occlusion, AD-AVSR achieves an average WER of 5.0; under audio noise, it achieves 5.1. The accompanying discussion characterizes these numbers as evidence that the model remains robust under corruption affecting either or both modalities (Xue et al., 11 Aug 2025).

The ablation study isolates the contributions of the three principal modules. On LRS2 at SNR = -5 dB, the reported progression is:

• baseline: 24.9 WER / 13.6 CER
• + AVRM: 20.2 / 12.4
• + CMNSM: 22.4 / 12.8
• + TBSM: 16.3 / 8.0
• + AVRM + CMNSM: 19.3 / 9.5
• + AVRM + CMNSM + TBSM: 9.4 / 4.9

This makes TBSM the largest single contributor in the reported ablation, while the full combination yields the strongest result (Xue et al., 11 Aug 2025). The same study reports that parameters increase from 74.05M to 90.07M, while FLOPs increase only modestly and training time increases only slightly, indicating that the gain is not solely a consequence of large-scale parameter growth (Xue et al., 11 Aug 2025).

4. Relation to data scaling, pre-trained backbones, and LLM-based AVSR

AD-AVSR sits alongside several distinct strategies for robust AVSR. Some expand the training set with automatically generated labels, some add lightweight audiovisual modules to strong ASR backbones, and others use LLM-centered decoding or rescoring. These approaches are complementary rather than mutually exclusive.

Line of work	Representative mechanism	Representative reported result
Automatic-label scaling	Add automatically transcribed AVSpeech and VoxCeleb2 to LRS2/LRS3	LRS3 AV-ASR WER 0.9% (Ma et al., 2023)
Upstream fusion for pre-trained ASR	Self-supervised fusion module before Whisper	8.3% average WER improvement over AV-HuBERT (Simic et al., 2023)
Noise-specific adapters	Frozen Whisper + AV fusion + LoRA + scenario-specific adapter-sets	Up to 88.5% fewer trainable parameters (Simic et al., 3 Feb 2025)
Sparse projector scaling	Sparse Mixture of Projectors with modality-specific routers/experts	AVSR WER 3.31 / 2.29 / 0.96 (Cappellazzo et al., 20 May 2025)
LLM refinement with visual units	Sparse alignment, adaptive fusion, visual-unit-guided rescoring	AVSR WER 0.75% at 433 h, 0.68% at 1759 h (Su et al., 4 Mar 2026)

The automatic-label approach of Auto-AVSR is especially important for understanding the broader AD-AVSR landscape. That paper uses public pre-trained ASR systems to transcribe unlabelled AVSpeech and VoxCeleb2, then retrains ASR, VSR, and AV-ASR on the augmented set formed from LRS2, LRS3, and the newly transcribed data. Its headline result is 0.9% WER on LRS3, described as a 30% relative improvement over the previous state of the art and achieved using only public datasets and public ASR models (Ma et al., 2023). A notable finding is that better LibriSpeech WER for the pseudo-labeling ASR teacher does not correlate strongly with downstream ASR or VSR after retraining; the paper therefore selects Conformer-Transducer because it gives the best average downstream results rather than because it is the best standalone labeler on LibriSpeech (Ma et al., 2023).

A second major line adds audiovisual conditioning to a pre-trained ASR model rather than training a large AVSR model from scratch. The “Self-Supervised Adaptive AV Fusion Module for Pre-Trained ASR Models” paper places a 12 multi-head cross-attention fusion block upstream of Whisper, uses a 13M trainable-parameter fusion module, and reports an 8.3% average WER improvement over AV-HuBERT across model sizes, noise categories, and a broad SNR range (Simic et al., 2023). The related Adapter-Based Multi-Agent AVSR Extension for Pre-Trained ASR Models keeps Whisper frozen, inserts LoRA adapters into Query, Key, Value, and Output projections with rank 64, trains noise-scenario-specific adapter-sets, and reports performance close to full fine-tuning in many conditions while using up to 88.5% fewer trainable parameters than AV-HuBERT large (Simic et al., 3 Feb 2025).

LLM-based AVSR adds a third axis of development. Llama-SMoP replaces the standard multimodal projector with a Sparse Mixture of Projectors, and its best DEDR configuration uses modality-specific routers and experts; the reported AVSR WERs for Llama 3.2-1B / 3.2-3B / 3.1-8B are 3.31 / 2.29 / 0.96 (Cappellazzo et al., 20 May 2025). AVUR-LLM instead combines Sparse Modality Alignment, Adaptive Modulated Fusion, and Visual Unit-Guided Refinement, and reports AVSR WER of 0.75% at 433 h, 0.68% at 1759 h, and 1.7% at 0 dB SNR, with the abstract stating a 37% relative improvement over the baseline at 0 dB (Su et al., 4 Mar 2026). These results suggest that the asymmetry and adaptive weighting emphasized by AD-AVSR can also be realized within LLM-based architectures, although by different mechanisms.

5. Diagnostics, visual contribution, and real-world failure modes

A central issue in AVSR is that low WER does not by itself reveal whether the model meaningfully exploits vision. Dr. SHAP-AV addresses this by using Shapley attribution to quantify relative modality contributions. Across six models and two benchmarks, it finds that models become more visually reliant as SNR decreases, yet remain substantially audio-dominant. On LRS3, the reported audio contribution in clean conditions is about 63–73%, and even at -10 dB many models remain around 38–46% audio contribution (Cappellazzo et al., 12 Mar 2026). The same paper states that Auto-AVSR shows almost no adaptation at all, staying near constant audio contribution, reinforcing the claim that strong recognition accuracy can coexist with persistent audio bias (Cappellazzo et al., 12 Mar 2026).

The paper “Uncovering the Visual Contribution in Audio-Visual Speech Recognition” makes a related argument through effective SNR gain. Using a 0 dB reference, it reports effective SNR gains of 3.7 dB on LRS2 and 2.5 dB on LRS3 for Auto-AVSR, 4.0 dB and 2.3 dB for AV-RelScore, and 6.1 dB and 4.4 dB for AVEC (Lin et al., 2024). Its main conclusion is explicit: low WER does not imply high visual contribution. In its comparison, Auto-AVSR achieves the best WER overall but the lowest effective SNR gain, whereas AVEC has worse WER and the highest effective SNR gain (Lin et al., 2024). This result is directly relevant to AD-AVSR because it distinguishes end performance from actual multimodal exploitation.

Real-world deployment further complicates the picture. “When AVSR Meets Video Conferencing” reports that Auto-AVSR on LRS3 can degrade from about 0.93% WER / 0.56% CER offline to 33.09% WER / 33.01% CER in VC, with the primary source of distribution shift identified as speech enhancement algorithms used by VC platforms rather than codec compression alone (Huang et al., 24 Mar 2026). That paper introduces MLD-VC, a dataset with 31 speakers, 22.79 hours, 4 mainstream VC platforms, and explicit use of the Lombard effect, and reports an average 17.5% reduction in CER across VC platforms after fine-tuning on MLD-VC (Huang et al., 24 Mar 2026). Its distribution-shift analysis points specifically to upward shifts in $f_{a2} \in \mathbb{R}^{T_1 \times C_1}.$9 and $$f_v \in \mathbb{R}^{T_1 \times C_1 \times H_1 \times W_1}$$0, suggesting that robustness to additive noise is not equivalent to robustness to transmission pipelines and platform enhancement.

A different deployment failure mode appears in cocktail-party AVSR. The paper “Cocktail-Party Audio-Visual Speech Recognition” argues that standard AVSR datasets mostly depict visible faces that are actively speaking, whereas real multi-speaker scenes contain both talking-face and silent-face segments. It introduces AVCocktail, about 6.1 hours from 45 speakers, and AVYT, a 1526-hour training corpus with 77 hours of silent-face clips and 1449 hours of talking-face clips (Nguyen et al., 2 Jun 2025). Under fixed 10-second chunks without segmentation cues, many prior methods exceed 100% WER; the reported headline improvement is a reduction from 119% to 39.2% in extreme noise, described as a 67% relative improvement, achieved without relying on explicit segmentation cues (Nguyen et al., 2 Jun 2025). For AD-AVSR, this is significant because it shows that the correct multimodal action is sometimes to ignore the visible face rather than trust it.

6. Limitations, misconceptions, and research trajectory

Several recurring misconceptions are corrected by the literature around AD-AVSR. One is that a stronger standalone ASR teacher must produce better pseudo-label training data. Auto-AVSR reports the opposite trend: ASR performance on LibriSpeech does not correlate strongly with downstream ASR or VSR after retraining on automatically labelled audiovisual data (Ma et al., 2023). Another is that lower WER necessarily reflects stronger use of the visual modality. Both the effective-SNR analysis and the Shapley analysis contradict that assumption, showing that models can be accurate while remaining heavily audio-biased (Lin et al., 2024, Cappellazzo et al., 12 Mar 2026).

A third misconception is that robustness under synthetic or offline noise is sufficient evidence of robustness in deployment. The VC study shows severe collapse under real conferencing pipelines, with speech enhancement rather than compression identified as the major source of acoustic drift (Huang et al., 24 Mar 2026). The cocktail-party study shows a different mismatch: a visible face cannot be assumed to be an actively speaking face, and fixed-chunk evaluation can produce a majority of silent-face segments (Nguyen et al., 2 Jun 2025). These findings suggest that adverse-domain AVSR must handle both modality corruption and modality irrelevance.

The strict 2025 AD-AVSR formulation has its own bounded scope. Its empirical evidence is centered on LRS2 and LRS3, and its central innovations—dual-stream audio encoding, AVRM, CMNSM, and TBSM—are validated mainly through those benchmarks and controlled noise settings (Xue et al., 11 Aug 2025). A plausible implication is that the thresholding and bidirectional enhancement design should be tested more extensively in settings with platform-induced distortion, silent-face ambiguity, and cross-lingual variation. That implication is consistent with adjacent papers that recommend evaluating AVSR in real VC settings, reporting effective SNR gain alongside WER, including human hyper-expression / Lombard speech and silent-face clips in training, and using explicit modality-weighting analyses such as Shapley attribution to verify whether the visual stream is actually being used (Huang et al., 24 Mar 2026, Lin et al., 2024, Nguyen et al., 2 Jun 2025, Cappellazzo et al., 12 Mar 2026).

In that broader trajectory, AD-AVSR marks a shift from simple audiovisual concatenation toward structured cross-modal control. Its defining claim is not merely that audio and vision should be fused, but that they should be fused asymmetrically, refined bidirectionally, and filtered so that only strong cross-modal correspondences survive (Xue et al., 11 Aug 2025). The surrounding literature suggests that this claim aligns with several emerging design principles in AVSR: more training data can help even when labels are noisy, fusion should be condition-dependent rather than static, evaluation should separate accuracy from true visual contribution, and robustness must be demonstrated under the specific distribution shifts of the intended deployment environment (Ma et al., 2023, Cappellazzo et al., 12 Mar 2026, Lin et al., 2024, Huang et al., 24 Mar 2026).