BasicAVSR: Baseline Audiovisual Speech Recognition
- BasicAVSR is a baseline framework for audiovisual speech recognition that combines synchronized acoustic signals with mouth-region visuals to enhance performance under noise.
- It utilizes modality-specific front ends, temporal encoders, and fusion mechanisms to accurately align and integrate features from both audio and visual channels.
- Recent advancements leverage pretrained multimodal encoders and adaptive fusion strategies, significantly reducing word error rates and broadening task applications.
Searching arXiv for papers on “BasicAVSR” and core AVSR baselines. Searching arXiv for “BasicAVSR audiovisual speech recognition” and related AVSR baseline papers. BasicAVSR, in current speech-recognition usage, most plausibly denotes a baseline-style audio-visual speech recognition configuration rather than a single universally standardized speech model. Across the AVSR literature, the expression is used to refer to systems that combine synchronized acoustic speech with visual speech cues, usually from the mouth region, through modality-specific front ends, a fusion mechanism, and a sequence recognizer trained for transcription. This suggests a design family rather than a fixed architecture: recurrent CTC systems, hybrid LF-MMI recognizers, Conformer and AV-HuBERT encoder-decoder models, and Whisper- or LLM-based extensions all instantiate the same underlying problem of aligning and exploiting audio and visual speech, especially when the acoustic channel is degraded (Sanabria et al., 2016, Yu et al., 2020, Shi et al., 2022, Su et al., 4 Mar 2026).
1. Historical formation of the baseline AVSR paradigm
Early systems established the core BasicAVSR pattern: synchronized audio and mouth-region video are encoded separately or jointly, then fused for recognition. In one 2016 large-vocabulary end-to-end formulation on IBM ViaVoice, audio, video, and audiovisual inputs were all processed by a 4-layer BLSTM trained with CTC; the audiovisual system used feature concatenation and benefited especially under noisy conditions, while alignment analysis showed that visual CTC peaks preceded audio peaks (Sanabria et al., 2016). In the same year, a GRID-based character-level system used a deep bidirectional LSTM acoustic model trained with CTC, a visual bottleneck network derived from mouth-region DCT features, and both feature fusion and decision fusion; under babble noise, visual information sharply reduced CER even though the model had been trained on clean data (Thanda et al., 2016).
A second early line emphasized explicit multimodal temporal modeling. The Auxiliary Multimodal LSTM used parallel audio and video LSTMs, a shared-space fusion layer, and auxiliary modality-specific supervision, and was evaluated on AVLetters, AVLetters2, and AVDigits rather than continuous sentence transcription (Tian et al., 2017). A related end-to-end study on GRID then argued that fusion should occur after some modality-specific processing rather than at raw-feature level: its best architecture was an intermediate-fusion “2 + 2” network with hidden size 128, using 27 log-Mel audio features and 80×40 grayscale mouth ROIs upsampled to 100 fps (Wand et al., 2018).
This trajectory suggests that “basic” AVSR was historically defined less by one canonical network than by a recurring set of engineering commitments: synchronized multimodal input, explicit temporal modeling, and evaluation under acoustic noise.
2. Canonical architectural structure
Despite differences in scale and objective, representative BasicAVSR-style systems share a stable decomposition into front ends, temporal encoders, fusion, and decoding. In contemporary work, this scaffold appears in recurrent, hybrid, Conformer, self-supervised, and ASR-adapter formulations.
| Family | Encoder and fusion pattern | Output formulation |
|---|---|---|
| BLSTM-CTC AVSR | Early feature fusion of synchronized audio and visual features | CTC over phonemes, visemes, or characters |
| Hybrid LF-MMI AVSR | Acoustic TDNN/TDNN-F with visual features from LipNet; concat or gated fusion | Grapheme-state LF-MMI decoding |
| Conformer / AV-HuBERT AVSR | Separate or pretrained multimodal encoders with mid-level fusion | Joint CTC/attention or seq2seq |
| Pretrained ASR extension | Lightweight AV fusion module or adapters before frozen Whisper-style ASR | Whisper decoding or rescoring |
In recurrent baselines, the architecture is often minimal: audio and video feature vectors are concatenated and passed into a shared BLSTM stack trained with CTC (Sanabria et al., 2016). Hybrid systems replace end-to-end decoding with LF-MMI sequence discrimination and graph-based decoding. On LRS2, a Kaldi-based audiovisual recognizer used 40-bin log Mel-filterbanks, a pretrained LipNet visual front end, grapheme-state units, a TDNN/TDNN-F recognition network, and LF-MMI with CE regularization; on clean speech it reported 5.93 WER for A+V versus 6.71 for audio-only (Yu et al., 2020).
Modern baseline formulations are predominantly encoder-decoder. A zero-AV-resource Catalan system initialized from AV-HuBERT Large pretrained on English LRS3 and VoxCeleb2, attached a randomly initialized 6-layer Transformer decoder, used SentencePiece unigram subwords, and fine-tuned the full model in a sequence-to-sequence setting while freezing the pretrained encoder for the first 22,500 updates (Buitrago et al., 9 Mar 2026). Cocktail-party AVSR baselines similarly used AV-HuBERT or Conformer encoders with 6-layer Transformer decoders and joint CTC/attention training (Nguyen et al., 2 Jun 2025). LLM-based extensions retain the same backbone logic while adding sparse alignment or rescoring layers rather than replacing the recognizer wholesale (Su et al., 4 Mar 2026).
This suggests that the architectural center of BasicAVSR has shifted from handcrafted front ends and recurrent sequence models toward pretrained multimodal encoders and decoder-side sequence modeling, while preserving the same modular structure.
3. Inputs, preprocessing, synchronization, and task regimes
BasicAVSR-style systems are strongly shaped by input design. Audio inputs range from 40-bin log Mel-filterbanks with 40 ms windows and 10 ms hops (Yu et al., 2020), to 80-dimensional log-mel or filter-bank features at 16 kHz (Gabeur et al., 2022, Serdyuk et al., 2021), to stacked 240-dimensional acoustic features at 33.3 Hz (Serdyuk et al., 2021). Visual inputs are usually mouth-centered and low-resolution by image-recognition standards: 64×64 mouth-and-chin crops from GRID (Thanda et al., 2016), 80×40 grayscale mouth ROIs (Wand et al., 2018), 96×96 lip-centered crops following AV-HuBERT (Simic et al., 2023, Su et al., 4 Mar 2026), and 112×112 center crops on LRS2 mouth ROI (Yu et al., 2020). The visual stream is commonly converted to grayscale, normalized, and resampled or upsampled to match the acoustic frame rate.
Temporal synchronization is explicit. One GRID system upsampled video by a factor of 4 to 100 fps to match audio features (Wand et al., 2018); another used four contiguous audio frames per video frame and length equalization by truncation or completion (Tian et al., 2017); a transformer front end for AV-ASR resampled video to 33.3 Hz using nearest-neighbor interpolation to align with stacked acoustic features (Serdyuk et al., 2021). These design choices encode a core AVSR assumption: audio and visual streams are temporally correlated, but alignment must be engineered at the feature level before sequence learning can exploit that correlation.
Task formulations vary widely. GRID-based systems treat the problem as isolated-word or character recognition (Thanda et al., 2016, Wand et al., 2018). LRS2 and LRS3 support continuous sentence transcription and larger-vocabulary decoding (Yu et al., 2020, Shi et al., 2022). How2 and VisSpeech represent a different, unconstrained regime in which full-frame RGB may encode scene semantics, objects, and actions rather than only lip articulation; AVATAR explicitly used two full RGB frames sampled at 2.5 fps and fused them through MBT bottleneck tokens, while also introducing word-masking strategies to prevent audio dominance (Gabeur et al., 2022).
A frequent misconception is that all BasicAVSR systems are lipreading-centric. The literature instead contains two partially overlapping regimes: a mouth-ROI tradition centered on articulatory synchrony, and a full-frame contextual tradition in which visual context can help even when the mouth is not visible (Gabeur et al., 2022).
4. Fusion mechanisms and the problem of robustness
Fusion design is the central technical fault line in BasicAVSR research. The 2018 GRID study showed that early fusion was substantially worse than intermediate fusion, partly because the audio stream had 27 dimensions per frame while the visual stream had 3200 raw pixel values; its best “2 + 2” architecture fused after two modality-specific layers per stream (Wand et al., 2018). This established a durable principle: some modality-specific abstraction is usually beneficial before fusion.
Later work moved from concatenation toward selective or adaptive interaction. In overlapped-speech recognition on LRS2, simple feature-concatenation AVSR reached average WER 15.20, while gated integrated models reduced this to 10.31; the strongest configuration used an audio-visual-driven gate plus residual visual concatenation (Yu et al., 2020). In noise-robust Conformer AVSR, CoBRA introduced 32 learnable bottleneck tokens beginning at the 4th encoder layer and used them as the sole channel for cross-modal exchange. Its main configuration reduced LRS3 clean WER from 2.30 to 1.96 relative to the reproduced baseline and lowered babble-noise WER at dB from 18.58 to 11.79 (Ok et al., 9 Feb 2026).
Adaptive fusion can also be driven by estimated acoustic unreliability. AVUR-LLM inserted Sparse Modality Alignment into upper Whisper encoder layers, then used Adaptive Modulated Fusion in the decoder, with token-wise visual injection scaled by acoustic uncertainty. On noisy LRS3 at 0 dB, the full system reported 1.70 WER; removing Visual Unit-Guided Refinement raised this to 5.50, and retaining SMA without AMF or VUR yielded 6.70 (Su et al., 4 Mar 2026). A smaller-scale precursor, built around a cross-attentive upstream module in front of Whisper, reported an average 8.3% WER improvement over AV-HuBERT across model sizes, noise categories, and SNRs, while using a 13M-parameter fusion module and substantially fewer trainable parameters than training a full AVSR transformer from scratch (Simic et al., 2023).
These results collectively suggest that BasicAVSR should not be understood as mere feature concatenation. Its decisive design question is how the visual stream is permitted to influence the acoustic stream when the latter becomes unreliable.
5. Training regimes, transfer, and zero-AV-resource bootstrapping
The contemporary BasicAVSR literature is marked by three training regimes: fully supervised audiovisual training, self-supervised multimodal pretraining followed by fine-tuning, and retrofitting of strong audio-only ASR backbones.
Self-supervised AV-HuBERT is the clearest turning point. Using unlabeled English VoxCeleb2 for pretraining and LRS3 for fine-tuning, a robust AVSR framework built upon AV-HuBERT reported 1.4 WER on clean LRS3 with 433 hours of labels, and under babble noise reduced average WER from 28.0 in prior work to 14.1 with only 30 hours of labeled data. In the same study, an audio-only model with noisy pretraining had average noisy WER 25.8, whereas the audio-visual model reached 5.8 (Shi et al., 2022). This made self-supervised pretraining a core component of strong baseline AVSR rather than an optional enhancement.
The zero-AV-resource setting extends the same baseline logic to languages without audiovisual corpora. A Catalan study defined this setting as the absence of paired audiovisual training data despite the possible availability of transcribed audio corpora. It selected static face images from FFHQ, filtered them with a morphological face detector, paired each utterance with a randomly selected face, and animated the still image with a pre-trained Wav2Lip+GAN model to synthesize talking-head video. The resulting Catalan corpus comprised about 723 hours of synthetic audiovisual data; fine-tuning AV-HuBERT Large on real audio plus synthetic video yielded 19.6% WER in AV mode, compared with 23.1% when the visual input was masked and 105% in video-only mode, on a manually verified Catalan benchmark of 51 minutes and 38 seconds (Buitrago et al., 9 Mar 2026). The same paper reported that adding synthetic visual data to real Spanish AV corpora reduced WER from 9.3 to 8.1 on LIP-RTVE and from 15.4 to 12.9 on CMU-MOSEAS (Buitrago et al., 9 Mar 2026).
A third regime adapts pretrained ASR rather than training a multimodal encoder from scratch. One adapter-based approach extended Whisper with an AV fusion module and LoRA adapters, kept the Whisper backbone frozen, and routed among noise-specific adapter sets via a separate classifier. It reported almost comparable performance to full fine-tuning in most conditions with up to 88.5% fewer trainable parameters than AV-HuBERT large (Simic et al., 3 Feb 2025). This suggests that, in practice, BasicAVSR can denote not only a native multimodal recognizer but also a parameter-efficient audiovisual extension of a strong audio-only foundation model.
6. Evaluation practice, common misconceptions, and terminological ambiguity
Evaluation is as constitutive of BasicAVSR as architecture. Many papers report matched AV, audio-only, and video-only conditions from the same trained model or backbone (Buitrago et al., 9 Mar 2026, Yu et al., 2020). Noise robustness is typically probed with controlled SNR sweeps, including babble, music, natural sounds, white Gaussian noise, or competing speech (Shi et al., 2022, Buitrago et al., 9 Mar 2026, Ok et al., 9 Feb 2026). In more realistic settings, segmentation becomes part of the problem: cocktail-party AVSR on AVCocktail evaluated ASD-based segmentation, fixed 10-second chunks, and gold segmentation, and showed that a fine-tuned AV-HuBERT CTC/Attention model could reduce WER in the hardest fixed-chunk condition from 119.0 for Muavic-EN to 39.2 after training with interferer augmentation, dialog concatenation, and silent-face data (Nguyen et al., 2 Jun 2025).
One major misconception is that low WER automatically implies strong visual use. A human-perception-inspired analysis explicitly rejected that conclusion by comparing Auto-AVSR, AVEC, and AV-RelScore under pink noise. On LRS2, effective SNR gains were 3.7 dB for Auto-AVSR, 4.0 dB for AV-RelScore, and 6.1 dB for AVEC; on LRS3, the values were 2.5, 2.3, and 4.4 dB, respectively. The same study found that the best WER did not coincide with the strongest evidence of visual contribution, and recommended reporting effective SNR gain alongside WER (Lin et al., 2024). A related misconception is that a visible face always implies target speech activity; AVCocktail and AVYT directly challenged this by incorporating silent-face segments, with AVYT contributing 1449 hours of talking-face clips and 77 hours of silent-face clips (Nguyen et al., 2 Jun 2025).
The term itself is also ambiguous. In the speech-recognition literature summarized here, “BasicAVSR” functions descriptively, as a baseline audiovisual speech-recognition template. By contrast, “BasicAVSR” is also the title of an unrelated 2025 model for arbitrary-scale video super-resolution, which integrates adaptive multi-scale frequency priors, flow-guided propagation, second-order motion compensation, and hyper-upsampling (Shang et al., 30 Oct 2025). This suggests that the label is context-dependent: in AVSR speech research it denotes a baseline family, whereas in computer vision it names a specific super-resolution architecture.