---
title: 'LRS3 AV-ASR Benchmark: Multimodal Speech Recognition'
url: https://www.emergentmind.com/topics/lrs3-av-asr-benchmark
type: topic
---

# LRS3 AV-ASR Benchmark: Multimodal Speech Recognition

The LRS3 AV-ASR benchmark is a critical evaluation corpus and protocol for automatic speech recognition leveraging both audio and video modalities, designed to rigorously assess the integration of multimodal information for robust transcription, especially under noisy conditions and data-scarce regimes. Its adoption has catalyzed methodological advances ranging from self-supervised representation learning and large-scale pseudo-labeling to unified LLM-based frameworks, making it the gold standard for the fair empirical comparison of AV-ASR systems.

## 1. Dataset Composition and Benchmark Protocol

The LRS3 dataset comprises approximately 433 hours of English-language TED and TEDx talk excerpts, with dense frame-level audio and video aligned to transcribed utterances. Standard splits consist of a 408 h pre-train partition, a 30 h “train-val” split for supervised fine-tuning, and a ≈1–1.6 h held-out test set (~1,300 utterances) [2201.10439][2303.12187][2211.02133][2201.02184].

**Audio Processing**
- 16 kHz mono waveforms, log-Mel or filterbank features (26–80 dims), typical frame rates: 25–100 Hz.
- Augmentation: additive babble/pink/music noise (MUSAN, NOISEX-92), time-masking, SpecAugment.

**Video Processing**
- Face detection (Dlib, MediaPipe, RetinaFace), mouth region-of-interest (ROI) crops (e.g., 88×88, 96×96, 128×128 px), usually grayscale, frame rates of 25–33.3 Hz.
- Augmentation: horizontal flip, random crop, adaptive time masking, patch masking.

**Metric**
- Word Error Rate (WER): 
  \[
  \mathrm{WER} = \frac{S + D + I}{N}\times 100\%
  \]
  with $S=$ substitutions, $D=$ deletions, $I=$ insertions, and $N=$ reference words.

Protocols may add babble or other noise at inference to gauge robustness, and all major studies report results using the official splits to ensure comparability.

## 2. Canonical and Advanced AV-ASR Architectures

### Conformer-Based and Hybrid Models

Early state-of-the-art systems leverage a dual-stream architecture:
- Separate audio and video encoders (ResNet-18, Conformer stacks), fused via MLPs or addition, followed by joint CTC/seq2seq or attention-based decoders [2211.02133][2303.14307][2109.09536].
- Innovations include streamable Conformer blocks (chunk-wise self-attention, causal convolution), triggered attention for streaming inference, and alignment regularization losses synchronizing encoder activations.

### Video Transformer and Efficient Encoders

Recent work demonstrates video transformers (ViT, tubelet-based) can match or surpass 3D-conv baselines for mouth ROI feature extraction [2201.10439][2109.09536].
- Fine-tuned on LRS3, ViT front-ends yield up to 15% relative gains in lip-reading WER versus VGG-style convolutional nets.

### Fusion Strategies and Modality Dropout

Fusion is achieved by concatenation, addition, or cross-modal attention; modality dropout randomly zeroes out either stream during training to ensure unimodal robustness [2302.06419][2201.02184][2312.09369][2309.17395].

Innovations such as gated linear units (GLU) and Flamingo blocks address stable fusion and enable decoders to weigh modalities dynamically under variable SNR [2601.18396][2303.12187].

## 3. Self-Supervised, Semi-Supervised, and Pseudo-Labeling Methods

The LRS3 benchmark catalyzed the development of self-supervised pretext tasks (masked cluster prediction, context regression) and pseudo-labeling strategies.

### AV-HuBERT and Successors

AV-HuBERT [2201.02184], the seminal approach, iteratively clusters fused features, masking multi-stream inputs and predicting phonetic units. With 1,759 h unlabeled AV data, lip-reading WER drops from 33.6% (prior SOTA, 31K h transcript) to 26.9% (433 h labeled plus self-training). For audio-only ASR, pre-training with AV clusters yields a SOTA WER of 1.3% (against 2.3% previous best).

### AV-data2vec and Joint Contextualization

AV-data2vec [2302.06419] builds a joint (audio-video) transformer encoder trained to regress contextualized targets from a momentum teacher, leveraging early fusion and multi-layer averaging. On LRS3, audio-visual fine-tuning (433 h) achieves 1.7% WER (Large, 1,759 h SSL), −35.7% relative improvement over audio-only, with modality dropout scheduling critical for parameter efficiency and transfer across tasks.

### Continuous Pseudo-Labeling Without External Teacher

AV-CPL [2309.17395] replaces static external pseudo-labels (as in Auto-AVSR [2303.14307]) with on-the-fly, EMA-teacher-generated targets, striking a trade-off between ASR and VSR performance via modality dropout hyperparameter $p_m'$. For VSR, WER drops by 21.7% absolutely (Base, p_m'=0.1).

### Large-Scale Pseudo-Labeling and Data Expansion

Auto-AVSR [2303.14307] demonstrates large-scale noisy pseudo-labels (from Conformer, HuBERT, Whisper teachers) substantially reduce WER, saturating audio-only improvement, but continuing to benefit VSR. Best reported AVSR WER on LRS3 is 0.9% using 3,448 h of training data (818 h manually labeled, 2,630 h pseudo-labeled).

## 4. Large Language Model (LLM)-Driven AV-ASR

Unified frameworks leveraging LLMs have emerged for AV-ASR thanks to high-dimensional token compression and elastic granularity.

### Token Compression and Multi-Granularity

Omni-AVSR [2511.07253] and MMS-LLaMA [2503.11315] employ audio and visual encoders (Whisper, AV-HuBERT) followed by matryoshka/early-fusion token compression, feeding a frozen LLaMA with LoRA adapters.

- Omni-AVSR achieves 1.0% WER (av-rates 4,2), outperforming baselines while using a single 58M param model across ASR, VSR, and AVSR.
- MMS-LLaMA compresses features to 3.5 multimodal tokens per second (versus 25 tps baseline), with negligible loss in accuracy (0.74% clean WER), and a 35.7% FLOPs reduction.

### Decoding and Losses

Decoding employs beam search on next-token cross-entropy or hybrid scoring (joint CTC/seq2seq), possibly integrating external 4-gram LMs [2511.07253][2309.17395].

Trade-offs between accuracy, computational cost, and token length are managed by dynamic query allocations based on estimated speech rate.

## 5. Benchmark Performance and Comparative Analysis

Table: SOTA LRS3 AV-ASR Results from Recent Studies

| System                            | Training Data (h) | AVSR WER (%) | Robustness Under Noise |
|------------------------------------|-------------------|--------------|-----------------------|
| MMS-LLaMA [2503.11315]             | 1,759            | 0.74         | Maintained SOTA, dynamic tokens |
| Auto-AVSR [2303.14307]             | 3,448            | 0.9          | SOTA with pseudo-labels |
| Omni-AVSR-ST [2511.07253]          | 433              | 1.0 (rates 4,2)      | 18% @ –5 dB SNR |
| AV-data2vec (Large, 1,759 h) [2302.06419] | 1,759      | 1.3          | – |
| AV-HuBERT (Large) [2201.02184]     | 1,759            | 1.3          | – |
| Dual-Use Whisper [2601.18396]      | 1,929            | 4.08*        | SOTA in noisy/MUSAN    |
| Streaming Conformer [2211.02133]   | 438              | 2.0 (offline)       | – |
| VALLR (Visual-only) [2503.21408]   | 30 (labeled)      | 18.7         | N/A                   |

*Under MUSAN babble, 0 dB SNR, using Whisper medium + AV-HuBERT visual.

**Key findings**
- SOTA WERs are now well below 1% (MMS-LLaMA, Auto-AVSR) given large-scale pseudo-labels and compressed LLM tokenization.
- Semi-supervised and SSL models (AV-data2vec, AV-HuBERT) approach SOTA at far lower annotation cost.
- Data scaling saturates audio-only gains, whereas VSR and AVSR continue to improve with pseudo-labeled augmentation.

## 6. Streaming and Practical ASR

Streaming AV-ASR architectures using chunk-wise and triggered attention enable real-time recognition, with delays as low as 1.06 s at the expense of a small WER degradation (2.0→2.6%) [2211.02133]. Alignment regularization critically synchronizes modal streams, reducing prediction offsets to <6 ms across SNRs.

LLM-based approaches now support elastic inference, balancing model size, token compression, and accuracy for variable compute and latency budgets [2511.07253].

## 7. Outstanding Limitations and Future Directions

Key limitations include residual viseme ambiguity in visual-only models (VALLR [2503.21408]), saturation of audio performance beyond ~1,500 h pseudo-labels, and sensitivity of SSL methods to hyperparameters (modality scheduler, EMA rate). At present, robust AVSR under severe acoustic noise (>–5 dB SNR) remains challenging, with best systems plateauing at WER ~18% [2511.07253][2601.18396].

Emerging directions:
- Speaker-adaptive and cross-lingual AVSR models (Mandarin CSTS [2303.12187]).
- Large-scale, truly end-to-end multimodal LLMs with contextual reasoning over compressed tokens.
- Unified parameter-efficient adaptation (e.g., LoRA, matryoshka) minimizing model multiplicity.
- Deeper investigation of interpretable intermediate structures (phonemes, articulatory features) for visual-only and AVSR pipelines [2503.21408].

The LRS3 AV-ASR benchmark remains the definitive standard for evaluating progress in multimodal speech recognition, combining rigor, scale, and diversity; its systematic use across architectures, training regimes, and scalability studies ensures that it continues to drive innovation and fair comparison in the field.

Source: https://www.emergentmind.com/topics/lrs3-av-asr-benchmark