WildVSR: Robust Visual Speech Recognition Benchmark

• WildVSR is a visual speech recognition benchmark that evaluates models’ ability to generalize beyond the LRS3 test set using diverse, real-world YouTube videos.
• It is constructed using LRS3 methodologies—employing face detection, active speaker filtering, and forced alignment—to ensure high-quality, unconstrained lip sequence data.
• WildVSR reveals approximately a 30-point increase in word error rates over LRS3, underlining challenges in cross-domain robustness and visual encoder performance.

WildVSR is a visual-speech-recognition test benchmark designed to measure whether lipreading systems generalize beyond the highly reused LRS3 test set. It was introduced in “Do VSR Models Generalize Beyond LRS3?” and later described as an open-domain test set constructed with the same processing pipeline as LRS3 but deliberately drawn from unconstrained YouTube videos in order to stress-test cross-domain robustness under greater speaker diversity, lighting variation, camera variation, background clutter, compression artifacts, accents, and noise conditions (Djilali et al., 2023). In the WildVSR literature, the acronym refers to visual speech recognition rather than video super-resolution.

1. Origin, scope, and intended role

WildVSR emerged from a specific methodological concern: the LRS3 benchmark had become the primary focus of VSR research, while its test split is only about $0.9$ hours in duration. The benchmark was therefore introduced to alleviate the risk of overfitting to an excessively used test set and to expose current models to “slightly harder and in the wild lip sequences than those found in the LRS3 test set” (Djilali et al., 2023).

The benchmark is test-only. Later work consistently describes WildVSR as a single released test partition with no official training or validation splits, so models are trained elsewhere—typically on LRS3, LRS2, and optionally larger labeled or unlabeled corpora—and then evaluated on WildVSR (Haliassos et al., 2024). Its role is thus not to provide a full training ecosystem, but to function as a stress test for out-of-domain lipreading.

A recurrent theme across subsequent papers is that WildVSR probes cross-domain robustness rather than in-domain optimization. The benchmark is used to ask whether performance gains obtained on LRS2/LRS3 transfer to uncontrolled web video, and whether improvements come from stronger visual modeling, better decoding, additional data diversity, or semi-supervised learning (Jain et al., 18 Sep 2025).

2. Dataset construction and composition

The construction procedure closely follows LRS3 methodology. The source is English Creative-Commons YouTube video, queried using 21 thematic keywords such as “interview”, “news”, “health”, and “technology”, while filtering for relevance and non-overlap with LRS3 video IDs. Processing includes shot-boundary detection with 3D-histogram scene cuts, face detection and tracking with YOLOv5n0.5-Face plus frame-to-frame association, active-speaker filtering with SyncNet, and Whisper ASR for discarding non-English tracks and obtaining word-level timestamps. Utterances are segmented into clips of $0.5$–$16$ seconds aligned to Whisper word timestamps, followed by manual verification for lip visibility and transcript accuracy (Djilali et al., 2023).

The benchmark-introduction summary reports the following statistics for the released test set (Djilali et al., 2023):

Statistic	WildVSR
Videos	478
Speakers	618
Utterances	2,854
Words	45,182
Vocabulary	6,040
Hours	4.8
Utterance duration	0.5–16 s
Median utterance duration	≈3 s

Later work also characterizes WildVSR as “approximately 5 hours of video” and again emphasizes that only a test partition is provided (Haliassos et al., 2024). Another later description states that the original WildVSR paper does not report exact counts of clips or total hours (Jain et al., 18 Sep 2025). The coexistence of these descriptions indicates a minor reporting inconsistency in later summaries, whereas the benchmark-introduction overview provides explicit counts.

WildVSR is explicitly more diverse than LRS3 in both content and demographics. The benchmark-introduction summary reports speaker metadata estimated with DeepFace plus manual checking: $52\%$ male and $48\%$ female; $10\%$ young, $77\%$ adult, and $13\%$ old; $68\%$ White, $18\%$ Black, and $0.5$0 Other; and $0.5$1 native versus $0.5$2 non-native accents (Djilali et al., 2023). Later work describes the dataset more generally as spanning a wide age range, multiple nationalities and accents, and both male and female speakers, while also noting that the exact breakdown is not specified there (Jain et al., 18 Sep 2025).

WildVSR also inherits important lexical properties from the LRS ecosystem. One later description states that it inherits the LRS3 vocabulary—approximately $0.5$3 k distinct word types in the $0.5$4 h training set—and its English-only lexicon (Jain et al., 18 Sep 2025). This should be distinguished from the WildVSR test-set vocabulary count of $0.5$5 reported in the benchmark-introduction summary (Djilali et al., 2023).

3. Preprocessing, alignment, and evaluation protocol

WildVSR is not tied to a single model-specific preprocessing stack; rather, multiple papers evaluate on the same benchmark using pipelines aligned with their own architectures. In the benchmark-introduction study, preprocessing is described as identical cropping, normalization, and frame-rate as per the LRS3 pipeline: a $0.5$6 mouth ROI is upsampled to $0.5$7 at $0.5$8 fps (Djilali et al., 2023). In the unified speech recognition work, original videos are downsampled to $0.5$9 frames per second, each frame is pre-cropped to a $16$0 mouth ROI, converted to grayscale, and augmented during training with a random spatial crop to $16$1 plus horizontal flip with probability $16$2 applied consistently in time (Haliassos et al., 2024). In the LLM-decoder study, videos are temporally resampled to $16$3 frames per second, face and mouth detection is performed via RetinaFace, mouth patches are cropped to $16$4 grayscale, training uses random spatial crops and horizontal flips, and testing uses fixed center cropping (Jain et al., 18 Sep 2025). This suggests that WildVSR functions as a shared evaluation corpus across several established VSR preprocessing conventions.

Alignment and tokenization are likewise architecture-dependent. One evaluation protocol states that video frames and transcript word-level alignments are inherited from the LRS3 pipeline, with each token aligned to a short clip of frames via forced alignment, and transcripts tokenized using the target decoder’s subword vocabulary, such as Llama-2 byte-pair encoding (Jain et al., 18 Sep 2025). Another uses SentencePiece subword units with vocabulary size $16$5 (Haliassos et al., 2024).

WildVSR is fundamentally a visual-only benchmark. Later work explicitly states that no ASR or AVSR scores are reported on WildVSR because the benchmark is VSR-only, even when a broader framework supports auditory or audiovisual inputs (Haliassos et al., 2024). Similarly, the LLM-decoder study notes that no audio modality is used and only visual frames feed the AV-HuBERT encoder (Jain et al., 18 Sep 2025).

The common evaluation metric is Word Error Rate. In one formulation,

$16$6

where $16$7 is the number of substitutions, $16$8 deletions, $16$9 insertions, and $52\%$0 the total number of reference words (Haliassos et al., 2024). The benchmark-introduction study also defines per-utterance WER and weighted aggregate statistics:

$52\%$1

$52\%$2

$52\%$3

and the consistency-aware ranking

$52\%$4

This ranking penalizes models with high variance across samples and was proposed as a complement to mean WER (Djilali et al., 2023).

4. Benchmark results and comparative performance

Across the published benchmark, WildVSR consistently produces substantially worse WER than LRS3 for the same models. The benchmark-introduction study reports a linear fit

$52\%$5

and an average degradation of approximately $52\%$6 WER points across VSR models, which indicates a systematic difficulty increase rather than isolated failure cases (Djilali et al., 2023).

Representative WildVSR results reported across later studies include the following:

Model	Training data	WildVSR WER
AV-HuBERT	433 h labels + 1 326 h unlabels	51.7%
Auto-AVSR	3 448 h labels	38.6%
USR	433 h + 1 326 h	46.4%
AV-HuBERT S2S	LRS2+LRS3 (657 h)	47.9%
VSP-LLM (13B, frozen encoder)	LRS2+LRS3 (657 h)	47.0%

These numbers illustrate several patterns. First, supervised or self-supervised baselines trained only on LRS3 often remain above $52\%$7 WER on WildVSR, including AV-HuBERT at $52\%$8 (Haliassos et al., 2024). Second, substantially larger labeled corpora improve results: Auto-AVSR reaches $52\%$9 when trained on $48\%$0 h (Djilali et al., 2023). Third, unified training can be competitive in the wild setting: USR, a single model spanning auditory, visual, and audiovisual inputs, reports $48\%$1 WER with shared parameters and $48\%$2 h labels plus $48\%$3 h unlabeled data (Haliassos et al., 2024).

The LLM-decoder study sharpens the comparison between conventional sequence-to-sequence decoding and large decoder back ends. With LRS2-only finetuning, AV-HuBERT S2S reports $48\%$4 WER, while VSP-LLM reports $48\%$5 for an 8B frozen encoder and $48\%$6 for a 13B frozen encoder. With LRS3-only finetuning, AV-HuBERT reports $48\%$7, VSP-LLM 8B $48\%$8, and VSP-LLM 13B $48\%$9. With combined LRS2+LRS3 finetuning, AV-HuBERT S2S reaches $10\%$0, VSP-LLM 1B $10\%$1, and VSP-LLM 13B $10\%$2 (Jain et al., 18 Sep 2025). The same study states that $10\%$3 is a new state of the art for systems without additional self-training or pseudo-labeling.

Quantization and adaptation effects appear limited relative to data diversity. For a 1B VSP-LLM trained on LRS3 with finetuned encoder, QLoRA at 4-bit yields $10\%$4, LoRA at 16-bit yields $10\%$5, and full-precision without LoRA yields $10\%$6 on WildVSR (Jain et al., 18 Sep 2025). The stated interpretation is that numerical precision is less critical than the amount and diversity of training data.

5. Error structure and failure modes

The dominant interpretation in the benchmark-introduction study is that WildVSR exposes a distribution gap rather than merely adaptive overfitting to LRS3. The consistent model ranking across LRS3 and WildVSR, together with the linear relationship between their WERs, is presented as evidence that the primary issue is the harder “in-the-wild” content distribution (Djilali et al., 2023).

Several specific difficulty factors are identified. WildVSR contains more challenging camera angles, lighting changes, background clutter, speaker variation, occlusion, and compression artifacts (Haliassos et al., 2024). Extreme head pose is especially salient: using 3D face fitting, the benchmark-introduction study estimates that $10\%$7 of the “easy” fold and $10\%$8 of the “hard” fold exhibit extreme poses beyond $10\%$9 yaw (Djilali et al., 2023). Short utterances are another major source of error: WER increases sharply for clips shorter than $77\%$0 seconds, which reduces temporal context for lip disambiguation (Djilali et al., 2023).

Lexical and accent variation also contribute materially. In the bottom-$77\%$1 fold, the benchmark-introduction study associates high WER with less common vocabulary and non-native accents (Djilali et al., 2023). Attribute-specific results reinforce this pattern: for Auto-AVSR, WER is $77\%$2 on native accents and $77\%$3 on non-native accents; for AV-HuBERT, $77\%$4 versus $77\%$5; and for RAVEn, $77\%$6 versus $77\%$7 (Djilali et al., 2023). Gender effects are smaller in the same table.

Representation analysis points in the same direction. Tucker decomposition of encoder features reportedly shows clear dominant spatial modes over time on LRS3, but flatter, less salient modes on WildVSR, consistent with a mismatch between learned representations and the harder WildVSR manifold (Djilali et al., 2023).

A distinct but related misconception is that stronger decoders alone can solve WildVSR. The LLM-decoder study argues otherwise: it identifies a visual-encoder bottleneck, stating that the overwhelming determinant of cross-domain performance is the quality of the visual front end. Even with a 13B LLM decoder, freezing AV-HuBERT yields WER above $77\%$8 unless the decoder is exposed to a more diverse training set (Jain et al., 18 Sep 2025). The same study further argues that decoder gains are primarily lexical, improving contextual disambiguation of visually similar units such as homophenes rather than providing deeper semantic reasoning. Full semantic metrics were not computed on WildVSR, but the authors infer from parallel analyses on LRS2/LRS3 that WildVSR improvements are likewise confined mainly to lexical corrections rather than substantial semantic paraphrasing or world-knowledge inference (Jain et al., 18 Sep 2025).

6. Significance for VSR research and open directions

WildVSR has become a benchmark for robustness rather than peak in-domain accuracy. Later work explicitly describes it as a stress test for future VSR systems and an instrument for measuring how far current approaches remain from reliable real-world lipreading (Haliassos et al., 2024). Its importance lies in separating progress that is genuinely transferable from progress that is largely benchmark-specific.

Several methodological implications recur across the literature. One is data diversity: models trained on combined LRS2+LRS3 outperform those trained on either set alone on WildVSR, and larger labeled corpora further improve WER (Jain et al., 18 Sep 2025). Another is semi-supervised leverage: AV-HuBERT with offline self-training, RAVEn with self-training, and USR all recover some robustness under wild conditions, although performance remains well above corresponding LRS3 levels (Haliassos et al., 2024). A third is architectural pressure on the visual front end: later work argues that meaningful progress will require stronger visual encoders, including large-scale self-supervised ViTs trained on tens of thousands of hours of unlabeled video, rather than decoder scaling alone (Jain et al., 18 Sep 2025).

The benchmark-introduction study also proposes concrete research directions: incorporating more in-the-wild sources during training, integrating explicit 3D head-pose normalization or pose-aware augmentations, improving architectures for very short clips, and adopting $77\%$9 to benchmark consistency across samples (Djilali et al., 2023). Taken together, these recommendations position WildVSR as a diagnostic benchmark for the next generation of robust visual speech recognition systems.