Oxford-BBC Lip Reading (LRS2-BBC) Dataset
- LRS2-BBC is a public audio-visual dataset offering sentence-level lip reading from BBC broadcasts with diverse accents and real-world visual challenges.
- It employs an automatic curation pipeline using SSD-based face detection, forced subtitle alignment, and multi-view SyncNet for audio-video synchronization.
- Benchmark results using CTC and seq2seq models show significant word error rate improvements, establishing LRS2-BBC as a key resource in audio-visual speech research.
Searching arXiv for recent and foundational papers on LRS2-BBC. LRS2-BBC, also referred to as the Oxford-BBC Lip Reading Sentences 2 (LRS2) dataset, is a public audio-visual speech recognition and lip-reading corpus introduced by Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman for sentence-level recognition “in the wild,” using unconstrained natural-language utterances from BBC television rather than studio or grammar-constrained recordings (Afouras et al., 2018). It was designed to support transcription from a talking face with or without audio, to study the complementarity of visual mouth cues and speech acoustics, and to replace the earlier non-public LRS dataset with a publicly released benchmark (Afouras et al., 2018).
1. Origins and intended problem setting
LRS2-BBC was created to enable end-to-end audio-visual speech recognition and lip reading on unconstrained, natural-language sentences from diverse broadcast footage, with substantial variation in pose, expressions, lighting, backgrounds, ethnic origin, speakers, and accents (Afouras et al., 2018). The source material is BBC television programming spanning varied genres and production styles; examples explicitly listed include Dragon’s Den, Top Gear, and Countryfile (Afouras et al., 2018). The dataset targets sentence transcription rather than isolated-word classification, and no vocabulary restrictions are imposed during sentence extraction (Afouras et al., 2018).
The task definition distinguishes LRS2-BBC from earlier lip-reading benchmarks. GRID is described as 27.5 hours, 51 speakers, constrained grammar, and a 51-word vocabulary, while LRW is a 500-word closed-vocabulary word-level classification dataset; by contrast, LRS2-BBC is sentence-level, unrestricted in vocabulary, and “in the wild” (Afouras et al., 2018). The dataset also supersedes the original LRS corpus, which could not be made public because of license restrictions, and it precedes later public sentence-transcription resources such as LRS3-TED (Afouras et al., 2018).
2. Automatic curation and annotation pipeline
The released corpus is the result of a multi-stage automatic pipeline operating on BBC broadcasts (Afouras et al., 2018). Shot boundaries are detected using consecutive-frame color histogram comparison. Face detection is performed with an SSD-based CNN face detector, and face tracks within each shot are formed by associating detections by position rather than by feature trackers such as KLT, which were avoided because of failures with extreme viewpoint changes (Afouras et al., 2018).
Subtitle alignment is handled with the Penn Phonetics Lab Forced Aligner, and the aligned labels are filtered against IBM Watson Speech to Text because broadcast subtitles are not verbatim (Afouras et al., 2018). A multi-view adaptation of a two-stream SyncNet-like network is then used both to synchronize audio and video and to verify that the lip movements of a detected face track match the audio; clips are rejected as voice-overs if no face matches (Afouras et al., 2018). Sentences and phrases are extracted using punctuation in the transcripts, specifically full stops, commas, and question marks, and clips are capped at 100 characters or 10 seconds because of GPU memory constraints (Afouras et al., 2018).
This pipeline is central to the character of the dataset. It produces sentence-level audio, face-track video, and transcript pairs from unconstrained television footage, but it also introduces the same sources of uncertainty that later papers explicitly note: subtitle mismatch, residual A/V sync errors, and noisy pre-train annotations (Afouras et al., 2018).
3. Modalities, splits, and technical characteristics
The dataset provides face tracks of the speaking person, 16 kHz audio, and sentence-level transcripts derived from broadcast subtitles (Afouras et al., 2018). For model input in the dataset paper, video is sampled at 25 fps from 224×224 face crops, and a 112×112 mouth region of interest is used by the visual front-end; acoustic features are 321-dimensional spectral magnitudes computed with a 40 ms STFT window and 10 ms hop, with four audio frames corresponding to each 40 ms video frame (Afouras et al., 2018). The modeling setup removes punctuation and uses character outputs with 40 output tokens: 26 letters, 10 digits, plus [space] and [pad], with [sos] for seq2seq and [blank] for CTC (Afouras et al., 2018).
The core split structure is defined by broadcast date. The pre-train partition contains sentence excerpts and word-level alignment boundaries for every word, whereas Train-val and Test are sentence-level evaluation partitions (Afouras et al., 2018).
| Split | Statistics | Notes |
|---|---|---|
| Pre-train | 96k utterances; 2M word instances; 41k vocabulary; 195 hours | 01/2010–02/2016; sentence excerpts; word-level alignment boundaries |
| Train-val | 47k utterances; 337k word instances; 18k vocabulary; 29 hours | 01/2010–02/2016 |
| Test | 1,243 utterances; 6,663 word instances; 1,693 vocabulary; 0.5 hours | 03/2016–09/2016 |
| Text-only corpus | 8M subtitle lines; 26M word instances; 60k vocabulary | 01/2016–02/2016; LM training |
Several downstream papers describe the corpus through task-specific subsets. A speaker-extraction study reports clean-speech splits of 45,839 training utterances, 1,082 validation utterances, and 1,243 test utterances, with synchronized face-track video at 25 frames per second and audio at 16 kHz, because its experiments construct mixtures from the original materials (Pan et al., 2022). A viseme-to-word mapping study, working only on the test section, reports 1,243 sentences, 6,660 word instances, and a vocabulary of 1,697 unique words (Fenghour et al., 2020). These are not contradictory dataset definitions; they are task-specific descriptions of how LRS2-BBC is operationalized in later work.
4. Benchmark protocols and reported performance
The dataset paper recommends using Pre-train for representation learning and curriculum learning, Train-val for fine-tuning, and Test for final reporting, with the text-only subtitle corpus serving as external language-model training data (Afouras et al., 2018). The principal evaluation metric is word error rate,
where , , and denote substitutions, deletions, and insertions, and is the number of reference words (Afouras et al., 2018).
Afouras et al. report strong baseline results on the LRS2-BBC test split for both CTC and seq2seq Transformer systems. For TM-CTC, the reported clean-test WERs are 65.0% for video-only, 15.3% for audio-only, and 13.7% for audio-visual, improving to 54.7%, 10.1%, and 8.2% respectively with an external LLM; under noisy babble, the corresponding audio-only and audio-visual WERs are 64.7% and 33.5%, improving to 53.4% and 23.6% with the external LLM (Afouras et al., 2018). For TM-seq2seq, the clean-test WERs are 49.8% for video-only, 10.5% for audio-only, and 9.4% for audio-visual, with language-model-assisted results of 48.3%, 9.7%, and 8.5%; under noisy babble, the audio-only and audio-visual WERs are 58.0% and 35.9%, improving to 57.4% and 34.2% with the external LLM (Afouras et al., 2018).
Subsequent work used the same benchmark to compare alternative model classes. In visual-only continuous lip reading, the Transformer seq2seq model in “Deep Lip Reading: a comparison of models and an online application” reaches 50.0% WER on the LRS2 test set with the larger subtitle LLM, improving over the prior 70.4% WER state of the art cited in that paper (Afouras et al., 2018). In hybrid sequence-discriminative modeling, “Audio-visual Recognition of Overlapped speech for the LRS2 dataset” reports LF-MMI TDNN performance of 48.86% WER for video-only, 6.71% for audio-only, and 5.93% for audio-visual recognition on clean LRS2, which that paper describes as state-of-the-art on the dataset (Yu et al., 2020). A later language-model rescoring study for overlapped speech reports that, on its LRS2-BBC overlapped-recognition setup, a five-way “All GP NNLMs” combination obtains average WER 9.4 across clean, TF masking, Filter-and-Sum, and Mask-based MVDR conditions, with statistically significant absolute WER reductions of 1.3% over the LSTM baseline and 1.2% over the Transformer baseline (Xue et al., 2022).
5. Derived tasks built from LRS2-BBC
A defining feature of LRS2-BBC is that it has become a substrate for multiple derived tasks beyond standard sentence transcription. One line of work simulates two-speaker overlapped speech from LRS2. In “Audio-visual Recognition of Overlapped speech for the LRS2 dataset,” one reference audio-visual utterance is mixed with an interfering audio utterance; the longer source is truncated to the length of the shortest one, and six SNR conditions are used during training: clean, 15 dB, 10 dB, 5 dB, 0 dB, and −5 dB (Yu et al., 2020). The best integrated AV-driven gating system with concatenation reports average WER 10.31 over the 10, 5, 0, and −5 dB test conditions, versus 40.29 for the audio-only LF-MMI baseline, corresponding to the 29.98% absolute WER reduction stated in that paper (Yu et al., 2020).
A closely related derivative is LRS2-mix for audio-visual target speaker extraction. That corpus is built from the original LRS2 audio-visual materials and contains 200,000 training mixtures, 5,000 validation mixtures, and 3,000 test mixtures (Pan et al., 2022). Interference speech is truncated or zero-padded to the target utterance length, the target–interference SNR is uniformly set between 10 dB and −10 dB, and an optional noisy condition adds babble noise from WHAM! with SNR randomly set between 15 dB and −5 dB (Pan et al., 2022). In that setting, the dataset’s synchronized clean speech and face-track video support evaluation with SI-SDR, SDR, PESQ, STOI, WER, CER, and explicit over-suppression measures defined on STFT magnitudes (Pan et al., 2022).
LRS2-BBC is also used for symbolic viseme studies that do not process video pixels directly. In “Disentangling Homophemes in Lip Reading using Perplexity Analysis,” the authors operate on the test transcripts, convert words to phonemes with the Carnegie Mellon Pronouncing Dictionary, and then map phonemes to visemes using Lee and Yook’s inventory of 6 consonant visemes, 7 vowel visemes, and 1 silent viseme (Fenghour et al., 2020). On the 1,243-sentence test section, they report Scenario 1 results of CER 10.7%, WER 18.0%, and SAR 56.8% when word boundaries are known, and Scenario 2 results of CER 36.1%, WER 48.3%, and SAR 35.1% when boundaries are unknown (Fenghour et al., 2020). A separate study, “Lip reading using external viseme decoding,” proposes splitting video-to-character recognition into video-to-viseme and viseme-to-character stages and states a 4% word error rate improvement relative to a normal sequence-to-sequence lip-reading model on LRS2 (Peymanfard et al., 2021).
6. Limitations, caveats, and recurrent points of confusion
Several limitations are explicit in the dataset paper. Broadcast subtitles are not verbatim, so alignment errors can remain after forced alignment and filtering; audio and video can be out of sync by up to around one second; clips with no visible speaker are rejected as voice-overs, although borderline cases may persist; pre-train is noisier by design and includes sentence excerpts rather than only full sentences; sentences are clipped to 100 characters or 10 seconds; punctuation is removed in the modeling transcripts; and the paper does not specify file formats, bit depths, native resolutions, or the number of unique speakers (Afouras et al., 2018).
A recurring misconception is that all “LRS2-BBC” results are directly comparable. In practice, later papers often evaluate different problem formulations built on the same corpus. The viseme-perplexity study does not use a visual front-end at all; it derives viseme sequences from ground-truth transcripts and therefore evaluates only viseme-to-word mapping rather than full visual speech recognition from frames (Fenghour et al., 2020). The speaker-extraction study evaluates LRS2-mix, a simulated two-speaker mixture corpus derived from LRS2 rather than the clean benchmark itself (Pan et al., 2022). The Bayesian language-model study evaluates multi-channel cocktail-party overlapped speech with an average overlap ratio of 85%, trains word-level LLMs on 2.5 million words from Pre-train + Train-val, and rescores hypotheses from a 4-gram LLM using a 41K recognition lexicon (Xue et al., 2022). This suggests that “performance on LRS2-BBC” should always be interpreted together with the exact protocol: clean sentence transcription, video-only lip reading, overlapped speech recognition, target speaker extraction, or transcript-derived viseme decoding.
The dataset’s importance therefore lies not only in its original benchmark role, but also in the fact that its synchronized broadcast audio, face-track video, subtitle-derived transcripts, and standard test partition support a wide spectrum of audio-visual speech tasks under consistent source material (Afouras et al., 2018).
