SlideAVSR: Slide-Aware Recognition Benchmark

• The paper introduces SlideAVSR, a benchmark that integrates slide text from scientific presentations to improve transcription of technical terms.
• It details a robust collection and alignment process using OCR and segmentation, demonstrating improvements in word error rates up to 14.3%.
• DocWhisper, the baseline model, conditions Whisper’s transcription on slide-derived prompts, underscoring the value of multimodal context in AVSR.

SlideAVSR is an audio-visual speech recognition benchmark built from scientific paper explanation videos with slides, designed to evaluate transcription systems that can use not only audio but also textual information visible on presentation slides. It was introduced to test a form of AVSR that prior benchmarks largely underemphasized: recognition in which the visual stream is a text-bearing document source rather than primarily a speaker’s face or lips. The benchmark targets scientific presentation speech, where technical terminologies are notoriously challenging to transcribe without reference texts, and it is accompanied by a baseline model, DocWhisper, that refers to textual information from slides during transcription (Wang et al., 2024).

1. Scope and conceptual framing

SlideAVSR was proposed against the background of AVSR research that has been shaped mainly by lip-reading-oriented datasets. In those settings, the visual stream mainly consists of facial features, especially the mouth region, and the AVSR problem is effectively “speech + lips.” SlideAVSR instead centers scientific presentation videos in which the visual evidence is dominated by screen-shared slides containing technical terms, names, acronyms, and other domain-specific vocabulary that are hard to recover from audio alone (Wang et al., 2024).

The benchmark therefore reframes the role of the visual modality. In SlideAVSR, the useful visual information is not primarily articulatory motion. It is document-like context, especially slide text. This makes the task a form of multimodal speech transcription in which the model must transcribe utterances with texts on the slides on the presentation recordings. The emphasis on scientific paper explanation videos also makes the benchmark lexically distinctive: the target errors are often substitutions on technical terms, names, and uncommon expressions rather than only degradation under acoustic noise (Wang et al., 2024).

A recurrent misconception is to treat SlideAVSR as a conventional lip-reading benchmark with an additional screen context. The benchmark was explicitly introduced to evaluate “image comprehension capabilities” beyond facial features. Its main challenge is therefore not only temporal audiovisual synchronization around speech articulation, but also whether an AVSR system can use visible slide text as transcription support (Wang et al., 2024).

2. Data collection, filtering, and alignment

The dataset was constructed from YouTube scientific presentation videos by following a pipeline based on JTubeSpeech, with additional filtering and cleansing specialized for paper-explanation videos with slides. Search queries use the form "{Conference} {Year} {Form}", where Year ranges from 2020 to 2023 and Form includes "paper", "workshop", and "talk". The targeted conference list includes venues from NLP, CV, speech, AI, ML, data mining, database, IR, and HCI, such as ACL, EMNLP, CVPR, INTERSPEECH, ICASSP, ICLR, ICML, NeurIPS, KDD, SIGIR, and CHI (Wang et al., 2024).

Initial collection retains videos with manual subtitles, duration between 5 and 20 minutes, MP4/720P/H264 video, and single-channel 16 bit 16 kHz audio. Filtering then proceeds in several stages. A ChatGPT filter based on gpt-3.5-turbo uses the YouTube description to decide whether the video is a presentation video of a research paper and whether the description is written in English. A BLIP-2 filter using blip2-flan-t5-xl inspects screenshots from the beginning, end, and three quartile points to determine whether the video actually shows research presentation slides. A further manual filter removes videos that automatic filters miss, including conference openings and videos that rarely show slides (Wang et al., 2024).

Because subtitle timestamps are inaccurate, utterance-level alignment is refined with CTC segmentation using kamo-naoyuki_wsj from ESPnet. Utterances with segmentation score below -7 are excluded. At the utterance level, a screenshot at the midpoint of each utterance is again checked by BLIP-2 so that retained utterances correspond to moments when slides are visible. Adjacent utterances are merged if the previous end time equals the next start time and their total duration is ≤ 15 seconds. The paper reports that this merging substantially improved Whisper ASR performance by approximately 20%, making segmentation quality a central part of the benchmark definition (Wang et al., 2024).

The resulting transcripts are not newly transcribed from scratch. They are derived from manual YouTube subtitles that are cleaned and time-aligned. This is important for interpreting the benchmark: SlideAVSR is simultaneously a dataset contribution and a curation/alignment contribution.

3. Corpus structure and partitioning

The final SlideAVSR dataset totals about 36 hours from 245 videos, with 220 speakers and 19,463 utterances. The split design is speaker-disjoint, and the test set is divided into a standard held-out split and a South Asian English subset intended to probe accent difficulty (Wang et al., 2024).

Split	Videos / Speakers	Utterances / Hours
Train	195 / 172	15,803 / 29.26
Dev	20 / 20	1,515 / 3.08
TestA	15 / 15	1,034 / 2.21
TestB	15 / 13	1,111 / 1.90
Total	245 / 220	19,463 / 36.45

Partitioning is accent-aware in a limited sense. Native English speakers were asked to classify speakers’ accents, and TestB was reserved as a smaller but important subset for South Asian English accents. The same speaker does not appear in multiple splits. The paper also notes a limitation: accents could not be labeled precisely for all videos, so only South Asian English videos were specifically identified while the rest remained unlabeled (Wang et al., 2024).

The corpus is lexically oriented toward technical presentation speech, but the paper does not provide a formal vocabulary size. Instead, it characterizes the dataset through the kinds of corrected errors observed later in evaluation: technical term, inflection, mishearing, and name. Another important structural property is that OCR output per sample has a long-tail distribution. The paper states that only 70% of samples can be covered even if 100 words are included in the prompt, and that some samples contain over 500 words in OCR results. Since Whisper typically allows a maximum prompt length of 224 tokens, slide text selection becomes a modeling bottleneck rather than a peripheral preprocessing choice (Wang et al., 2024).

4. Task formulation and DocWhisper baseline

The benchmark task is multimodal speech transcription for scientific presentation segments. In the audio-only setting, the input is utterance audio. In the slide-aware setting used by the baseline, the system takes utterance audio plus OCR text extracted from a screenshot of the slide shown during the utterance. The output is the transcribed speech utterance text, and the evaluation metric is word error rate, with Whisper text normalization used for evaluation (Wang et al., 2024).

DocWhisper is the proposed baseline AVSR model. Its pipeline is deliberately simple. It takes the utterance audio, captures a screenshot at the midpoint of the utterance, applies OCR using Google Cloud Vision API, converts the recognized text into a prompt word sequence such as "word 1, word 2, ..., word n", and then fine-tunes Whisper large-v3 to decode the audio under that prompt. The fusion mechanism is therefore prompt-based conditioning rather than a learned visual encoder or hidden-state cross-attention module (Wang et al., 2024).

This design choice is central to the meaning of SlideAVSR. The benchmark is nominally AVSR, but the baseline does not ingest raw video frames into a trainable visual backbone. Video is operationalized as a document-image source, and OCR is the mechanism that turns the visual stream into model-usable context. A second misconception follows from this: DocWhisper is not a raw-video multimodal transformer. It is a fine-tuned Whisper model conditioned on OCR prompts derived from slide screenshots (Wang et al., 2024).

To handle prompt budget limits, the paper studies prompt lengths $K \in \{25, 50, 75, 100\}$. It also proposes FQ Ranker, which computes word ranks using word frequency counts from English Wikipedia (April 2023) and sorts OCR words in ascending order of frequency so that rarer words appear earlier. The intended intuition is that rare words are often technical terms and therefore especially informative. The paper’s later findings, however, show that this heuristic is only partially effective, because corrected errors are not limited to extremely rare words (Wang et al., 2024).

Training uses AdamW with learning rate 2e-5, linear warmup of 1,000 steps, batch size 16, 10 epochs, and a single NVIDIA A100 (40G). Results are averaged over 3 random seeds, and the best checkpoint is selected by Dev performance (Wang et al., 2024).

5. Empirical behavior and error patterns

The audio-only Whisper baselines establish that straightforward fine-tuning is not sufficient to solve the benchmark. Whisper without fine-tuning yields WER 8.23 on TestA and 11.18 on TestB. Fine-tuned Whisper yields 8.07 on TestA and 11.25 on TestB. The paper summarizes this as a 1.9% improvement on TestA and no notable improvement on TestB, indicating that the South Asian English subset remains harder even after supervised adaptation (Wang et al., 2024).

DocWhisper improves systematically as more OCR words are included. The best reported results are obtained at K=100, with WER 6.91 on TestA and 10.01 on TestB. Relative to fine-tuned Whisper, the paper reports maximum improvements of 14.3% on TestA and 11% on TestB (Wang et al., 2024).

Model	TestA	TestB
Whisper, fine-tuned, K=0	8.07	11.25
DocWhisper, K=25	7.35	10.82
DocWhisper, K=50	7.08	10.43
DocWhisper, K=75	7.02	10.04
DocWhisper, K=100	6.91	10.01

FQ Ranker yields mixed effects. At small prompt budgets it can help, especially on TestB, but its advantage disappears or reverses when K > 50. The paper suggests two reasons. First, DocWhisper corrections are not only technical terms; many involve common words and inflections. Second, sorting by frequency destroys the original contextual or order information, which may reduce prompt utility for Whisper’s decoder (Wang et al., 2024).

The error analysis of 100 cases where Whisper made a substitution error and DocWhisper corrected it is especially informative. Corrected substitutions are classified as technical term: 41%, inflection: 28%, mishearing: 24%, and name: 7%. This distribution shows that visible slide text helps with technical terminology, but not only with technical terminology. It also supports the paper’s criticism of purely rare-word prioritization: prompt utility extends to more ordinary lexical corrections as well (Wang et al., 2024).

6. Relation to slide-grounded AVSR and outstanding limitations

SlideAVSR belongs to a broader shift from mouth-centered AVSR toward slide-grounded and document-grounded transcription. Closely related work on SlideSpeech releases a large-scale slide-enriched audio-visual corpus and shows that current-slide text can improve ASR through OCR, keyword extraction, and contextual biasing, though with a benchmark that is better understood as slide-text-enhanced ASR than as a full slide-aware AVSR architecture (Wang et al., 2023). LCB-net extends this direction by treating slide-derived context as long-context biasing, modeling the context at the token or BPE level and using a bi-encoder architecture with cross-attention and a biasing prediction module (Yu et al., 2024). Later work on SlideASR introduces an end-to-end image-conditioned formulation in which a synchronized slide image and speech are consumed jointly by an omni-modal LLM, with Visually-Anchored Policy Optimization enforcing a structured “Look before Transcription” procedure (Hu et al., 8 Oct 2025). Another line of work on conference talks explores both cascaded slide-word prompting and direct image conditioning, reporting approximately 34% relative WER reduction across all words and 35% for domain-specific terms compared to the baseline model (Sinhamahapatra et al., 15 Oct 2025).

Within this literature, SlideAVSR’s distinctive contribution is benchmark definition rather than model complexity. It provides a dataset of paper explanation videos with speaker-disjoint splits, a South Asian English evaluation subset, and a baseline that makes slide usage measurable in a controlled way. A plausible implication is that SlideAVSR functions as an intermediate benchmark between lip-reading AVSR and later slide-grounded end-to-end systems: it is more document-aware than traditional AVSR, but its baseline remains OCR-dependent and prompt-based rather than fully multimodal.

The paper is explicit about several limitations. Compared with major AVSR datasets, SlideAVSR has a relatively small number of speakers: 220 total. The domain is narrow, focusing on scientific paper explanation videos, especially AI-related conferences. Accent labeling was only partially successful. DocWhisper depends on OCR quality, and OCR outputs are long-tailed relative to Whisper’s prompt budget. Temporal grounding is also simplified, because DocWhisper uses a single midpoint screenshot per utterance. The paper therefore identifies future work in two directions: continued improvement of OCR-centered methods and an end-to-end AVSR model that is not dependent on OCR (Wang et al., 2024).

Taken together, these properties make SlideAVSR a benchmark for a specific but important AVSR regime: scientific presentation transcription in which visible slide text is a primary visual cue. Its central contribution is to formalize that regime as a dataset and evaluation problem, and to demonstrate that even a simple OCR-prompted baseline can materially improve recognition on speech that would otherwise remain difficult for audio-only ASR (Wang et al., 2024).