SlideASR: Integrating Slide Context in ASR

• SlideASR is a multimodal speech recognition task that integrates slide visuals with audio to accurately transcribe technical lectures.
• It addresses challenges in recognizing domain-specific terminology by using slide text as a reliable lexical cue in benchmarks like SlideSpeech and SlideAVSR.
• Recent methods employ OCR and policy optimization to prune noisy slide context, thereby reducing word error rates and boosting overall transcription performance.

Searching arXiv for papers on SlideASR and closely related slide-assisted ASR benchmarks/methods. SlideASR denotes a multimodal speech recognition task in which speech is transcribed with the help of synchronized presentation slides, rather than from audio alone. In its formal statement, the task is to transcribe speech $A$ with the help of a synchronized slide image $I$, maximizing $P(Y \mid A, I; \theta)$. The setting is motivated by lecture-like and conference-like speech, where slides often contain the exact domain-specific terminology, named entities, acronyms, or jargon that audio-only ASR systems misrecognize. The literature consistently treats slide content as a high-value auxiliary modality because it provides lexical grounding for rare and acoustically ambiguous terms (Hu et al., 8 Oct 2025, Wang et al., 2023, Wang et al., 2024).

1. Problem setting and conceptual scope

SlideASR emerged from a gap between conventional ASR and the realities of scientific presentations, webinars, online lectures, and paper explanation videos. In these settings, the central difficulty is often not generic acoustic robustness, but recognition of specialized terminology: paper titles, method names, acronyms, proper nouns, and field-specific vocabulary. The slide frequently contains these expressions explicitly, which makes it a natural contextual signal for transcription (Wang et al., 2023, Sinhamahapatra et al., 15 Oct 2025).

This distinguishes SlideASR from both ordinary ASR and classical audio-visual speech recognition. Ordinary ASR listens only to audio. Standard AVSR benchmarks are usually dominated by facial features such as lip-reading, whereas SlideASR shifts the auxiliary modality toward slide text and slide imagery. SlideAVSR makes this contrast explicit by defining a benchmark in which “the video shows presentation slides, and the model must leverage text on those slides to correctly transcribe speech, especially technical terms, names, and other out-of-vocabulary or rare words” (Wang et al., 2024).

The field also distinguishes SlideASR from generic contextual ASR. A central claim of the end-to-end literature is that SlideASR is not merely “ASR with extra context,” but a setting driven by entity-rich, domain-specific terms in academic lectures and reports. This suggests that the decisive question is not only whether contextual information is available, but whether the model can use slide evidence without ignoring audio or collapsing into OCR-only behavior (Hu et al., 8 Oct 2025).

2. Benchmarks and corpora

The empirical study of SlideASR has been organized around several benchmarks that differ in scale, modality, and emphasis.

Resource	Scale	Distinctive property
SlideSpeech	1,705 videos; 1,000+ total hours; 473 hours of high-quality transcribed speech	real-time synchronized slides
SlideAVSR	245 videos; 36.45 h; 19,463 utterances	paper explanation videos with slide text as AVSR signal
SlideASR-Bench	SlideASR-S: 6,413 train / 2,054 test; SlideASR-R: 60 real-world clips	entity-rich synthetic + real-world evaluation

SlideSpeech established the large-scale corpus setting for slide-enriched ASR. It contains 1,705 videos, more than 1,000 total hours, and 473 hours of high-quality transcribed speech with confidence above 95%. Its construction emphasizes real-time synchronized slides, and a manual check reported that 94% of dev/test segments contain slides. The paper frames the benchmark around a “what you see is what you get” paradigm, in which the slide text visible at a moment is expected to help recognize the speech being spoken at that moment (Wang et al., 2023).

SlideAVSR reframed the problem as a broader AVSR benchmark based on scientific paper explanation videos. Its final release contains about 36 hours from 245 videos, with Train, Dev, TestA, and TestB splits; TestB is reserved for speakers with South Asian English accents. The corpus was designed to test whether models can use non-facial visual information, especially textual content on slides, rather than lip motion alone (Wang et al., 2024).

SlideASR-Bench was introduced to address a different limitation: earlier public slide datasets did not contain enough specialized entities to stress-test domain terminology transcription. It consists of SlideASR-S, a synthetic dataset with 6,413 training samples, 44,240 entities, and 67.3 hours, plus a 2,054-sample test set with 13,895 entities and 18.5 hours; and SlideASR-R, a real-world set of 60 manually collected presentation clips from chemistry, medicine, biology, and artificial intelligence, with 200 annotated entities total (Hu et al., 8 Oct 2025).

A parallel evaluation line appears in “Do Slides Help? Multi-modal Context for Automatic Transcription of Conference Talks,” which extends the ACL 60/60 dataset and defines domain-specific words as words that do not appear in a general-domain corpus. This benchmark directly measures whether slide context improves transcription of special terminology rather than only overall WER (Sinhamahapatra et al., 15 Oct 2025).

3. Pipeline and prompt-based formulations

The earliest SlideASR systems are predominantly pipeline methods. In SlideSpeech, the benchmark system extracts the middle frame of each speech segment, applies Text Detection using TDNet and OCR using OCRNet through MMOCR, reformats recognized words by coordinates, and then applies KeyBERT to obtain keywords. These keywords feed a contextual ASR model based on Contextualized CTC/AED with Contextual Phrase Prediction. On the test set, the strongest SlideSpeech configuration is the L95 model with keyword-based contextual biasing, which reaches 12.38 WER versus a 12.89 baseline; the paper also shows that slide keywords help most on words that the audio-only baseline misses (Wang et al., 2023).

DocWhisper, introduced with SlideAVSR, takes a similarly explicit OCR-to-prompt route. For each utterance it captures a screenshot at the midpoint, runs OCR using Google Cloud Vision API, and passes the recognized text to Whisper as a prompt. With fine-tuning and OCR prompts, DocWhisper improves substantially over audio-only Whisper: at $K=100$, it reports 6.91 WER on TestA and 10.01 WER on TestB. The authors’ error analysis attributes 41% of corrected substitutions to technical terms, 28% to inflection, 24% to mishearing, and 7% to names (Wang et al., 2024).

MaLa-ASR recasts the same intuition in an LLM-based architecture. It uses a frozen WavLM-Large speech encoder, a trainable linear projector, and a frozen Vicuna 7B decoder, while inserting slide-derived keywords directly into the prompt. The abstract reports average WERs of 9.4% and 11.7% on the L95 and S95 subsets of the SlideSpeech corpus, with a relative WER drop of 27.9% and 44.7% over the baseline model reported in SlideSpeech. It further states that adding keywords reduces B-WER relatively by 46.0% and 44.2%, establishing a new SOTA on that dataset (Yang et al., 2024).

A recurrent finding across these systems is that slide assistance is strongest on “biased” words: terms that are present on the slide and relevant to the current utterance. This suggests that pipeline methods are most effective when slide text can be converted into compact lexical priors rather than treated as unrestricted context.

4. Long-context pruning and multimodal context integration

As the field matured, a major systems problem became apparent: OCR text from multiple slides is long, noisy, and only sparsely relevant to the current utterance. The SAP$^{2}$ framework, also described in the paper as 2-TPI, addresses this by first pruning long slide context down to speech-relevant keywords and then integrating only the compacted context into the ASR model. Its core mechanism is Speech-Driven Attention-based Pooling, which uses the speech signal itself to guide context compression (Rong et al., 14 Nov 2025).

The practical motivation is explicit. On SlideSpeech, context length grows from roughly 65 tokens for one slide to over 1500 tokens for 25 slides, while the information rate falls below 1%. The paper argues that naive prompt concatenation therefore suffers from constrained model context windows and contextual noise. Empirically, 2-TPI achieves 7.71% WER on SlideSpeech and reduces B-WER by 41.1% relative to a non-contextual baseline, while also reporting 1.12% WER on LibriSpeech (Rong et al., 14 Nov 2025).

“Do Slides Help?” broadens the context question from OCR keywords to multimodal slide evidence. It studies both cascaded text-prompt approaches and end-to-end image conditioning, mitigates limited training data with augmentation built from MuST-C, and reports a relative reduction in word error rate of approximately 34%, across all words and 35%, for domain-specific terms compared to the baseline model. The paper also reports $p < 0.001$ in a matched-pair test for error counts, indicating that the improvements from slide context are statistically significant (Sinhamahapatra et al., 15 Oct 2025).

A plausible implication is that SlideASR is increasingly bifurcating into two engineering regimes: compact text-biasing methods, which remain attractive for efficiency and controllability, and richer multimodal systems, which preserve layout, figures, and slide imagery when text alone is insufficient.

5. End-to-end SlideASR and visually anchored reasoning

The most explicit attempt to define SlideASR as an end-to-end task appears in “Look before Transcription: End-to-End SlideASR with Visually-Anchored Policy Optimization.” That paper argues that prior pipeline methods are complex and underperform, while omni-modal LLMs often fail in two opposite ways: they either ignore the slide and behave like ordinary ASR, or overcorrect and degenerate into OCR-only behavior (Hu et al., 8 Oct 2025).

To address this failure mode, the paper proposes Visually-Anchored Policy Optimization, or VAPO. VAPO enforces a structured “Look before Transcription” procedure with a rigid > <answer> format. In <think>, the model performs OCR on the slide content; in <answer>, it generates the transcription while referencing entities recognized in the reasoning stage. Training uses GRPO with four equally weighted rewards: Format Reward, OCR Reward, ASR Reward, and Visual Anchoring Reward (Hu et al., 8 Oct 2025).

The reported results are strongest on entity-centric evaluation. On SlideSpeech, VAPO-7B achieves WER 10.31, B-WER 10.84, U-WER 2.87, and Recall 97.32. On the real-world SlideASR-R set, VAPO-7B reports NE-WER 26.48 and NE-FNR 15.35, substantially improving named-entity recognition relative to the cited baselines (Hu et al., 8 Oct 2025).

This work also clarifies a common misconception: end-to-end multimodality does not automatically solve SlideASR. The paper’s central claim is that the reasoning process itself must be constrained so that the model first “looks” and only then “transcribes.” That position marks a shift from prompt engineering toward explicit policy optimization over multimodal reasoning.

6. Evaluation, failure modes, and adjacent directions

SlideASR evaluation has gradually moved beyond plain WER. SlideSpeech established the now-standard quartet of WER, U-WER, B-WER, and Recall. SlideASR-Bench adds NE-WER and NE-FNR for named-entity portions. “Do Slides Help?” introduces reference-centric and hypothesis-centric special-word error rates, $\text{WER}_{t_{ref}}$ and $\text{WER}_{t_{hyp}}$, to separate omission from hallucination in domain terminology recognition (Wang et al., 2023, Hu et al., 8 Oct 2025, Sinhamahapatra et al., 15 Oct 2025).

The literature also converges on several limitations. Slide content is often summarized rather than verbatim; speakers paraphrase slides; OCR and VLM extraction can hallucinate or miss text; relevant keywords are sparse inside extensive contextual noise; and strong multimodal models may still be misled by context. SlideAVSR further shows that larger prompt sizes generally help, but its FQ Ranker is not universally beneficial, suggesting that reordering OCR terms by rarity can disrupt useful contextual order (Wang et al., 2024, Sinhamahapatra et al., 15 Oct 2025, Rong et al., 14 Nov 2025).

Adjacent work extends the same intuition to preprocessing rather than transcription. pTSE-T conditions target speaker extraction on semantic cues derived from limited and unaligned text content such as condensed points from a presentation slide. Its direct Text Prompt Extractor reports SI-SDRi of 12.16 dB, SDRi of 12.66 dB, PESQi of 0.830, and STOIi of 0.150, indicating that slide text can also be used to isolate the presenter’s speech before ASR (Jiang et al., 2024).

Taken together, these results suggest that SlideASR has evolved from OCR-assisted contextual biasing into a broader multimodal transcription paradigm. Its central technical problem is no longer whether slides help, but how to exploit slide evidence without overwhelming the model with contextual noise, losing acoustic grounding, or collapsing into OCR imitation.