SlideSpeech: Multimodal ASR and Slide Narration

• SlideSpeech is a multimodal ASR framework that leverages synchronized slide content to enhance speech recognition in conference and course videos.
• It integrates OCR-extracted text, keyword biasing, and long-context modeling to improve transcription accuracy, especially for domain-specific terminology.
• SlideSpeech underpins systems for automated slide narration and presentation delivery, broadening applications to accessibility and content creation.

SlideSpeech is a research term with two closely related uses. In its canonical sense, it denotes a “large-scale audio-visual corpus enriched with slides” introduced for multi-modal automatic speech recognition (ASR), motivated by the observation that existing approaches had largely overlooked supplementary textual information despite the abundance of online conference videos with synchronized slides (Wang et al., 2023). In the broader literature, the same label also appears for systems that convert slides into speech-oriented outputs, including accessibility-oriented slide narration and fully automated presentation delivery (V. et al., 2022, Aggarwal et al., 11 Jan 2025). Across these usages, the unifying idea is that slide content is not peripheral metadata but a structured, time-linked signal that can bias recognition, disambiguate domain terminology, or drive narration.

1. Conceptual scope and problem setting

The original SlideSpeech problem is a form of multimodal ASR in which speech is transcribed with access to real-time synchronized slide content. The motivating setting is online conferences and courses, where slides provide “rich domain-specific information in the form of text and images,” and where speech often contains proper names, technical terms, and discipline-specific expressions that are hard to recover from acoustics alone (Wang et al., 2023).

This framing quickly broadened. Later work argued that slides differ from conventional contextual phrase lists because they are synchronized in real time with the speech and can therefore provide “long contextual bias” rather than only sparse rare-word hints (Yu et al., 2024). Other work reformulated the same general setting as SlideASR, taking as input a synchronized pair $(A, I)$ of audio waveform and slide image and training omni-modal models to exploit the slide directly during transcription (Hu et al., 8 Oct 2025). In parallel, accessibility and presentation-automation papers used “SlideSpeech” for systems that narrate slide images or generate spoken delivery from slide decks (V. et al., 2022, Holmberg, 5 May 2025).

A common misconception is that SlideSpeech is merely OCR-assisted post-correction. The published task formulations are broader. They include keyword-biased decoding, contextual CTC/AED with auxiliary phrase prediction, prompt-based LLM decoding, long-context pruning and integration, and end-to-end reasoning over audio and slide images (Wang et al., 2023, Yang et al., 2024, Rong et al., 14 Nov 2025, Hu et al., 8 Oct 2025).

2. Corpus construction and annotation design

The SlideSpeech corpus introduced in 2023 was assembled from YouTube conference and course videos through a multi-stage pipeline. Candidate videos were retrieved with keywords of the form “$\langle$domain$\rangle$ + conference + online,” queried from YouTube playlists and individual channels, and then manually filtered so that slides occupied at least 50% of the video duration. Video was downloaded with yt-dlp at 720p, while audio was extracted at 16 kHz, mono, 16-bit (Wang et al., 2023).

Candidate segments were produced by applying voice activity detection to segment audio into utterances, followed by an in-house ASR system that generated rough transcripts per segment. Validation then used either YouTube auto-subtitles or uploader-provided captions. Subtitles covering each ASR segment were concatenated, aligned against the ASR candidate transcript via Smith–Waterman, and scored by word error rate, with confidence defined as $1-\mathrm{WER}$. All segments were retained for self-supervised use, but segments with confidence at least 95% were marked for high-quality training (Wang et al., 2023).

The released corpus statistics define the benchmark’s scale and coverage.

Component	Statistics	Notes
Total videos	1,705	YouTube
Raw footage duration	1,000+ hours	Full collection
L95 set	473 h	From 1,659 videos
S95 subset	161 h	From 279 videos
Dev	21 videos, 5.07 h	Manually verified
Test	25 videos, 8.75 h	Manually verified

The dev/test sets are notable because 94% of their segments contain slides, and the overall corpus spans 22 categories, including Computer Science, History, and Parenting (Wang et al., 2023). The data model is similarly explicit: videos are MP4 at 720p; audio is WAV at 16 kHz, mono, 16-bit; transcripts are stored one text file per segment with timecodes and confidence scores; slides are represented by an extracted frame per segment plus OCR output in JSON with bounding boxes and raw text; keywords are stored as per-segment KeyBERT output; and metadata CSV/JSON includes video ID, channel, playlist ID, domain tag, segment boundaries, transcript, confidence, and paths to slide images, OCR results, and keyword lists (Wang et al., 2023).

This annotation design made SlideSpeech more than a speech corpus. It exposed aligned acoustic, visual, OCR, and keyword views of each segment, enabling both pipeline and end-to-end multimodal experiments.

3. Slide-text extraction and the original baseline systems

Slide synchronization in the original SlideSpeech benchmark is segment-centric. For each ASR segment, the middle video frame is selected, TDNet from MMOCR localizes text regions, and OCRNet transcribes each region (Wang et al., 2023). Recognized words are then sorted by $(x,y)$ coordinates to reconstruct reading order, punctuation is cleaned, spurious short tokens with fewer than 2 characters are removed, and case is unified. KeyBERT is applied to the cleaned slide_text to extract the top-$N$ keywords, with $N=50$ (Wang et al., 2023).

The first baseline family is keyword-extraction ASR. At inference time, the CTC beam search is biased with the top-$N$ slide keywords, producing a “keyword bias list” $\{k_1,\dots,k_N\}$ (Wang et al., 2023). The second family is a contextual ASR model that combines CTC/AED with a CPP network. Its architecture uses a 12-block Conformer encoder producing $h^E \in \mathbb{R}^{T\times d}$, a BiLSTM context encoder over BPE-tokenized phrases, a multi-head cross-attention biasing layer, and a CPP network that predicts which contextual phrases appear in speech through an auxiliary CTC loss (Wang et al., 2023). The core biasing equations are

$\langle$0

$\langle$1

Training contextual phrase lists mix simulated phrases, obtained by randomly sampling 3 phrases from the transcript plus distractors and <nobias>, with actual top-$\langle$2 OCR keywords (Wang et al., 2023). Benchmarking used 80-dimensional log-Mel FBank features with a 25 ms window and 10 ms shift, SpecAugment with $\langle$3 and $\langle$4, SentencePiece BPE with 5k subwords plus <nobias>, and Adam with a Noam schedule using warmup of 15k steps and learning rate $\langle$5 (Wang et al., 2023).

The reported metrics distinguish overall WER from B-WER and U-WER, defined over words in and not in the bias list, and also report recall for bias words (Wang et al., 2023). On L95, “Contextual w K” reduced test WER from 12.89% to 12.38%, while improving B-WER and recall, which the paper interprets as evidence that slide text helps correct proprietary and domain words (Wang et al., 2023). The qualitative example in the paper highlights a baseline error on “electroencephalogram” corrected by contextual ASR to “electroencephalographer” when the term is present on the slide (Wang et al., 2023).

These baselines established the fundamental SlideSpeech claim: even a single frame per utterance, reduced to OCR text and keywords, can measurably improve recognition of difficult terminology.

4. Post-release evolution of slide-conditioned ASR

Subsequent work on SlideSpeech substantially expanded the modeling space. One line adopted LLM-based decoding with prompt injection. MaLa-ASR combines a frozen WavLM-Large speech encoder, a trainable linear projector, and a frozen Vicuna-7B decoder; slide keywords are inserted directly into the prompt text, with the instruction to “Use keywords in PPT to improve speech recognition accuracy. But if the keywords are irrelevant, just ignore them” (Yang et al., 2024). On the SlideSpeech test sets, this yielded 11.26% WER and 7.67% B-WER on S95, and 9.14% WER and 5.47% B-WER on L95, with the paper reporting average relative WER drops of 44.7% on S95 and 27.9% on L95 over the original baseline (Yang et al., 2024).

A second line treated slides as long-context bias rather than short phrase lists. LCB-net uses a bi-encoder architecture with a 12-layer Conformer audio encoder, a 12-layer Transformer context encoder, Audio-Context and Context-Audio cross-attention, a decoder, and a biasing prediction module trained with binary cross-entropy (Yu et al., 2024). Its final objective is

$\langle$6

LCB-net also introduces dynamic contextual phrases simulation, including BPE-based simulation of contiguous subsets of BPE units from transcript words (Yu et al., 2024). On SlideSpeech Train-S, the full model with simulation and bias prediction reduced test WER from 21.2 to 19.2, with relative reductions of 10.9%, 10.8%, and 10.9% in WER, U-WER, and B-WER (Yu et al., 2024).

A third line focused on context pruning under very long OCR-derived inputs. SAP$\langle$7, also called “2-TPI,” decomposes contextualized ASR into a speech-driven context pruning model $\langle$8 and a contextual ASR model $\langle$9, both instantiated with a Qwen2-Audio-7B-Instruct backbone and a Speech-Driven Attention-based Pooling module (Rong et al., 14 Nov 2025). On SlideSpeech, SAP$\rangle$0 reports 8.04% WER and 5.29% B-WER on S95 with single-slide context, and 7.71% WER with 4.54% B-WER and 95.59% recall on L95 with 5-slide context (Rong et al., 14 Nov 2025). The same study shows WER remaining nearly flat at 7.7% when a single model trained on 5-slide L95 is evaluated on contexts ranging from 1 to 25 slides, while raw token length grows from about 64 to 1,563 (Rong et al., 14 Nov 2025).

A fourth line pursued end-to-end omni-modal reasoning. VAPO formulates SlideASR as maximizing $\rangle$1 with a “Look before Transcription” procedure enforced through a > <answer> format and optimized with four rewards for format compliance, OCR accuracy, ASR quality, and visual anchoring consistency (Hu et al., 8 Oct 2025). On “SlideSpeech (English),” VAPO-7B reports WER 10.31, B-WER 2.87, U-WER 10.84, and Recall 97.32%, outperforming both audio-only and prior pipeline baselines in the reported table (Hu et al., 8 Oct 2025).

System	Core mechanism	Reported SlideSpeech result
MaLa-ASR (Yang et al., 2024)	Prompt-injected keywords with WavLM-Large + Vicuna-7B	L95: WER 9.14%, B-WER 5.47%
LCB-net (Yu et al., 2024)	Bi-encoder long-context biasing + BCE bias prediction	Train-S Test: WER 19.2 vs 21.2 baseline
SAP$\rangle$2 (Rong et al., 14 Nov 2025)	Two-stage pruning/integration with SDAP	L95 (5-slide): WER 7.71%, B-WER 4.54%
VAPO (Hu et al., 8 Oct 2025)	End-to-end OLLM with <think><answer> policy optimization	SlideSpeech (English): WER 10.31, B-WER 2.87%

Taken together, these results show a clear methodological progression. This suggests that SlideSpeech research has shifted from local keyword biasing toward explicit modeling of context relevance, context length, and multimodal control of generation.

5. Related benchmarks, parallel task formulations, and comparative evidence

SlideSpeech did not remain an isolated benchmark. SlideAVSR introduced a second slide-centric AVSR dataset focused on paper explanation videos. Its corpus contains 245 videos, 220 speakers, 19,463 utterances, and 36.45 total hours, with a DocWhisper baseline that inserts OCR-extracted slide words as a decoder prefix prompt to Whisper large-v3 (Wang et al., 2024). Fine-tuned DocWhisper reduced TestA WER from 8.07 to 6.91 and TestB WER from 11.25 to 10.01 at prompt length $\rangle$3 (Wang et al., 2024). The same study found that gains grow as $\rangle$4 increases, but that the FQ Ranker can help at small $\rangle$5 and become detrimental when $\rangle$6 (Wang et al., 2024). This is an important counterpoint to the idea that simply adding more slide text is always beneficial.

The paper “Do Slides Help? Multi-modal Context for Automatic Transcription of Conference Talks” approached the problem through a different benchmark built from ACL 60/60 technical presentations and synthetic slide generation over MUST-C (Sinhamahapatra et al., 15 Oct 2025). Its experiments with SALMONN and Phi-4-multimodal compared cascaded keyword prompts against end-to-end image fusion. The reported fine-tuning results include SALMONN improving from 17.42% WER zero-shot to 10.24% with LLaVA-extracted words, while Phi reaches 8.70% WER with slide images and 6.73% with reference words (Sinhamahapatra et al., 15 Oct 2025). The paper summarizes the overall effect as a relative reduction in WER of approximately 34% across all words and 35% for domain-specific terms compared to the baseline model (Sinhamahapatra et al., 15 Oct 2025).

Presentation text cues have also been used outside transcription. The pTSE-T task conditions target speaker extraction on “limited and unaligned text content, such as condensed points from a presentation slide,” using either a Text Prompt Extractor with FiLM-conditioned fusion or a DPRNN-TSR contrastive selector over blindly separated sources (Jiang et al., 2024). On MMSpeech-2mix, TPE achieves SI-SDRi of 12.16 dB, SDRi of 12.66 dB, PESQi of 0.830, STOIi of 0.150, and Acc of 96.5% (Jiang et al., 2024). This result broadens the empirical claim of the SlideSpeech family: slide text can function as a semantic cue for source selection, not only as a bias list for ASR.

6. Accessibility, narration, and automated presentation delivery

In a distinct but related line, SlideSpeech refers to systems that generate spoken access to slide content. The “Classroom Slide Narration System” treats the task as image-to-markup generation for blind and visually impaired students, using CSSNet with a ResNet-101 backbone, a Location-Encoded Attention Module, and ASPP to segment slide images into title, paragraph, list, equation, figure, and table regions (V. et al., 2022). The full pipeline combines Tesseract OCR, an encoder–decoder math parser, DocFigure for figure classification, and TabStruct-Net for table structure recognition, then reads the resulting JSON description through TTS in “Read All” or interactive region-level modes (V. et al., 2022). On WiSe and SPaSe, CSSNet achieves 46.74 mIoU and 89.80 PA on WiSe, and 36.17 mIoU and 79.11 PA on SPaSe; in user feedback, CSNS scores 8.1 in non-interactive mode and 8.5 in interactive mode, compared with 5.2 for Facebook AAT and 4.7 for Tesseract only (V. et al., 2022).

Presentation automation extends the same idea from accessibility to end-to-end content delivery. PASS generates slides from general Word or PDF documents, converts bullet lists into narrative paragraphs, and synthesizes one .wav per slide with a Tacotron-2–based model from SpeechBrain (Aggarwal et al., 11 Jan 2025). On the SciDuet test set, GPT-PASS reports Coherence 8.79, Redundancy 8.34, Relevance 9.75, and Average 8.96, while Qwen-PASS reports Average 8.89 (Aggarwal et al., 11 Jan 2025). AutoLectures moves further toward lecture-video synthesis: given a PDF slide deck, it generates spoken-style scripts, extracts OCR elements with polygons, obtains word-level timestamps from TTS, aligns highlight phrases to slide locations with exact, fuzzy, or LLM-based matching, and renders synchronized highlights during video synthesis (Holmberg, 5 May 2025). On a 1,000-sample manually annotated dataset, LLM-based word-level alignment achieves MSR 96.6%, Precision 95.1%, Recall 90.1%, and $\rangle$7 92.5%, with average generation cost under \$1 per hour of video (Holmberg, 5 May 2025).

These systems reverse the information flow of the ASR-centered SlideSpeech corpus. Instead of using slides to improve speech understanding, they use slide structure to generate, synchronize, or narrate speech.

7. Limitations, misconceptions, and open research directions

Several limitations recur across the literature. The original SlideSpeech paper identifies OCR errors from small or decorative fonts, the use of only one frame per segment, and a contextual model limited to 50 keywords (Wang et al., 2023). LCB-net likewise notes reliance on accurate OCR and keyword extraction, with degradation when slide text is noisy or absent (Yu et al., 2024). SAP$\rangle$8 frames the core difficulty as a combination of constrained context windows and the sparsity of relevant information within extensive contextual noise, especially when OCR-derived contexts contain hundreds to thousands of tokens (Rong et al., 14 Nov 2025). VAPO reports inference overhead of about 7 s per sample versus about 2.5 s for vanilla OLLM, and also notes that its current focus is on textual slide content rather than diagrams or formula images (Hu et al., 8 Oct 2025).

A second misconception is that end-to-end multimodal models automatically solve the problem. The VAPO formulation explicitly argues that naive omni-modal LLMs often either ignore the slide or degenerate into pure OCR systems (Hu et al., 8 Oct 2025). Conversely, pipeline systems are not obsolete: DocWhisper, MaLa-ASR, and SAP$\rangle$9 show that carefully designed OCR-to-ASR integration can remain highly competitive, especially when context management is explicit (Wang et al., 2024, Yang et al., 2024, Rong et al., 14 Nov 2025).

The future directions stated in the papers are consistent. They include incorporating the full video stream rather than one frame per segment, adding multilingual slide–speech pairs, jointly training OCR and ASR in an end-to-end multimodal architecture, leveraging slide images and diagrams through visual encoders, refining context masking and dynamic list sizing, and improving real-time efficiency (Wang et al., 2023, Yu et al., 2024, Rong et al., 14 Nov 2025, Hu et al., 8 Oct 2025). This suggests that the long-term trajectory of SlideSpeech is toward richer multimodal grounding rather than simple keyword injection: more temporal coverage, more visual structure, and more selective use of context.