---
title: 'SlideSpeech: Multimodal ASR and Slide Narration'
url: https://www.emergentmind.com/topics/slidespeech
type: topic
---

# SlideSpeech: Multimodal ASR and Slide Narration

SlideSpeech is a research term with two closely related uses. In its canonical sense, it denotes a “large-scale audio-visual corpus enriched with slides” introduced for multi-modal automatic speech recognition (ASR), motivated by the observation that existing approaches had largely overlooked supplementary textual information despite the abundance of online conference videos with synchronized slides [2309.05396]. In the broader literature, the same label also appears for systems that convert slides into speech-oriented outputs, including accessibility-oriented slide narration and fully automated presentation delivery [2201.08574; 2501.06497]. Across these usages, the unifying idea is that slide content is not peripheral metadata but a structured, time-linked signal that can bias recognition, disambiguate domain terminology, or drive narration.

## 1. Conceptual scope and problem setting

The original SlideSpeech problem is a form of multimodal ASR in which speech is transcribed with access to real-time synchronized slide content. The motivating setting is online conferences and courses, where slides provide “rich domain-specific information in the form of text and images,” and where speech often contains proper names, technical terms, and discipline-specific expressions that are hard to recover from acoustics alone [2309.05396].

This framing quickly broadened. Later work argued that slides differ from conventional contextual phrase lists because they are synchronized in real time with the speech and can therefore provide “long contextual bias” rather than only sparse rare-word hints [2401.06390]. Other work reformulated the same general setting as SlideASR, taking as input a synchronized pair $(A, I)$ of audio waveform and slide image and training omni-modal models to exploit the slide directly during transcription [2510.08618]. In parallel, accessibility and presentation-automation papers used “SlideSpeech” for systems that narrate slide images or generate spoken delivery from slide decks [2201.08574; 2505.02966].

A common misconception is that SlideSpeech is merely OCR-assisted post-correction. The published task formulations are broader. They include keyword-biased decoding, contextual CTC/AED with auxiliary phrase prediction, prompt-based large language model decoding, long-context pruning and integration, and end-to-end reasoning over audio and slide images [2309.05396; 2406.05839; 2511.11139; 2510.08618].

## 2. Corpus construction and annotation design

The SlideSpeech corpus introduced in 2023 was assembled from YouTube conference and course videos through a multi-stage pipeline. Candidate videos were retrieved with keywords of the form “$\langle$domain$\rangle$ + conference + online,” queried from YouTube playlists and individual channels, and then manually filtered so that slides occupied at least 50% of the video duration. Video was downloaded with `yt-dlp` at 720p, while audio was extracted at 16 kHz, mono, 16-bit [2309.05396].

Candidate segments were produced by applying voice activity detection to segment audio into utterances, followed by an in-house ASR system that generated rough transcripts per segment. Validation then used either YouTube auto-subtitles or uploader-provided captions. Subtitles covering each ASR segment were concatenated, aligned against the ASR candidate transcript via Smith–Waterman, and scored by word error rate, with confidence defined as $1-\mathrm{WER}$. All segments were retained for self-supervised use, but segments with confidence at least 95% were marked for high-quality training [2309.05396].

The released corpus statistics define the benchmark’s scale and coverage.

| Component | Statistics | Notes |
|---|---:|---|
| Total videos | 1,705 | YouTube |
| Raw footage duration | 1,000+ hours | Full collection |
| L95 set | 473 h | From 1,659 videos |
| S95 subset | 161 h | From 279 videos |
| Dev | 21 videos, 5.07 h | Manually verified |
| Test | 25 videos, 8.75 h | Manually verified |

The dev/test sets are notable because 94% of their segments contain slides, and the overall corpus spans 22 categories, including Computer Science, History, and Parenting [2309.05396]. The data model is similarly explicit: videos are MP4 at 720p; audio is WAV at 16 kHz, mono, 16-bit; transcripts are stored one text file per segment with timecodes and confidence scores; slides are represented by an extracted frame per segment plus OCR output in JSON with bounding boxes and raw text; keywords are stored as per-segment KeyBERT output; and metadata CSV/JSON includes video ID, channel, playlist ID, domain tag, segment boundaries, transcript, confidence, and paths to slide images, OCR results, and keyword lists [2309.05396].

This annotation design made SlideSpeech more than a speech corpus. It exposed aligned acoustic, visual, OCR, and keyword views of each segment, enabling both pipeline and end-to-end multimodal experiments.

## 3. Slide-text extraction and the original baseline systems

Slide synchronization in the original SlideSpeech benchmark is segment-centric. For each ASR segment, the middle video frame is selected, TDNet from MMOCR localizes text regions, and OCRNet transcribes each region [2309.05396]. Recognized words are then sorted by $(x,y)$ coordinates to reconstruct reading order, punctuation is cleaned, spurious short tokens with fewer than 2 characters are removed, and case is unified. KeyBERT is applied to the cleaned `slide_text` to extract the top-$N$ keywords, with $N=50$ [2309.05396].

The first baseline family is keyword-extraction ASR. At inference time, the CTC beam search is biased with the top-$N$ slide keywords, producing a “keyword bias list” $\{k_1,\dots,k_N\}$ [2309.05396]. The second family is a contextual ASR model that combines CTC/AED with a CPP network. Its architecture uses a 12-block Conformer encoder producing $h^E \in \mathbb{R}^{T\times d}$, a BiLSTM context encoder over BPE-tokenized phrases, a multi-head cross-attention biasing layer, and a CPP network that predicts which contextual phrases appear in speech through an auxiliary CTC loss [2309.05396]. The core biasing equations are

$$
c^{E} = \mathrm{MHA}\bigl(\mathrm{LN}(h^{E}),\,\mathrm{LN}(h^{CE}),\,\mathrm{LN}(h^{CE})\bigr),
$$

$$
h^{CA} = \mathrm{FFN}\bigl([\mathrm{LN}(h^{E});\,\mathrm{LN}(c^{E})]\bigr).
$$

Training contextual phrase lists mix simulated phrases, obtained by randomly sampling 3 phrases from the transcript plus distractors and `<nobias>`, with actual top-$N$ OCR keywords [2309.05396]. Benchmarking used 80-dimensional log-Mel FBank features with a 25 ms window and 10 ms shift, SpecAugment with $F=30$ and $T=40$, SentencePiece BPE with 5k subwords plus `<nobias>`, and Adam with a Noam schedule using warmup of 15k steps and learning rate $2\mathrm{e}{-3}$ [2309.05396].

The reported metrics distinguish overall WER from B-WER and U-WER, defined over words in and not in the bias list, and also report recall for bias words [2309.05396]. On L95, “Contextual w K” reduced test WER from 12.89% to 12.38%, while improving B-WER and recall, which the paper interprets as evidence that slide text helps correct proprietary and domain words [2309.05396]. The qualitative example in the paper highlights a baseline error on “electroencephalogram” corrected by contextual ASR to “electroencephalographer” when the term is present on the slide [2309.05396].

These baselines established the fundamental SlideSpeech claim: even a single frame per utterance, reduced to OCR text and keywords, can measurably improve recognition of difficult terminology.

## 4. Post-release evolution of slide-conditioned ASR

Subsequent work on SlideSpeech substantially expanded the modeling space. One line adopted LLM-based decoding with prompt injection. MaLa-ASR combines a frozen WavLM-Large speech encoder, a trainable linear projector, and a frozen Vicuna-7B decoder; slide keywords are inserted directly into the prompt text, with the instruction to “Use keywords in PPT to improve speech recognition accuracy. But if the keywords are irrelevant, just ignore them” [2406.05839]. On the SlideSpeech test sets, this yielded 11.26% WER and 7.67% B-WER on S95, and 9.14% WER and 5.47% B-WER on L95, with the paper reporting average relative WER drops of 44.7% on S95 and 27.9% on L95 over the original baseline [2406.05839].

A second line treated slides as long-context bias rather than short phrase lists. LCB-net uses a bi-encoder architecture with a 12-layer Conformer audio encoder, a 12-layer Transformer context encoder, Audio-Context and Context-Audio cross-attention, a decoder, and a biasing prediction module trained with binary cross-entropy [2401.06390]. Its final objective is

$$
L = L_{CTC} + L_{CE} + \lambda \cdot L_{bce}.
$$

LCB-net also introduces dynamic contextual phrases simulation, including BPE-based simulation of contiguous subsets of BPE units from transcript words [2401.06390]. On SlideSpeech Train-S, the full model with simulation and bias prediction reduced test WER from 21.2 to 19.2, with relative reductions of 10.9%, 10.8%, and 10.9% in WER, U-WER, and B-WER [2401.06390].

A third line focused on context pruning under very long OCR-derived inputs. SAP$^{2}$, also called “2-TPI,” decomposes contextualized ASR into a speech-driven context pruning model $p_\psi(\tilde Z \mid X,Z)$ and a contextual ASR model $p_\theta(Y \mid X,\tilde Z)$, both instantiated with a Qwen2-Audio-7B-Instruct backbone and a Speech-Driven Attention-based Pooling module [2511.11139]. On SlideSpeech, SAP$^{2}$ reports 8.04% WER and 5.29% B-WER on S95 with single-slide context, and 7.71% WER with 4.54% B-WER and 95.59% recall on L95 with 5-slide context [2511.11139]. The same study shows WER remaining nearly flat at 7.7% when a single model trained on 5-slide L95 is evaluated on contexts ranging from 1 to 25 slides, while raw token length grows from about 64 to 1,563 [2511.11139].

A fourth line pursued end-to-end omni-modal reasoning. VAPO formulates SlideASR as maximizing $P(Y \mid A, I; \theta)$ with a “Look before Transcription” procedure enforced through a `<think><answer>` format and optimized with four rewards for format compliance, OCR accuracy, ASR quality, and visual anchoring consistency [2510.08618]. On “SlideSpeech (English),” VAPO-7B reports WER 10.31, B-WER 2.87, U-WER 10.84, and Recall 97.32%, outperforming both audio-only and prior pipeline baselines in the reported table [2510.08618].

| System | Core mechanism | Reported SlideSpeech result |
|---|---|---|
| MaLa-ASR [2406.05839] | Prompt-injected keywords with WavLM-Large + Vicuna-7B | L95: WER 9.14%, B-WER 5.47% |
| LCB-net [2401.06390] | Bi-encoder long-context biasing + BCE bias prediction | Train-S Test: WER 19.2 vs 21.2 baseline |
| SAP$^{2}$ [2511.11139] | Two-stage pruning/integration with SDAP | L95 (5-slide): WER 7.71%, B-WER 4.54% |
| VAPO [2510.08618] | End-to-end OLLM with `<think><answer>` policy optimization | SlideSpeech (English): WER 10.31, B-WER 2.87% |

Taken together, these results show a clear methodological progression. This suggests that SlideSpeech research has shifted from local keyword biasing toward explicit modeling of context relevance, context length, and multimodal control of generation.

## 5. Related benchmarks, parallel task formulations, and comparative evidence

SlideSpeech did not remain an isolated benchmark. SlideAVSR introduced a second slide-centric AVSR dataset focused on paper explanation videos. Its corpus contains 245 videos, 220 speakers, 19,463 utterances, and 36.45 total hours, with a DocWhisper baseline that inserts OCR-extracted slide words as a decoder prefix prompt to Whisper large-v3 [2401.09759]. Fine-tuned DocWhisper reduced TestA WER from 8.07 to 6.91 and TestB WER from 11.25 to 10.01 at prompt length $K=100$ [2401.09759]. The same study found that gains grow as $K$ increases, but that the FQ Ranker can help at small $K$ and become detrimental when $K>50$ [2401.09759]. This is an important counterpoint to the idea that simply adding more slide text is always beneficial.

The paper “Do Slides Help? Multi-modal Context for Automatic Transcription of Conference Talks” approached the problem through a different benchmark built from ACL 60/60 technical presentations and synthetic slide generation over MUST-C [2510.13979]. Its experiments with SALMONN and Phi-4-multimodal compared cascaded keyword prompts against end-to-end image fusion. The reported fine-tuning results include SALMONN improving from 17.42% WER zero-shot to 10.24% with LLaVA-extracted words, while Phi reaches 8.70% WER with slide images and 6.73% with reference words [2510.13979]. The paper summarizes the overall effect as a relative reduction in WER of approximately 34% across all words and 35% for domain-specific terms compared to the baseline model [2510.13979].

Presentation text cues have also been used outside transcription. The pTSE-T task conditions target speaker extraction on “limited and unaligned text content, such as condensed points from a presentation slide,” using either a Text Prompt Extractor with FiLM-conditioned fusion or a DPRNN-TSR contrastive selector over blindly separated sources [2411.03109]. On MMSpeech-2mix, TPE achieves SI-SDRi of 12.16 dB, SDRi of 12.66 dB, PESQi of 0.830, STOIi of 0.150, and Acc of 96.5% [2411.03109]. This result broadens the empirical claim of the SlideSpeech family: slide text can function as a semantic cue for source selection, not only as a bias list for ASR.

## 6. Accessibility, narration, and automated presentation delivery

In a distinct but related line, SlideSpeech refers to systems that generate spoken access to slide content. The “Classroom Slide Narration System” treats the task as image-to-markup generation for blind and visually impaired students, using CSSNet with a ResNet-101 backbone, a Location-Encoded Attention Module, and ASPP to segment slide images into title, paragraph, list, equation, figure, and table regions [2201.08574]. The full pipeline combines Tesseract OCR, an encoder–decoder math parser, DocFigure for figure classification, and TabStruct-Net for table structure recognition, then reads the resulting JSON description through TTS in “Read All” or interactive region-level modes [2201.08574]. On WiSe and SPaSe, CSSNet achieves 46.74 mIoU and 89.80 PA on WiSe, and 36.17 mIoU and 79.11 PA on SPaSe; in user feedback, CSNS scores 8.1 in non-interactive mode and 8.5 in interactive mode, compared with 5.2 for Facebook AAT and 4.7 for Tesseract only [2201.08574].

Presentation automation extends the same idea from accessibility to end-to-end content delivery. PASS generates slides from general Word or PDF documents, converts bullet lists into narrative paragraphs, and synthesizes one `.wav` per slide with a Tacotron-2–based model from SpeechBrain [2501.06497]. On the SciDuet test set, GPT-PASS reports Coherence 8.79, Redundancy 8.34, Relevance 9.75, and Average 8.96, while Qwen-PASS reports Average 8.89 [2501.06497]. AutoLectures moves further toward lecture-video synthesis: given a PDF slide deck, it generates spoken-style scripts, extracts OCR elements with polygons, obtains word-level timestamps from TTS, aligns highlight phrases to slide locations with exact, fuzzy, or LLM-based matching, and renders synchronized highlights during video synthesis [2505.02966]. On a 1,000-sample manually annotated dataset, LLM-based word-level alignment achieves MSR 96.6%, Precision 95.1%, Recall 90.1%, and $F_1$ 92.5%, with average generation cost under \$1 per hour of video [2505.02966].

These systems reverse the information flow of the ASR-centered SlideSpeech corpus. Instead of using slides to improve speech understanding, they use slide structure to generate, synchronize, or narrate speech.

## 7. Limitations, misconceptions, and open research directions

Several limitations recur across the literature. The original SlideSpeech paper identifies OCR errors from small or decorative fonts, the use of only one frame per segment, and a contextual model limited to 50 keywords [2309.05396]. LCB-net likewise notes reliance on accurate OCR and keyword extraction, with degradation when slide text is noisy or absent [2401.06390]. SAP$^{2}$ frames the core difficulty as a combination of constrained context windows and the sparsity of relevant information within extensive contextual noise, especially when OCR-derived contexts contain hundreds to thousands of tokens [2511.11139]. VAPO reports inference overhead of about 7 s per sample versus about 2.5 s for vanilla OLLM, and also notes that its current focus is on textual slide content rather than diagrams or formula images [2510.08618].

A second misconception is that end-to-end multimodal models automatically solve the problem. The VAPO formulation explicitly argues that naive omni-modal large language models often either ignore the slide or degenerate into pure OCR systems [2510.08618]. Conversely, pipeline systems are not obsolete: DocWhisper, MaLa-ASR, and SAP$^{2}$ show that carefully designed OCR-to-ASR integration can remain highly competitive, especially when context management is explicit [2401.09759; 2406.05839; 2511.11139].

The future directions stated in the papers are consistent. They include incorporating the full video stream rather than one frame per segment, adding multilingual slide–speech pairs, jointly training OCR and ASR in an end-to-end multimodal architecture, leveraging slide images and diagrams through visual encoders, refining context masking and dynamic list sizing, and improving real-time efficiency [2309.05396; 2401.06390; 2511.11139; 2510.08618]. This suggests that the long-term trajectory of SlideSpeech is toward richer multimodal grounding rather than simple keyword injection: more temporal coverage, more visual structure, and more selective use of context.

Source: https://www.emergentmind.com/topics/slidespeech