---
title: 'SurgPub-Video: Surgical Video-Language Benchmark'
url: https://www.emergentmind.com/topics/surgpub-video
type: topic
---

# SurgPub-Video: Surgical Video-Language Benchmark

SurgPub-Video is a surgical video-language dataset and benchmark introduced to address a specific limitation of prior surgical vision-language models: most available resources are frame-level datasets and therefore do not adequately represent temporal dynamics, procedural flow, and narrative context. The dataset contains **3,538 original surgical videos**, **10,926 surgical video clips**, **48,520 VQA pairs**, and **25 million annotated frames** across **11 surgical specialties**, all crawled from **25 peer-reviewed medical journals**. The same work introduces **SurgLLaVA-Video**, a surgical vision-language model built on **TinyLLaVA-Video**, and a **video-level surgical Visual Question Answering benchmark** intended to evaluate video-native reasoning across multiple specialties [2508.10054].

## 1. Scope and defining characteristics

SurgPub-Video is defined by scale, specialty breadth, and source curation. Its reported scope includes **11 surgical specialties**, **75 kinds of surgery**, **1,823 anatomical structures**, **40 surgical instruments**, and **1,290 unique procedures**. The dataset comprises **3,538 original surgical videos** that are segmented into **10,926 surgical video clips**, with an **average clip duration of 29.85 seconds** and a **raw source video average length of 323.5 seconds**. The raw videos are described as **high-quality: 1920×1080, 30 Hz** [2508.10054].

A central design choice is the use of **peer-reviewed medical journals** as the sole source of raw videos. The paper identifies this as a corrective to datasets mined from **social media platforms like YouTube**, which it characterizes as clinically unreliable because their visual content and audio narration may be unverified. In this sense, SurgPub-Video is not merely a scale-oriented corpus; it is structured as a clinically curated video-language resource intended for instruction-following and reasoning-based supervision rather than label conversion alone [2508.10054].

| Component | Reported scale | Notes |
|---|---:|---|
| Original videos | 3,538 | From 25 peer-reviewed medical journals |
| Video clips | 10,926 | Average clip duration 29.85 seconds |
| VQA pairs | 48,520 | Video-level instruction-following supervision |
| Annotated frames | 25 million | Across 11 specialties |

Two misconceptions are directly contradicted by the dataset description. First, SurgPub-Video is not a frame collection with post hoc question generation; it is built around **video clips** produced through transcript-guided segmentation. Second, it is not a single-procedure benchmark: the benchmark spans **11 surgical specialties**, including examples explicitly listed as **vascular, cardiology, thoracic, orthopedic, urologic, endocrine, esophageal, liver, lung, mediastinal, and breast-related surgery** [2508.10054].

## 2. Data acquisition and curation pipeline

The curation workflow is a multi-stage pipeline that transforms long journal videos into short surgical clips and then into reasoning-oriented question-answer pairs. The first stage is **raw video collection**, in which the authors crawled **3,538** original videos from **25 peer-reviewed journals**. The second stage is **audio-guided preprocessing**, centered on **OpenAI Whisper** as an audio-to-text agent producing **timestamped transcripts**. These transcripts drive a **coarse-grained video processing (CGVP)** stage that removes **introductions**, **endings**, and other **irrelevant content**, merges adjacent fragments with related semantics, and produces clips of about **15–30 seconds**. The stated goal is to preserve complete surgical actions or procedural substeps such as **vessel dissection**, **fat clearance**, **hemostatic clamp placement**, and **vessel incision** [2508.10054].

The third stage is **fine-grained concept extraction (FGCE)**. An FGCE agent uses the transcript of each clip together with textual information from the corresponding article to determine whether the clip contains concepts of interest and to organize them. The five concept categories are **Instruments**, **Procedural steps**, **Anatomical structures**, **Treatment planning**, and **General surgical knowledge**. The agent records **clip length**, **text**, and the **concepts contained in the clip**, thereby converting the corpus from raw audiovisual material into a semantically indexed instructional dataset [2508.10054].

The fourth stage is **QA generation (QAG)**. Here the paper emphasizes that questions are designed to be **reasoning-based** rather than simple recognition prompts. They are derived from **causal explanations** and **rationales in narration**, and the generated supervision includes both **open-ended answers** and **multiple-choice questions**. The multiple-choice format is constructed with distractor options that are intentionally designed to reduce guessability. The fifth stage is **human expert refinement**, in which medical experts review and iteratively revise the generated VQA pairs to ensure **medical correctness**, **semantic consistency**, and alignment with the original surgical intent [2508.10054].

This workflow suggests a specific epistemic position: narration, article text, and clip structure are treated as complementary sources of procedural knowledge rather than as weak proxies for frame labels. A plausible implication is that SurgPub-Video is aimed less at static object naming than at procedural reasoning over short surgical segments.

## 3. Annotation design and benchmark construction

The dataset’s annotation scheme is organized around **video-level VQA** rather than around conventional frame-level categorical labels. This is a major distinction from many earlier surgical datasets, which are described in the paper as relying on **coarse categorical labels** and on label-to-instruction conversion. In SurgPub-Video, the benchmark is constructed by randomly sampling **20% of VQA pairs from 705 videos**, yielding a final subset of **3,337 samples**. The train and test partitions are **strictly separated by video**, explicitly to prevent leakage [2508.10054].

The benchmark evaluates five VQA task types: **Instrument recognition**, **Anatomical structure recognition**, **Procedural step identification**, **Surgical planning**, and **General surgical knowledge**. The authors also perform balance adjustments: some similar QA pairs from majority categories are removed, while **cardiac surgery samples were reduced to 38.6%** and **vascular** and **mediastinal surgery** were increased to **11.9%** and **7.6%**, respectively. These details indicate that the benchmark is not simply a random extraction from the full corpus; it is a distribution-shaped evaluation set intended to reduce domination by a single specialty [2508.10054].

The benchmark is explicitly positioned against prior surgical VQA resources in five ways. It is **video-level**, not frame-level; it uses **peer-reviewed journal videos**; it provides **multi-specialty coverage**; it includes **reasoning-oriented questions**; and it is built from **structured video clips**, not converted image labels. This benchmark design is important because it shifts the evaluation target from single-frame recognition to temporally grounded multimodal reasoning over short clips [2508.10054].

Within the broader landscape of surgical datasets, this places SurgPub-Video in a distinctive niche. **SurgVU** focuses on **280 video clips** of robotic surgery training with **tool presence labels**, **surgical task labels**, and a tool-detection validation set [2501.09209]. **SurgBench** instead defines a unified benchmarking framework with **53 million frames across 22 surgical procedures and 11 specialties**, and evaluates six task categories across **72 fine-grained tasks** [2506.07603]. SurgPub-Video differs from both by centering video-level VQA and instruction-style supervision.

## 4. SurgLLaVA-Video architecture

The model paired with SurgPub-Video is **SurgLLaVA-Video**, a surgical vision-language model built on **TinyLLaVA-Video** and designed for whole-video input. The architecture has three principal components: a **vision encoder**, a **video group resampler**, and an **LLM**. The input is a video clip
\[
X \in \mathbb{R}^{T \times H \times W \times 3},
\]
which is mapped by the vision encoder to frame-level features
\[
Z \in \mathbb{R}^{T \times N \times D}.
\]
These are reshaped into
\[
\hat{Z} \in \mathbb{R}^{(T \times N) \times D}.
\]
The paper identifies the **video resampler** \(\mathbf{P}_\theta\) as the main architectural distinction from ordinary LLaVA variants: it compresses the sequence into a fixed number of learnable queries while preserving temporal relations across frames and reducing computation. The resampler uses learnable queries \(Q \in \mathbb{R}^{L \times D}\), divides \(\hat{Z}\) and \(Q\) into sub-embeddings, performs cross-attention, and outputs the final visual representation
\[
V \in \mathbb{R}^{L \times D}.
\]
The language model then jointly reasons over visual embeddings \(V\) and a surgical text prompt \(S\) to generate the answer [2508.10054].

The training regime is selective rather than end-to-end from scratch. The paper states that SurgLLaVA-Video is trained by starting from the **pre-trained TinyLLaVA-Video model** and fine-tuning it on SurgPub-Video VQA pairs. The **vision encoder is frozen**, whereas the **LLM is fully fine-tuned** and the **video resampler is fully fine-tuned**. No separate custom loss formula is provided beyond standard VQA instruction tuning; the work implies standard autoregressive response generation over QA pairs [2508.10054].

This design distinguishes SurgLLaVA-Video from earlier surgical conversational systems such as **LLaVA-Surg**, which adapts **Video-ChatGPT** with a **CLIP ViT-L/14** visual encoder and a Llama-based language model for surgical video dialogue, but is trained on **Surg-QA**, a corpus derived from public surgical lecture videos [2408.07981]. It also differs from **VidLPRO**, which performs video-language pre-training through a three-part objective combining **video-text contrastive learning**, **video-text matching**, and **masked language modeling**, and uses **GenSurg+**, a **17k** clip-caption dataset derived from **GenSurgery** [2409.04732]. SurgLLaVA-Video is therefore less a general pretraining framework than a benchmark-driven, instruction-tuned surgical video VLM.

## 5. Empirical performance and ablation results

On the SurgPub-Video benchmark, SurgLLaVA-Video achieves **82.84% overall accuracy**, which the paper reports as the best among all tested models and **about 16.17% higher than GPT-4o**. It also reports that the model improves by **more than 10% on 10 of the 11 specialties**. At the specialty level, **cardiac surgery** reaches **84.38%**, which is reported as over **16% better than GPT-4o**, while **esophageal surgery** reaches **89.69%**, about **11.56% above the runner-up**. The comparison set includes **LLaVA-1.5**, **InternVL3**, **Qwen2.5-VL-Instruct**, **GPT-4o**, **Qwen2.5-Max**, and **Gemini 2.0 Flash** [2508.10054].

The paper emphasizes that these gains are obtained with only **3B parameters**, whereas several compared models are substantially larger. It therefore presents SurgLLaVA-Video as evidence that surgical-domain video supervision can outweigh model scale when the evaluation target is surgical procedural reasoning rather than generic multimodal competence [2508.10054].

The downstream evaluations extend beyond VQA. On **SAR-RARP** for fine-grained robotic prostatectomy action recognition, SurgLLaVA-Video reports **Accuracy 65.65**, **Recall 45.50**, **Precision 55.35**, and **Jaccard 33.31**, compared with the prior surgical baseline **SurgVLM** at **Accuracy 42.90**, **Recall 34.64**, **Precision 31.45**, and **Jaccard 19.22**. On **Endoscapes2023 CVS** for critical view of safety assessment, the model achieves **58.88% average balanced accuracy**, compared with **53.50%** for **InternVL3**. On **CholecT50** triplet recognition, SurgLLaVA-Video reports **Instrument accuracy 85.93**, **Verb 63.29**, **Target 47.12**, and **Triplet 39.30**, with corresponding mAP values of **58.98**, **42.75**, **19.93**, and **10.61**. The paper explicitly notes one exception: **SurgVLM** still has better **target accuracy**, at **57.07% vs. 47.12%** [2508.10054].

Ablation studies isolate two variables. For input length, the paper evaluates **8**, **16**, **24**, and **32** frames and concludes that **16 frames is optimal overall, with 80.82%**. The stated interpretation is that too few frames lose context, whereas too many introduce redundancy or noise. For dataset scale, the model is trained on **25%**, **50%**, **75%**, and **100%** of SurgPub-Video, and the paper states that performance rises almost linearly with more data, with the full dataset again giving the best overall accuracy of **80.82%** in the ablation setting [2508.10054].

These results align with a broader trend in the literature. **VidLPRO** shows that richer multimodal pretraining objectives improve zero-shot phase recognition on **Cholec80** and **AutoLaparo** [2409.04732]. **SurgBench** reports that surgical-domain continual pretraining improves average **top-1 accuracy by 7%** and **top-3 accuracy by 7.9%** over a general-domain VideoMAE baseline [2506.07603]. SurgPub-Video contributes to that trend by supplying instruction-tuning data and a benchmark centered on reasoning over video clips rather than on single downstream labels.

## 6. Position within surgical video-language research, significance, and constraints

SurgPub-Video sits at the intersection of several research threads: large-scale surgical data curation, multimodal pretraining, and video-native reasoning. Earlier work such as **SurgVLP** learned multimodal representations from **1,326 surgical lectures** using dual-encoder contrastive learning over ASR-derived text [2307.15220]. **LLaVA-Surg** then introduced **Surg-QA**, with **102K video-question-answer pairs** from public surgical lecture videos, and trained a video-conversational assistant [2408.07981]. **VidLPRO** moved toward a more formal surgical VL pre-training framework with a three-objective loss and **GenSurg+** captioned data [2409.04732]. Against this background, SurgPub-Video is distinguished by three features: direct use of **peer-reviewed journal videos**, emphasis on **video-level VQA**, and a benchmark that explicitly spans **11 specialties** [2508.10054].

Its significance is also easier to interpret when contrasted with adjacent dataset-building strategies. **SurgVU** is a large public robotic training-video resource with tool and task labels, but its supervision is not VQA-oriented [2501.09209]. **SurgBench** provides a broader pretraining and evaluation framework for surgical video analysis, but its task design is label-centric rather than instruction-centric [2506.07603]. Later work such as **SurgAtlas** expands scale dramatically to **15,291 videos (2,391 hours)** across **18 surgical specialties** and **over 5,000 procedure types**, and explicitly includes **open surgery** at scale, but it is sourced entirely from **publicly available YouTube content** [2606.25905]. This contrast suggests differing design priorities between source reliability and corpus breadth.

Several constraints are either stated or evident in the SurgPub-Video description. The benchmark and dataset are limited to the **selected 11 specialties** and to **peer-reviewed journal sources**. Data extraction relies on **ASR/transcripts**, which can still be noisy before filtering. The model uses a relatively standard architecture built on **TinyLLaVA-Video** and does not introduce a new temporal transformer beyond the **video resampler**. Performance is not uniformly superior across all downstream subtasks, as seen in the weaker **target accuracy** on **CholecT50** relative to **SurgVLM**. The work also depends on the quality and completeness of **journal narration and reports** [2508.10054].

Even with those constraints, the dataset’s role in the literature is clear. The paper presents SurgPub-Video as a move from **static frame classification** toward **procedural video understanding** grounded in **peer-reviewed clinical content** and expressed through **instruction-following supervision**. In that formulation, its primary contribution is not only a larger corpus, but a redefinition of what surgical video-language evaluation should measure: temporally grounded recognition, explanation, and procedural reasoning over real surgical clips rather than isolated images [2508.10054].

Source: https://www.emergentmind.com/topics/surgpub-video