SurgPub-Video: Surgical Video-Language Benchmark

• SurgPub-Video is a large-scale, clinically curated surgical video-language dataset featuring 3,538 videos and spanning 11 specialties.
• It segments long journal videos into 10,926 concise clips with 48,520 VQA pairs, preserving temporal dynamics and procedural context.
• The dataset underpins SurgLLaVA-Video, a model achieving high accuracy in multi-specialty VQA by leveraging instruction-following supervision.

SurgPub-Video is a surgical video-language dataset and benchmark introduced to address a specific limitation of prior surgical vision-LLMs: most available resources are frame-level datasets and therefore do not adequately represent temporal dynamics, procedural flow, and narrative context. The dataset contains 3,538 original surgical videos, 10,926 surgical video clips, 48,520 VQA pairs, and 25 million annotated frames across 11 surgical specialties, all crawled from 25 peer-reviewed medical journals. The same work introduces SurgLLaVA-Video, a surgical vision-LLM built on TinyLLaVA-Video, and a video-level surgical Visual Question Answering benchmark intended to evaluate video-native reasoning across multiple specialties (Li et al., 12 Aug 2025).

1. Scope and defining characteristics

SurgPub-Video is defined by scale, specialty breadth, and source curation. Its reported scope includes 11 surgical specialties, 75 kinds of surgery, 1,823 anatomical structures, 40 surgical instruments, and 1,290 unique procedures. The dataset comprises 3,538 original surgical videos that are segmented into 10,926 surgical video clips, with an average clip duration of 29.85 seconds and a raw source video average length of 323.5 seconds. The raw videos are described as high-quality: 1920×1080, 30 Hz (Li et al., 12 Aug 2025).

A central design choice is the use of peer-reviewed medical journals as the sole source of raw videos. The paper identifies this as a corrective to datasets mined from social media platforms like YouTube, which it characterizes as clinically unreliable because their visual content and audio narration may be unverified. In this sense, SurgPub-Video is not merely a scale-oriented corpus; it is structured as a clinically curated video-language resource intended for instruction-following and reasoning-based supervision rather than label conversion alone (Li et al., 12 Aug 2025).

Component	Reported scale	Notes
Original videos	3,538	From 25 peer-reviewed medical journals
Video clips	10,926	Average clip duration 29.85 seconds
VQA pairs	48,520	Video-level instruction-following supervision
Annotated frames	25 million	Across 11 specialties

Two misconceptions are directly contradicted by the dataset description. First, SurgPub-Video is not a frame collection with post hoc question generation; it is built around video clips produced through transcript-guided segmentation. Second, it is not a single-procedure benchmark: the benchmark spans 11 surgical specialties, including examples explicitly listed as vascular, cardiology, thoracic, orthopedic, urologic, endocrine, esophageal, liver, lung, mediastinal, and breast-related surgery (Li et al., 12 Aug 2025).

2. Data acquisition and curation pipeline

The curation workflow is a multi-stage pipeline that transforms long journal videos into short surgical clips and then into reasoning-oriented question-answer pairs. The first stage is raw video collection, in which the authors crawled 3,538 original videos from 25 peer-reviewed journals. The second stage is audio-guided preprocessing, centered on OpenAI Whisper as an audio-to-text agent producing timestamped transcripts. These transcripts drive a coarse-grained video processing (CGVP) stage that removes introductions, endings, and other irrelevant content, merges adjacent fragments with related semantics, and produces clips of about 15–30 seconds. The stated goal is to preserve complete surgical actions or procedural substeps such as vessel dissection, fat clearance, hemostatic clamp placement, and vessel incision (Li et al., 12 Aug 2025).

The third stage is fine-grained concept extraction (FGCE). An FGCE agent uses the transcript of each clip together with textual information from the corresponding article to determine whether the clip contains concepts of interest and to organize them. The five concept categories are Instruments, Procedural steps, Anatomical structures, Treatment planning, and General surgical knowledge. The agent records clip length, text, and the concepts contained in the clip, thereby converting the corpus from raw audiovisual material into a semantically indexed instructional dataset (Li et al., 12 Aug 2025).

The fourth stage is QA generation (QAG). Here the paper emphasizes that questions are designed to be reasoning-based rather than simple recognition prompts. They are derived from causal explanations and rationales in narration, and the generated supervision includes both open-ended answers and multiple-choice questions. The multiple-choice format is constructed with distractor options that are intentionally designed to reduce guessability. The fifth stage is human expert refinement, in which medical experts review and iteratively revise the generated VQA pairs to ensure medical correctness, semantic consistency, and alignment with the original surgical intent (Li et al., 12 Aug 2025).

This workflow suggests a specific epistemic position: narration, article text, and clip structure are treated as complementary sources of procedural knowledge rather than as weak proxies for frame labels. A plausible implication is that SurgPub-Video is aimed less at static object naming than at procedural reasoning over short surgical segments.

3. Annotation design and benchmark construction

The dataset’s annotation scheme is organized around video-level VQA rather than around conventional frame-level categorical labels. This is a major distinction from many earlier surgical datasets, which are described in the paper as relying on coarse categorical labels and on label-to-instruction conversion. In SurgPub-Video, the benchmark is constructed by randomly sampling 20% of VQA pairs from 705 videos, yielding a final subset of 3,337 samples. The train and test partitions are strictly separated by video, explicitly to prevent leakage (Li et al., 12 Aug 2025).

The benchmark evaluates five VQA task types: Instrument recognition, Anatomical structure recognition, Procedural step identification, Surgical planning, and General surgical knowledge. The authors also perform balance adjustments: some similar QA pairs from majority categories are removed, while cardiac surgery samples were reduced to 38.6% and vascular and mediastinal surgery were increased to 11.9% and 7.6%, respectively. These details indicate that the benchmark is not simply a random extraction from the full corpus; it is a distribution-shaped evaluation set intended to reduce domination by a single specialty (Li et al., 12 Aug 2025).

The benchmark is explicitly positioned against prior surgical VQA resources in five ways. It is video-level, not frame-level; it uses peer-reviewed journal videos; it provides multi-specialty coverage; it includes reasoning-oriented questions; and it is built from structured video clips, not converted image labels. This benchmark design is important because it shifts the evaluation target from single-frame recognition to temporally grounded multimodal reasoning over short clips (Li et al., 12 Aug 2025).

Within the broader landscape of surgical datasets, this places SurgPub-Video in a distinctive niche. SurgVU focuses on 280 video clips of robotic surgery training with tool presence labels, surgical task labels, and a tool-detection validation set (Zia et al., 16 Jan 2025). SurgBench instead defines a unified benchmarking framework with 53 million frames across 22 surgical procedures and 11 specialties, and evaluates six task categories across 72 fine-grained tasks (Wei et al., 9 Jun 2025). SurgPub-Video differs from both by centering video-level VQA and instruction-style supervision.

4. SurgLLaVA-Video architecture

The model paired with SurgPub-Video is SurgLLaVA-Video, a surgical vision-LLM built on TinyLLaVA-Video and designed for whole-video input. The architecture has three principal components: a vision encoder, a video group resampler, and an LLM. The input is a video clip

$X \in \mathbb{R}^{T \times H \times W \times 3},$

which is mapped by the vision encoder to frame-level features

$Z \in \mathbb{R}^{T \times N \times D}.$

These are reshaped into

$\hat{Z} \in \mathbb{R}^{(T \times N) \times D}.$

The paper identifies the video resampler $\mathbf{P}_\theta$ as the main architectural distinction from ordinary LLaVA variants: it compresses the sequence into a fixed number of learnable queries while preserving temporal relations across frames and reducing computation. The resampler uses learnable queries $Q \in \mathbb{R}^{L \times D}$, divides $\hat{Z}$ and $Q$ into sub-embeddings, performs cross-attention, and outputs the final visual representation

$V \in \mathbb{R}^{L \times D}.$

The LLM then jointly reasons over visual embeddings $V$ and a surgical text prompt $S$ to generate the answer (Li et al., 12 Aug 2025).

The training regime is selective rather than end-to-end from scratch. The paper states that SurgLLaVA-Video is trained by starting from the pre-trained TinyLLaVA-Video model and fine-tuning it on SurgPub-Video VQA pairs. The vision encoder is frozen, whereas the LLM is fully fine-tuned and the video resampler is fully fine-tuned. No separate custom loss formula is provided beyond standard VQA instruction tuning; the work implies standard autoregressive response generation over QA pairs (Li et al., 12 Aug 2025).

This design distinguishes SurgLLaVA-Video from earlier surgical conversational systems such as LLaVA-Surg, which adapts Video-ChatGPT with a CLIP ViT-L/14 visual encoder and a Llama-based LLM for surgical video dialogue, but is trained on Surg-QA, a corpus derived from public surgical lecture videos (Li et al., 2024). It also differs from VidLPRO, which performs video-language pre-training through a three-part objective combining video-text contrastive learning, video-text matching, and masked language modeling, and uses GenSurg+, a 17k clip-caption dataset derived from GenSurgery (Honarmand et al., 2024). SurgLLaVA-Video is therefore less a general pretraining framework than a benchmark-driven, instruction-tuned surgical video VLM.

5. Empirical performance and ablation results

On the SurgPub-Video benchmark, SurgLLaVA-Video achieves 82.84% overall accuracy, which the paper reports as the best among all tested models and about 16.17% higher than GPT-4o. It also reports that the model improves by more than 10% on 10 of the 11 specialties. At the specialty level, cardiac surgery reaches 84.38%, which is reported as over 16% better than GPT-4o, while esophageal surgery reaches 89.69%, about 11.56% above the runner-up. The comparison set includes LLaVA-1.5, InternVL3, Qwen2.5-VL-Instruct, GPT-4o, Qwen2.5-Max, and Gemini 2.0 Flash (Li et al., 12 Aug 2025).

The paper emphasizes that these gains are obtained with only 3B parameters, whereas several compared models are substantially larger. It therefore presents SurgLLaVA-Video as evidence that surgical-domain video supervision can outweigh model scale when the evaluation target is surgical procedural reasoning rather than generic multimodal competence (Li et al., 12 Aug 2025).

The downstream evaluations extend beyond VQA. On SAR-RARP for fine-grained robotic prostatectomy action recognition, SurgLLaVA-Video reports Accuracy 65.65, Recall 45.50, Precision 55.35, and Jaccard 33.31, compared with the prior surgical baseline SurgVLM at Accuracy 42.90, Recall 34.64, Precision 31.45, and Jaccard 19.22. On Endoscapes2023 CVS for critical view of safety assessment, the model achieves 58.88% average balanced accuracy, compared with 53.50% for InternVL3. On CholecT50 triplet recognition, SurgLLaVA-Video reports Instrument accuracy 85.93, Verb 63.29, Target 47.12, and Triplet 39.30, with corresponding mAP values of 58.98, 42.75, 19.93, and 10.61. The paper explicitly notes one exception: SurgVLM still has better target accuracy, at 57.07% vs. 47.12% (Li et al., 12 Aug 2025).

Ablation studies isolate two variables. For input length, the paper evaluates 8, 16, 24, and 32 frames and concludes that 16 frames is optimal overall, with 80.82%. The stated interpretation is that too few frames lose context, whereas too many introduce redundancy or noise. For dataset scale, the model is trained on 25%, 50%, 75%, and 100% of SurgPub-Video, and the paper states that performance rises almost linearly with more data, with the full dataset again giving the best overall accuracy of 80.82% in the ablation setting (Li et al., 12 Aug 2025).

These results align with a broader trend in the literature. VidLPRO shows that richer multimodal pretraining objectives improve zero-shot phase recognition on Cholec80 and AutoLaparo (Honarmand et al., 2024). SurgBench reports that surgical-domain continual pretraining improves average top-1 accuracy by 7% and top-3 accuracy by 7.9% over a general-domain VideoMAE baseline (Wei et al., 9 Jun 2025). SurgPub-Video contributes to that trend by supplying instruction-tuning data and a benchmark centered on reasoning over video clips rather than on single downstream labels.

6. Position within surgical video-language research, significance, and constraints

SurgPub-Video sits at the intersection of several research threads: large-scale surgical data curation, multimodal pretraining, and video-native reasoning. Earlier work such as SurgVLP learned multimodal representations from 1,326 surgical lectures using dual-encoder contrastive learning over ASR-derived text (Yuan et al., 2023). LLaVA-Surg then introduced Surg-QA, with 102K video-question-answer pairs from public surgical lecture videos, and trained a video-conversational assistant (Li et al., 2024). VidLPRO moved toward a more formal surgical VL pre-training framework with a three-objective loss and GenSurg+ captioned data (Honarmand et al., 2024). Against this background, SurgPub-Video is distinguished by three features: direct use of peer-reviewed journal videos, emphasis on video-level VQA, and a benchmark that explicitly spans 11 specialties (Li et al., 12 Aug 2025).

Its significance is also easier to interpret when contrasted with adjacent dataset-building strategies. SurgVU is a large public robotic training-video resource with tool and task labels, but its supervision is not VQA-oriented (Zia et al., 16 Jan 2025). SurgBench provides a broader pretraining and evaluation framework for surgical video analysis, but its task design is label-centric rather than instruction-centric (Wei et al., 9 Jun 2025). Later work such as SurgAtlas expands scale dramatically to 15,291 videos (2,391 hours) across 18 surgical specialties and over 5,000 procedure types, and explicitly includes open surgery at scale, but it is sourced entirely from publicly available YouTube content (Bellos et al., 24 Jun 2026). This contrast suggests differing design priorities between source reliability and corpus breadth.

Several constraints are either stated or evident in the SurgPub-Video description. The benchmark and dataset are limited to the selected 11 specialties and to peer-reviewed journal sources. Data extraction relies on ASR/transcripts, which can still be noisy before filtering. The model uses a relatively standard architecture built on TinyLLaVA-Video and does not introduce a new temporal transformer beyond the video resampler. Performance is not uniformly superior across all downstream subtasks, as seen in the weaker target accuracy on CholecT50 relative to SurgVLM. The work also depends on the quality and completeness of journal narration and reports (Li et al., 12 Aug 2025).

Even with those constraints, the dataset’s role in the literature is clear. The paper presents SurgPub-Video as a move from static frame classification toward procedural video understanding grounded in peer-reviewed clinical content and expressed through instruction-following supervision. In that formulation, its primary contribution is not only a larger corpus, but a redefinition of what surgical video-language evaluation should measure: temporally grounded recognition, explanation, and procedural reasoning over real surgical clips rather than isolated images (Li et al., 12 Aug 2025).