SurgLLaVA-Video: Temporal Surgical VLM

• SurgLLaVA-Video is a video-native surgical vision-language model that integrates temporal context to capture unfolding procedures.
• It leverages a novel video resampler to condense multi-frame input, optimizing both video-level and frame-level surgical analysis.
• Trained on the extensive SurgPub-Video dataset, the model significantly outperforms frame-centric systems on surgical VQA and related tasks.

SurgLLaVA-Video is a specialized surgical video vision-LLM introduced together with the SurgPub-Video dataset and a video-level surgical visual question answering benchmark to address the frame-centric bias of earlier surgical VLMs. Built upon TinyLLaVA-Video, it is designed for temporal surgical content rather than isolated frames, supports both video-level and frame-level inputs, and is trained on a large corpus of peer-reviewed surgical videos with question-answer supervision. The central premise is that surgical intelligence requires video-native supervision because tools, anatomy, and procedural steps unfold over time; accordingly, SurgLLaVA-Video is evaluated not only on video-level VQA but also on surgical action recognition, skill assessment, and triplet recognition (Li et al., 12 Aug 2025).

1. Problem formulation and design rationale

SurgLLaVA-Video was developed in response to four limitations identified in prior surgical VLMs: they are frame-centric, they lack temporal modeling, they are trained on weak or narrow supervision, and they do not cover enough procedures or specialties to support broad surgical understanding. The paper positions earlier systems such as SurgVLM, SurgicalGPT, EndoChat, and earlier surgical VQA systems as primarily image- or frame-based, which makes them less suitable for modeling workflow, motion, and the sequential structure of surgery. Compared with general-purpose VLMs such as LLaVA-1.5, InternVL3, Qwen2.5-VL, GPT-4o, and Gemini 2.0 Flash, SurgLLaVA-Video is specialized for surgical content and trained on domain-specific surgery videos and QA pairs (Li et al., 12 Aug 2025).

The model is explicitly described as video-native rather than frame-native. Its objective is not only recognition of instruments or anatomy in individual frames, but also capture of temporal relationships across clips. This is why the paper emphasizes a much larger and more diverse surgical video corpus, optimization for both video-level and frame-level input, and a new benchmark that evaluates video-level reasoning across multiple surgical tasks and specialties. A plausible implication is that SurgLLaVA-Video treats temporal continuity as a first-class source of supervision rather than as an incidental extension of frame-based perception.

2. Architectural formulation

SurgLLaVA-Video follows the TinyLLaVA-Video architecture and comprises three components: a vision encoder, a video group resampler, and an LLM. A video clip is represented as

$X \in \mathbb{R}^{T \times H \times W \times 3},$

where $T$ is the number of frames and $H, W$ are frame height and width. The vision encoder extracts per-frame patch features

$Z \in \mathbb{R}^{T \times N \times D},$

with $N$ denoting the number of patch tokens per frame and $D$ the feature dimension. These features are then reshaped into

$\hat{Z} \in \mathbb{R}^{(T \times N) \times D}.$

The architectural distinction from standard LLaVA-style models lies in the video resampler $\mathbf{P}_{\theta}$, which projects dense video features into a fixed number of learnable queries,

$Q \in \mathbb{R}^{L \times D}.$

The resampler divides the flattened visual features and queries into sub-embeddings,

$$\{\hat{Z}_1, \hat{Z}_2, \dots, \hat{Z}_L\}, \quad \{Q_1, Q_2, \dots, Q_L\},$$

and applies cross-attention between each visual chunk $T$0 and its query $T$1, yielding outputs $T$2. These outputs are concatenated into a compact visual representation,

$T$3

This mechanism is intended to keep computation manageable, preserve temporal relationships, and summarize a multi-frame clip into a small set of tokens for the LLM (Li et al., 12 Aug 2025).

The LLM then consumes the visual embeddings $T$4 together with a surgical text prompt $T$5 and generates an answer through joint multimodal reasoning. Training initializes the system from pre-trained TinyLLaVA-Video, keeps the vision encoder frozen, and fully fine-tunes the LLM and video resampler on SurgPub-Video VQA pairs. The paper also emphasizes that this design supports frame-level understanding as well as video-level reasoning because the model can process multi-frame clips through the resampler while retaining individual frame-level visual features from the vision encoder.

3. SurgPub-Video dataset

SurgPub-Video is the training corpus underlying SurgLLaVA-Video and is presented as a comprehensive surgical video dataset for enhanced surgical intelligence in vision-LLMs. The dataset contains 3,538 raw surgical videos, 10,926 surgical video clips, 48,520 VQA pairs, and 25 million annotated frames. Average clip duration is 29.85 seconds, average original video length is 323.5 seconds, resolution is $T$6, and frame rate is 30 Hz. Its reported semantic breadth includes 11 surgical specialties, 75 kinds of surgery, 1,823 anatomical structures, 40 surgical instruments, and 1,290 unique procedures (Li et al., 12 Aug 2025).

A defining property of SurgPub-Video is its provenance. Videos were crawled from 25 peer-reviewed medical journals, and each video is associated with a fully peer-reviewed article. The dataset is thus framed as clinically reliable and authentic, in contrast to YouTube- or social-media-derived surgical datasets. The benchmark tables indicate specialties such as breast, cardiac, endoscopic, esophageal, liver, lung, mediastinal, orthopedic, thoracic, urological, and vascular.

Curation proceeds through a multi-stage pipeline. Raw video collection gathers surgery videos accompanying peer-reviewed articles. Audio-guided preprocessing uses Whisper to generate transcripts, segments long videos into timestamped fragments, filters non-surgical content, and merges semantically related fragments. Fine-grained concept extraction classifies clips into five concept categories: instruments, procedural steps, anatomical structures, treatment planning, and general surgical knowledge. QA generation and refinement then uses an LLM to generate open-ended and multiple-choice QA pairs grounded in narrations, causal explanations, and clinical rationales, with distractors crafted for MCQ answers and final review and refinement by human medical experts. This curation strategy suggests an attempt to combine large-scale automatic generation with explicit clinical vetting.

4. Benchmark construction and experimental protocol

The SurgPub-Video benchmark is a video-level surgical VQA evaluation set derived from the larger dataset. It samples 20% of VQA pairs from 705 videos and yields a final benchmark size of 3,337 samples. To balance the evaluation set, cardiac surgery samples were reduced to 38.6%, vascular surgeries to 11.9%, and mediastinal surgeries to 7.6%. The benchmark covers five VQA tasks: instrument recognition, anatomical structure recognition, procedural step identification, surgical planning, and general surgical knowledge. It spans 11 specialties and is explicitly designed for video-level evaluation, unlike prior surgical VQA datasets that were mostly image-level (Li et al., 12 Aug 2025).

Benchmark reporting uses overall accuracy, specialty-specific accuracy, and task-specific accuracy. The experimental setup evaluates seven models on this benchmark: LLaVA-1.5, InternVL3, Qwen2.5-VL-Instruct, GPT-4o-2024-0806, Qwen2.5-Max, Gemini 2.0 Flash, and SurgLLaVA-Video. For downstream tasks, the study follows the SurgVLM evaluation setup and compares six general models, SurgVLM, and SurgLLaVA-Video on surgical action recognition, surgical skill assessment, and triplet recognition.

Implementation details reported in the paper include GPT-4o as an agent for dataset generation, training on 4 NVIDIA A40 GPUs, inference temperature of 0.2, and multiple-choice exact-match accuracy for evaluation. These details are important because the principal benchmark is formulated as structured VQA rather than free-form open-ended generation, which distinguishes it from conversational surgical LVLM evaluation protocols.

5. Empirical performance and ablation findings

On the SurgPub-Video benchmark, SurgLLaVA-Video achieves 82.84% overall accuracy. Among the listed general-purpose baselines, Qwen 2.5 Max reaches 73.56%, GPT-4o reaches 66.67%, and Gemini 2.0 Flash reaches 73.33%. The paper reports that SurgLLaVA-Video improves over GPT-4o by 16.17 percentage points overall. Specialty-specific results include 82.14 on breast, 84.38 on cardiac, 61.38 on endoscopic, 89.69 on esophageal, 74.19 on liver, 82.84 on lung, 79.76 on mediastinal, 78.24 on orthopedic, 81.74 on thoracic, 80.23 on urological, and 82.73 on vascular cases. The paper further notes improvements greater than 10% in 10 of 11 organs, cardiac surgery improved by more than 16% over GPT-4o, and esophageal surgery reached 89.69%, about 11.56% above the second best result. The broader conclusion presented is that a 3B-parameter surgical VLM outperformed much larger 72B–78B models and closed commercial systems on this benchmark (Li et al., 12 Aug 2025).

The downstream-task results extend the model’s scope beyond VQA. On SAR-RARP surgical action recognition, SurgLLaVA-Video obtains 65.65 accuracy versus 42.90 for SurgVLM, with reported gains of more than 22% accuracy, more than 23% precision, and +14% Jaccard. On Endoscapes2023 critical view of safety assessment, the model achieves 58.88% average balanced accuracy, compared with 53.50% for InternVL3 and 51.40% for SurgVLM, corresponding to gains of +5.38% and +7.48%, respectively. On CholecT50 triplet recognition, SurgLLaVA-Video achieves 85.93% instrument accuracy, 63.29% verb accuracy, 47.12% target accuracy, and 39.30% triplet accuracy; corresponding mAP values are 58.98% for instruments, 42.75% for verbs, 19.93% for targets, and 10.61% for triplets. Relative to SurgVLM, instrument accuracy improves by 23.73%, verb accuracy by 44.65%, and triplet accuracy by 26.78%, although SurgVLM still slightly leads in target accuracy while SurgLLaVA-Video remains competitive in target mAP.

Two ablations are reported. The video-length ablation evaluates 8, 16, 24, and 32 input frames and finds that performance peaks at 16 frames with overall accuracy of 80.82%; too few frames lose temporal context, while too many frames introduce redundancy or noise. The dataset-size ablation trains with 25%, 50%, 75%, and 100% of SurgPub-Video and finds that performance increases nearly linearly with more data, with the full dataset yielding the best overall result. Anatomical recognition benefits strongly from more data, whereas planning tends to saturate earlier. Together, these ablations support the claim that SurgLLaVA-Video depends both on a moderate temporal window and on large-scale, diverse supervision.

6. Position within the surgical video-language literature and constraints

SurgLLaVA-Video belongs to a broader transition from surgical image-language systems to temporally grounded surgical video-LLMs. An important precursor is "Surgical-LLaVA" (Jin et al., 2024), which adapts a LLaVA-style architecture with Vicuna, CLIP ViT-L/14, and a projection layer $T$7 to map surgical visual features into language space, and fine-tunes on surgical instruction-following data. Surgical-LLaVA demonstrated strong gains on Cholec80-VQA, EndoVis18-VQA, and PSI-AVA-VQA, but its video pathway is described at a relatively high level, relying on temporal and spatial average pooling of frame-level features rather than a dedicated video resampler. Relative to that earlier architecture, SurgLLaVA-Video can be understood as a more explicitly video-native formulation centered on multi-frame compression and video-level supervision.

A closely related line of work is "SurgViVQA: Temporally-Grounded Video Question Answering for Surgical Scene Understanding" (Drago et al., 5 Nov 2025), which also argues that surgical question answering should move beyond isolated frames. SurgViVQA uses a Masked Video–Text Encoder with VideoMAE, BLIP cross-attention, and a LoRA-adapted causal LLM to reason over short clips, and introduces REAL-Colon-VQA with motion-related questions and out-of-template perturbations. This suggests a convergence between two design philosophies: one emphasizing specialized video-language architectures and large peer-reviewed datasets, the other emphasizing temporally grounded surgical VideoQA with explicit robustness tests.

The same shift toward temporal surgical understanding also appears in downstream embodied applications. "Strategy-Supervised Autonomous Laparoscopic Camera Control via Event-Driven Graph Mining" (Zhou et al., 24 Feb 2026) couples high-level VLM inference with low-level IBVS-RCM control, using a fine-tuned Qwen2.5-VL 7B model to predict camera-handling strategy and discrete motion directions from live laparoscopic video. That work is more control-oriented than SurgLLaVA-Video, but it indicates that surgical video-LLMs are beginning to serve not only descriptive or QA functions but also structured decision support and robotic execution.

The constraints of SurgLLaVA-Video are also explicit or implied in the source paper. The model depends on peer-reviewed journal videos, which improve quality but may not cover every surgery type or edge-case scenario. The benchmark is built from a subset of the full dataset, so some specialty and task imbalance remains. Although the model supports video understanding, it still relies on a compact resampler and a fixed number of frames, so very long-range context may not be fully captured. Training and evaluation remain centered on VQA-style prompts and multiple-choice tasks, which do not exhaust the space of surgical reasoning. Within those constraints, SurgLLaVA-Video provides a concrete case for high-quality, peer-reviewed, video-native surgical data as the basis for temporal surgical intelligence (Li et al., 12 Aug 2025).