Scaling Video Pretraining for Surgical Foundation Models

Abstract: Surgical video understanding is essential for computer-assisted interventions, yet existing surgical foundation models remain constrained by limited data scale, procedural diversity, and inconsistent evaluation, often lacking a reproducible training pipeline. We propose SurgRec, a scalable and reproducible pretraining recipe for surgical video understanding, instantiated with two variants: SurgRec-MAE and SurgRec-JEPA. We curate a large multi-source corpus of 10,535 videos and 214.5M frames spanning endoscopy, laparoscopy, cataract, and robotic surgery. Building on this corpus, we develop a unified pretraining pipeline with balanced sampling and standardize a reproducible benchmark across 16 downstream datasets and four clinical domains with consistent data splits. Across extensive comparisons against SSL baselines and vision-LLMs, SurgRec consistently achieves superior performance across downstream datasets. In contrast, VLMs prove unreliable for fine-grained temporal recognition, exhibiting both performance gaps and sensitivity to prompt phrasing. Our work provides a reproducible, scalable foundation for the community to build more general surgical video models. All code, models, and data will be publicly released.

• The paper demonstrates that explicit procedural balancing in video pretraining drives significant performance gains in underrepresented surgical domains.
• SurgRec-MAE and SurgRec-JEPA leverage masked reconstruction and latent prediction to enhance cross-domain generalization over 16 surgical tasks.
• The study underscores that combining large-scale, heterogeneous video datasets with standardized evaluation protocols advances reliable surgical AI development.

Scaling Video Pretraining for Surgical Foundation Models: A Technical Overview

Introduction

The paper "Scaling Video Pretraining for Surgical Foundation Models" (2603.29966) addresses the limitations of current surgical foundation models, specifically the restricted data scale, lack of procedural diversity, and inconsistency in evaluation protocols. The authors introduce SurgRec, a comprehensive pretraining pipeline instantiated with two model variants—SurgRec-MAE (based on masked reconstruction) and SurgRec-JEPA (based on latent prediction)—and systematically analyze the effects of large-scale, procedurally balanced video pretraining in medical contexts. The study centers on three pillars: assembling a heterogeneous surgical video corpus, developing reproducible training and evaluation protocols, and benchmarking against state-of-the-art SSL and VLM architectures.

Dataset Curation and Sampling

A unique aspect of SurgRec is the scale and diversity of the curated video corpus: 10,535 videos comprising 214.5M frames spanning four clinical domains (endoscopy, laparoscopy, cataract, and robotic surgery) and extensive long-tail procedural coverage. The corpus is sourced from 32 public datasets, additional web-crawled content, and private institutional data, maximizing procedural breadth and domain heterogeneity.

To mitigate imbalanced representation across procedures—a key factor hampering previous benchmarks—the authors apply a hierarchical K-means clustering approach to 768-dimensional DINOv3 feature embeddings. The balanced sampling protocol, informed by cluster centroids, ensures representation of underexposed procedures, which directly impacts cross-domain generalization.

Figure 1: Overview of the proposed medical video pretraining and evaluation framework, detailing data curation, balanced sampling, model instantiation (MAE and JEPA), and downstream evaluation.

The distribution of the final training corpus is visualized to illustrate improved procedural and domain balance relative to prior datasets.

Figure 2: Distribution visualization of the pretraining corpus, demonstrating the diversity and representation achieved through corpus curation and balanced sampling.

Methods: Unified Pretraining and Benchmarking Framework

SurgRec-MAE and SurgRec-JEPA are pretrained on the curated corpus using a unified frame preprocessing protocol (resize, random crop) and a mixed-batch sampling schedule combining clinical and unlabeled data. Notably, training balances 59.5% unlabeled and 40.5% clinical videos per batch, with explicit sample balancing to counteract dataset-origin bias.

Evaluation is conducted on a standardized suite of 16 downstream datasets spanning the four surgical domains, employing consistent video-level train/test splits—an advance over previous benchmarks where clip-level leakage and inconsistent partitions confounded cross-study comparison.

Experimental Results

SSL and Domain Pretraining Performance

Both SurgRec variants substantially outperform general-domain SSL baselines (VideoMAE, JEPA, DINOv3) as well as models adapted with additional clinical pretraining. Averaged over all 16 tasks, SurgRec-MAE exceeds VideoMAE by +5.43 macro-averaged accuracy points, with the improvement being particularly pronounced in previously underrepresented domains such as Endoscopy (+9.37) and Robotic Surgery (+5.15). SurgRec-JEPA, while exhibiting more modest aggregate gains, similarly improves upon generalist JEPA in every primary domain except Cataract.

The ablation isolating balanced sampling reveals that improvements in cross-domain generalization cannot be attributed solely to increased data scale; balanced sampling is essential, as omitting it results in dramatic performance drops for robotic and endoscopic tasks.

Vision-LLM Comparison

VLM baselines (Qwen3-VL-8B, LLaVA-NeXT-7B, Qwen2.5-VL-7B), despite strong open-vocabulary image capabilities, display unstable and generally poor recognition on fine-grained surgical temporal tasks, with pronounced sensitivity to prompt phrasing. Across all datasets, SurgRec-MAE demonstrates consistent superiority, frequently outperforming VLMs by margins exceeding 20 percentage points in top-1 accuracy.

Generalization and Domain Robustness

Macro-averaged and worst-domain analysis confirm that SurgRec not only lifts mean performance but also narrows the gap in underrepresented procedural domains. This robustness is attributable to both corpus scale and balanced sampling. The result suggests that, at current scaling, simply adding more unlabeled data without systematic balancing yields diminishing or negative returns in out-of-domain generalization.

Theoretical and Practical Implications

The results have several notable implications. First, the study demonstrates that domain-adaptive, scale-augmented video pretraining pipelines with principled balancing outperform both domain-agnostic SSL models and large-scale VLMs for surgical understanding tasks requiring nuanced temporal reasoning. These findings position dedicated medical video pretraining as a necessary complement to the current trend of VLM scaling.

Practically, SurgRec's reproducible pipeline and standardized splits will catalyze reliable model evaluation and comparison in surgical AI, supporting more rapid iteration and validation for clinical deployment. Methodologically, the research highlights the importance of corpus composition—over raw scale—for robust multi-domain representation.

Limitations and Future Directions

While SurgRec substantially advances state-of-the-art surgical video understanding, domain gaps between web-sourced and clinical videos remain. In addition, challenging real-world phenomena (e.g., rare complications, institutional equipment variation) are under-represented. Future directions include integrating additional institutions for greater clinical diversity, addressing small/noisy subset limitations, and augmenting the evaluation framework for real-world clinical deployments.

Conclusion

SurgRec establishes that robust, generalizable surgical foundation models require both unprecedented data scale and explicit procedural balancing. The comprehensive evaluation demonstrates that domain-specific video pretraining decisively outperforms current VLMs in fine-grained surgical recognition. As the community adopts reproducible, balanced benchmarks and expands clinical pretraining corpora, the reliability and adaptability of AI systems deployed in surgical environments will correspondingly increase.