Can LLM-Generated Text Empower Surgical Vision-Language Pre-training?

Abstract: Recent advancements in self-supervised learning have led to powerful surgical vision encoders capable of spatiotemporal understanding. However, extending these visual foundations to multi-modal reasoning tasks is severely bottlenecked by the prohibitive cost of expert textual annotations. To overcome this scalability limitation, we introduce \textbf{LIME}, a large-scale multi-modal dataset derived from open-access surgical videos using human-free, LLM-generated narratives. While LIME offers immense scalability, unverified generated texts may contain errors, including hallucinations, that could potentially lead to catastrophically degraded pre-trained medical priors in standard contrastive pipelines. To mitigate this, we propose \textbf{SurgLIME}, a parameter-efficient Vision-Language Pre-training (VLP) framework designed to learn reliable cross-modal alignments using noisy narratives. SurgLIME preserves foundational medical priors using a LoRA-adapted dual-encoder architecture and introduces an automated confidence estimation mechanism that dynamically down-weights uncertain text during contrastive alignment. Evaluations on the AutoLaparo and Cholec80 benchmarks show that SurgLIME achieves competitive zero-shot cross-modal alignment while preserving the robust linear probing performance of the visual foundation model. Dataset, code, and models are publicly available at \href{https://github.com/visurg-ai/SurgLIME}{https://github.com/visurg-ai/SurgLIME}.

• The paper demonstrates that LLM-generated narratives can replace costly expert annotations in surgical vision-language pre-training.
• It introduces the LIME dataset and SurgLIME framework, incorporating confidence-weighted contrastive loss and parameter-efficient dual encoders.
• Empirical results on benchmarks like AutoLaparo and Cholec80 underscore improved zero-shot performance and preserved visual representation.

Vision-Language Pre-training for Surgery: Leveraging LLM-Generated Narratives

Introduction

The integration of multimodal learning into surgical intelligence is limited by the substantial annotation requirements for high-quality textual supervision, constraining the ability of models to perform cross-modal reasoning tasks in this domain. "Can LLM-Generated Text Empower Surgical Vision-Language Pre-training?" (2604.18134) systematically addresses the feasibility of replacing expensive expert-annotated text with scalable, LLM-generated narratives, examining both the utility and risks associated with noisy, potentially hallucinated supervision in surgical vision-language learning.

Construction of the LIME Dataset

To resolve the annotation bottleneck, the authors introduce LIME, a large-scale multimodal surgical dataset obtained from open-access YouTube surgical videos in the LEMON collection, systematically processed to yield clip-level video-text pairs with dense semantic annotations.

The construction pipeline begins by standardizing the raw videos in terms of spatial resolution. TranNetV2 is utilized for semantic shot boundary detection, followed by a sliding window approach to produce 5-second clips. Quality filtering is applied using a Laplacian-based sharpness metric and manual curation to eliminate blurred or ambiguous content. Crucially, domain-specific captions for each filtered clip are generated by prompting Gemini-2.5-Flash, an LLM augmented with surgical context, to yield structured textual narratives that describe procedural steps, instruments, anatomical structures, and technical factors.

Figure 1: LIME dataset curation pipeline: standardization, temporal segmentation, automated and manual filtering, and LLM-based narrative generation yield high-quality, multimodal pre-training pairs.

This automated pipeline produces over 54,000 densely annotated surgical clips, marking an order-of-magnitude increase in multimodal supervision within the surgical AI literature.

The SurgLIME Framework

Architecture and Parameter-efficient Dual Encoders

SurgLIME is a parameter-efficient Vision-Language Pre-training (VLP) framework targeting robust cross-modal alignment under noisy annotation. The system uses a dual-encoder architecture with ViT-Base (PL-Stitch) as the vision encoder and PubMedBERT for text. Both encoders remain frozen to preserve rich, domain-specific priors from visual and biomedical pre-training, with modality-bridging achieved through the injection of LoRA modules for lightweight, low-rank adaptation. The vision encoder processes $T$ frames per clip, aggregates framewise representations with a learnable temporal attention pooling module, and projects clip-level features into a shared metric space via a two-layer MLP head. Similarly, the text encoder maps surgical narratives into the common space. All projections are L2-normalized for spherical contrastive learning.

Confidence-weighted Contrastive Alignment

To explicitly handle the noise and hallucination inherent in LLM-derived supervision, SurgLIME introduces an automated confidence scoring scheme. PubMedBERT is repurposed in masked language modeling mode to score each LLM-generated narrative: for every token, its recovery probability given the sentence context is computed, and the average across tokens yields a soft textual confidence estimate.

This score dynamically modulates the contribution of each video-text pair in the bidirectional InfoNCE contrastive loss, down-weighting uncertain or hallucinated pairs. Optimization follows a learning rate schedule that aggressively updates randomly initialized projection heads while conservatively adapting the LoRA weights, thus preventing catastrophic forgetting or distortion of underlying medical representations.

Empirical Assessment

The framework is evaluated on two canonical surgical phase recognition benchmarks—AutoLaparo and Cholec80—in both zero-shot and linear probing settings.

• Zero-shot Evaluation: The pretrained vision and text encoders are probed for direct cross-modal alignment without additional finetuning, where visual features from test videos are matched to textual descriptions of surgical phases. SurgLIME demonstrates significant improvement over baseline CLIP (ViT-B/16) in both datasets and surpasses SurgVLP on AutoLaparo by a strong margin, indicating that LLM-generated supervision, when managed with noise-aware weighting, generates robust semantic alignment.
• Linear Probing: The integrity of vision encoder priors post-alignment is evaluated by keeping the encoder frozen and training a linear classifier for phase discrimination. Results indicate that SurgLIME preserves and marginally augments the discriminative power of the underlying PL-Stitch backbone, substantiating the claim that cross-modal learning can be conducted without detrimental drift to the base visual model.

A targeted ablation confirms that confidence-weighted loss is critical: standard contrastive training with uniform contributions from all samples yields substantially lower zero-shot performance, reaffirming the need for automated noise suppression when learning from unverified LLM narratives.

Implications and Future Directions

This work provides concrete evidence that surgical vision-language pre-training can be rendered scalable via LLM-generated narratives, bridging the domain gap for multimodal learning in contexts where expert annotation is infeasible. The proposed pipeline is both efficient and robust, leveraging domain-adapted encoders and parameter-efficient adaptation strategies to absorb noisy textual signals without representation collapse or loss of underlying visual priors.

The theoretical implications extend to the general use of synthetic, noisy, or weak supervision in high-stakes domains, especially where the hallucination rate of LLMs may propagate anomalous or harmful domain priors. The architectural decisions—frozen encoders, LoRA adapters, confidence-weighted losses—provide a template for future cross-modal foundation models where reliability and interpretability of alignment are paramount.

Practically, the demonstrated preservation and even enhancement of linear probing performance underline the viability of large-scale, human-free annotation for downstream deployment in surgical AI systems capable of semantic indexing, retrieval, and open-set reasoning.

Conclusion

The investigation rigorously demonstrates that LLM-generated textual supervision, when filtered with domain-adapted confidence estimation and incorporated via parameter-efficient cross-modal alignment strategies, can substantially accelerate the development of foundation vision-LLMs for surgery (2604.18134). The LIME dataset and SurgLIME framework constitute valuable baselines for follow-up studies, including those targeting token-level noise modeling, iterative label correction, and extension to actionable reasoning tasks like surgical captioning or intraoperative robotic assistance. The methodology generalized herein is relevant for multimodal foundation work throughout safety-critical expert domains.