SurgLIME: Surgical Vision-Language Pre-training

• SurgLIME is a surgical vision-language pre-training framework that leverages confidence-weighted contrastive learning to align video data with LLM-generated narratives.
• It employs a frozen-backbone strategy with LoRA adapters in PL-Stitch and PubMedBERT, preserving critical medical priors while enabling multimodal adaptation.
• The framework addresses the scarcity of expert textual supervision by converting automated surgical narratives into scalable, confidence-modulated training signals.

Searching arXiv for the specified SurgLIME paper and closely related work to ground the article. SurgLIME is a parameter-efficient surgical vision-language pre-training framework designed to learn cross-modal alignments from noisy, LLM-generated narratives while preserving the medical priors of a pre-trained surgical vision foundation model (Che et al., 20 Apr 2026). It is introduced together with LIME, an LLM-Inferred Multimodal Endoscopy dataset derived from open-access surgical videos. The central problem it addresses is not the absence of video data, but the scarcity and cost of expert textual supervision in surgery. SurgLIME therefore treats automatically generated text as a scalable supervision source, while explicitly modeling its unreliability through confidence-weighted contrastive learning and frozen-backbone adaptation (Che et al., 20 Apr 2026).

1. Problem setting and conceptual scope

The motivating premise is that recent self-supervised learning has produced strong surgical vision foundation models, but extending these visual foundations to multimodal reasoning tasks remains bottlenecked by the cost of expert textual annotations (Che et al., 20 Apr 2026). High-quality surgical text typically requires expert clinical knowledge, careful annotation, verification for correctness, and often privacy-sensitive institutional access. This makes manual video-text dataset construction difficult to scale.

SurgLIME is proposed in response to a specific question: whether LLM-generated narratives can serve as a scalable bridge between surgical video and language. The associated paper frames the answer as a two-part system. LIME provides large-scale human-free text supervision, and SurgLIME provides the training framework needed to use that supervision without catastrophically degrading pre-trained medical priors (Che et al., 20 Apr 2026).

A key conceptual distinction follows from this design. Despite its name, SurgLIME is not presented as a local explanation method in the style of classical LIME variants such as s-LIME, Focus-LIME, or SurvLIME; rather, it is a vision-language pre-training method for surgery (Gaudel et al., 2022, Liu et al., 4 Feb 2026, Kovalev et al., 2020). This suggests that in SurgLIME, “LIME” functions primarily as the name of the underlying multimodal dataset and its associated training regime, not as a direct reference to perturbation-based local surrogate explanations.

2. LIME dataset: source material and curation pipeline

LIME stands for an LLM-Inferred Multimodal Endoscopy dataset built from the open-access LEMON dataset (Che et al., 20 Apr 2026). The source collection contains 4194 surgical videos sourced from YouTube, with video durations ranging from several minutes to nearly an hour. The paper describes a multi-stage curation pipeline intended to convert these long-form videos into captioned clips suitable for vision-language pre-training.

The first stage is resolution standardization. Videos are resized with shortest-side resize and center-cropped to a final resolution of 832 × 480. The second stage is shot segmentation, performed with TransNetV2 for automatic shot boundary detection. Long videos are partitioned into semantically coherent shots, and shots shorter than 5 seconds are discarded. The third stage is temporal standardization: each shot is further divided into approximately 5-second clips using a sliding window with window size 5 seconds and stride 2 seconds. According to the paper, this increases sample diversity while preserving temporal context (Che et al., 20 Apr 2026).

The pipeline then applies data pruning. Blurry clips are removed using a Laplacian sharpness filter, and the remaining clips are manually reviewed to remove residual blur. After curation, the surviving clips are annotated automatically by Gemini-2.5-Flash. The prompt instructs the model to behave like a professional medical annotator and describe the field of view and surgery type, instruments, anatomy and tissues, step-by-step actions, and camera perspective and lighting. The output is a dense, detailed paragraph for each clip (Che et al., 20 Apr 2026).

The resulting dataset contains about 54k surgical clips with LLM-generated text. This construction is central to SurgLIME’s significance: it replaces human-written captions with structured synthetic narratives at scale, while leaving open the question of how to learn from such text robustly.

3. Architectural design and preservation of medical priors

SurgLIME is formulated as a parameter-efficient dual-encoder framework for video-text alignment (Che et al., 20 Apr 2026). Its principal architectural choice is to freeze both the base visual encoder and the base text encoder, then learn only low-rank updates via LoRA adapters. The paper presents this as the mechanism for retaining the representational structure of the underlying medical models while still permitting multimodal adaptation.

The visual branch uses PL-Stitch, a surgical foundation model based on ViT-Base and pre-trained on LEMON. Given a clip, the model processes frames to produce frame embeddings

$H_{vid} = \{h_1, h_2, \dots, h_T\}.$

LoRA is injected into the QKV projection matrices of all self-attention blocks. The text branch uses PubMedBERT, chosen for biomedical language competence. Its weights are also frozen, and LoRA is inserted into the query and value matrices. The sentence representation is taken from the $[CLS]$ token,

$h_{text} \in \mathbb{R}^{d_t}.$

This frozen-backbone strategy is paired with modality-specific projection heads and temporal attention pooling. The pooled video representation is computed as

$s_t = W_2 \tanh(W_1 h_t),$

$$a_t = \frac{\exp(s_t)}{\sum_{j=1}^T \exp(s_j)}, \quad \tilde{v} = \sum_{t=1}^T a_t h_t.$$

The projected and normalized video and text embeddings are then

$$z_v = \frac{\text{Proj}_v(\tilde{v})}{\|\text{Proj}_v(\tilde{v})\|_2}, \quad z_t = \frac{\text{Proj}_t(h_{text})}{\|\text{Proj}_t(h_{text})\|_2}.$$

The intended effect is explicit in the paper: the framework keeps the strong surgical visual priors from PL-Stitch, keeps biomedical language structure from PubMedBERT, and learns only small alignment updates through LoRA (Che et al., 20 Apr 2026). A plausible implication is that SurgLIME treats preservation of the vision encoder’s geometry as a first-class design constraint, rather than allowing unrestricted multimodal fine-tuning to rewrite the visual representation space.

4. Confidence estimation and weighted contrastive learning

The central methodological innovation of SurgLIME is its treatment of generated text as noisy supervision rather than trusted annotation (Che et al., 20 Apr 2026). The paper emphasizes that LLM-generated descriptions may contain wrong instruments, wrong anatomical labels, inaccurate procedural steps, hallucinated context, or overconfident but unsupported descriptions. In a standard contrastive pipeline, such errors are especially problematic because each video-text pair is assumed to be a reliable positive pair.

To mitigate this, SurgLIME computes a text confidence score for each generated narrative using PubMedBERT in a masked-token recovery style. For a narrative $S_i$ with tokens $w_1, \dots, w_{L_i}$, each token is masked one at a time and the model predicts the probability of recovering the original token. The confidence score is the average recovery probability:

$$c_i = \frac{1}{L_i} \sum_{k=1}^{L_i} P_{\text{MB}}(w_k \mid S_{i \backslash k}).$$

The score is used as a soft weight in training; uncertain text is down-weighted rather than discarded (Che et al., 20 Apr 2026).

Training uses a bidirectional InfoNCE objective weighted by this confidence. For the visual-to-text direction,

$$\mathcal{L}_{v \to t}^{(i)} = - c_i \log \frac{\exp(z_{v}^{(i)} \cdot z_{t}^{(i)} / \tau)}{\sum_{j=1}^B \exp(z_{v}^{(i)} \cdot z_{t}^{(j)} / \tau)},$$

and the total loss is

$[CLS]$0

where $[CLS]$1 is a learnable temperature and $[CLS]$2 is the symmetric text-to-video loss computed on the transposed similarity matrix (Che et al., 20 Apr 2026).

This design changes the role of noisy language from a binary accept-or-reject signal to a graded supervisory source. The paper does not describe a hard threshold for filtering captions. Instead, reliability is encoded directly into the contrastive objective. This suggests a training regime in which caption quality modulates alignment strength continuously, which is consistent with the paper’s goal of scalability under imperfect supervision.

5. Evaluation protocols and reported empirical results

The evaluation focuses on two standard surgical phase recognition benchmarks: Cholec80 and AutoLaparo (Che et al., 20 Apr 2026). Cholec80 consists of 80 laparoscopic cholecystectomy videos with 7 surgical phases. AutoLaparo consists of 21 laparoscopic hysterectomy videos with 7 phases and is described as more challenging for temporal reasoning.

Two evaluation protocols are used. In zero-shot evaluation, the text encoder generates embeddings for phase descriptions using prompt templates from SurgLaVi, the video embedding is compared against phase text embeddings by cosine similarity, and the highest-similarity phase is predicted. Metrics are Accuracy and F1-score. In linear probing, the vision encoder is frozen, a linear classifier is trained using official splits, and predictions are made frame-by-frame, again using Accuracy and F1-score (Che et al., 20 Apr 2026).

The paper compares SurgLIME against CLIP, SurgVLP, LemonFM, MAE, VideoMAEv2, DINO, iBOT, and PL-Stitch. The reported zero-shot results show a mixed but informative pattern. On AutoLaparo, CLIP achieves 8.0 accuracy / 4.8 F1, SurgVLP 10.0 / 7.2, and SurgLIME 18.1 / 11.2. On Cholec80, CLIP achieves 27.8 / 8.4, SurgVLP 34.7 / 24.4, and SurgLIME 33.0 / 8.7 (Che et al., 20 Apr 2026). The paper interprets this trade-off as evidence that expert lecture transcripts still provide a cleaner alignment signal on Cholec80, while SurgLIME generalizes well in a harder setting such as AutoLaparo.

The linear probing results are used to test whether vision-language pre-training damaged the visual representation. On AutoLaparo, PL-Stitch obtains 79.9 accuracy / 69.0 F1 and SurgLIME 80.7 / 68.8. On Cholec80, PL-Stitch obtains 80.4 / 73.0 and SurgLIME 80.6 / 73.2 (Che et al., 20 Apr 2026). These numbers support the paper’s claim that SurgLIME preserves the visual foundation model very well, with essentially no degradation and slight accuracy gains on both benchmarks.

The ablation on confidence weighting directly targets the method’s central hypothesis. On AutoLaparo zero-shot, an unweighted InfoNCE baseline with $[CLS]$3 yields 13.4 accuracy / 10.1 F1, whereas SurgLIME yields 18.1 accuracy / 11.2 F1 (Che et al., 20 Apr 2026). This result is presented as evidence that confidence weighting materially helps and that down-weighting noisy or hallucinated text is important.

6. Relation to adjacent surgical and LIME-named research

SurgLIME occupies a specific position within a broader landscape of surgical multimodal systems. It addresses pre-training from noisy video-text supervision, whereas other recent surgical systems emphasize interaction, refinement, or task planning at inference time. For example, IR-SIS proposes a language-based surgical segmentation framework with adaptive self-refinement capabilities, using natural language queries, VLM-based detection, segmentation-quality assessment, and clinician-in-the-loop correction (Lou et al., 9 Feb 2026). SurgicalVLM-Agent, in turn, is framed as an interactive AI co-pilot for pituitary surgery that supports conversation, planning, and task execution across segmentation, overlaying, instrument tracking, and VQA (Huang et al., 12 Mar 2025). Compared with these systems, SurgLIME is upstream: it is primarily a representation-learning framework rather than an online agentic or clinician-interactive assistant.

It is also useful to distinguish SurgLIME from the several methods whose names include “LIME” but belong to different methodological families. s-LIME modifies standard LIME to reconcile locality and fidelity in linear explanations by replacing post-hoc kernel weighting with sampling from a continuous locality-controlled distribution (Gaudel et al., 2022). Focus-LIME is a proxy-guided, coarse-to-fine LIME variant for long-context LLM interpretation, aimed at restoring faithful feature-level explanations under attribution dilution (Liu et al., 4 Feb 2026). SurvLIME adapts LIME to survival analysis by fitting a local Cox proportional hazards surrogate to cumulative hazard functions (Kovalev et al., 2020). SurgLIME differs fundamentally from all three: it is not a local explainer, but a surgical vision-language pre-training framework built around a dataset named LIME (Che et al., 20 Apr 2026).

This naming overlap can invite confusion. A precise reading of the literature indicates that SurgLIME should be situated within surgical VLP and multimodal foundation-model adaptation, not within the theory of local surrogate explanation.

7. Limitations, implementation details, and future directions

The paper’s own limitations are tightly coupled to its motivating assumptions. Because LIME is built from LLM-generated narratives, the supervision source is scalable but imperfect. SurgLIME therefore depends on the quality of its automated confidence estimation and on the extent to which sentence-level confidence scores capture medically meaningful reliability (Che et al., 20 Apr 2026). The authors explicitly identify token-level confidence weighting as a future direction, implying that sentence-level weighting may be too coarse for some error modes.

The implementation details reported in the paper are specific: the system is implemented in PyTorch; the vision backbone is ViT-Base initialized with PL-Stitch; the text backbone is PubMedBERT; LoRA uses rank $[CLS]$4 and scaling $[CLS]$5; training runs for 10 epochs with AdamW, learning rate $[CLS]$6, cosine decay, and temporal window 8 frames (Che et al., 20 Apr 2026). For linear probing, the learned LoRA weights are merged back into the frozen ViT backbone before training the classifier.

The future directions listed by the authors include token-level confidence weighting, iterative self-correction for LLM-generated supervision, and extension to harder downstream tasks such as surgical captioning and robotic action generation (Che et al., 20 Apr 2026). These proposals are consistent with the framework’s underlying premise: that multimodal surgical learning can be scaled through synthetic language, provided the training procedure remains robust to hallucination and preserves the geometry of the surgical visual encoder.

Taken together, SurgLIME defines a specific research program within surgical AI: scalable vision-language pre-training from human-free narratives, constrained by the need to preserve visual medical priors rather than overwrite them. Dataset, code, and models are publicly available through the project repository reported in the paper (Che et al., 20 Apr 2026).