PeskaVLP: Surgical Video-Language Pretraining

• PeskaVLP is a surgical video–language model that uses a dual-encoder CLIP-style framework combined with hierarchical language augmentation to align multimodal surgical data.
• It employs dynamic time warping and clip-level visual self-supervision to capture fine-grained temporal patterns and overcome the scarcity and noise in surgical video–text pairs.
• PeskaVLP serves as a foundation for downstream tasks such as Critical View of Safety recognition, demonstrating improved performance when adapted via CVS-AdaptNet.

Searching arXiv for the specified papers and related PeskaVLP work. PeskaVLP, short for Procedure-Encoded Surgical Knowledge-Augmented Video-Language Pretraining, is a surgical video–language foundation model designed to learn aligned multimodal representations from surgical lecture videos by coupling CLIP-style dual-encoder pretraining with hierarchical language augmentation and procedure-aware alignment objectives (Yuan et al., 2024). It was introduced to address three recurrent constraints in surgical VLP: the domain gap between surgical and natural imagery, the scarcity and noisiness of paired video–text data, and the strong temporal regularities of surgical workflows that are not captured by standard contrastive objectives. Subsequent work used PeskaVLP as the initialization backbone for CVS-AdaptNet, a multi-label adaptation strategy for fine-grained Critical View of Safety recognition in laparoscopic cholecystectomy, where aligned image–text embeddings are leveraged with positive and negative criterion prompts (Baby et al., 7 Jul 2025).

1. Research context

PeskaVLP was proposed in a setting where standard video–language pretraining methods were considered insufficient for surgical data. The stated motivation is threefold: surgical imagery is highly specialized, multimodal surgical corpora are limited in scale, and the language attached to lecture videos is often fragmentary, noisy, or semantically weak. The pretraining corpus discussed for PeskaVLP contains approximately $10^3$ lecture videos rather than the $10^6$-scale corpora common in the general domain, and the videos themselves are long, narrow-field, and dominated by fine-grained tool–anatomy interactions and specialized jargon (Yuan et al., 2024).

A second premise is that surgical procedure structure is not incidental metadata but a supervisory signal. The model is therefore not framed as a generic CLIP variant applied to endoscopic frames. Instead, it is explicitly “procedure-aware”: it uses parent–child relations across clip, phase, and video hierarchies, and it enforces monotonic video–text alignment at longer temporal scales through Dynamic Time Warping. This design distinguishes PeskaVLP from multimodal baselines such as CLIP trained on SVL, SurgVLP, and HecVL, which are treated as related surgical or adapted multimodal baselines rather than equivalents.

The later CVS study further clarifies the role of PeskaVLP in downstream adaptation. In that setting, zero-shot multimodal transfer was inadequate for fine-grained, multi-label Critical View of Safety recognition: zero-shot mAP was 26.64 for PeskaVLP, 26.18 for SurgVLP, and 28.46 for HecVL, against a random baseline of 19. This establishes a distinction between general-purpose surgical multimodal pretraining and task-specific adaptation for clinically constrained recognition problems (Baby et al., 7 Jul 2025).

2. Model architecture and representation space

PeskaVLP is a dual-encoder framework in the CLIP style. Its visual encoder $f$ is a ResNet-50 applied per frame, with ImageNet initialization reported as the most stable choice for the surgical domain. A ViT-B/16 was tested but was described as more sensitive to initialization and worse under the limited-data regime. Its text encoder $g$ is ClinicalBERT (BioClinicalBERT), chosen to better handle medical terminology than general BERT (Yuan et al., 2024).

Both encoders produce $L_2$-normalized embeddings, and a linear projection head maps visual and textual features into a shared embedding space. Similarities are computed as cosine similarities via dot products between normalized vectors. The model does not introduce an explicit temporal module such as an RNN or attention stack across time. Instead, temporal structure is injected through sampling at multiple hierarchies and through DTW-based alignment losses. For video segments, frames may be treated individually for DTW alignment or average-pooled for retrieval and classification evaluation.

This shared embedding space is the critical interface later exploited by CVS-AdaptNet. In the adaptation study, PeskaVLP is characterized as comprising a vision encoder and a text encoder trained to produce compatible embeddings in a joint image–text space. That study emphasizes the CLIP-like nature of the alignment, including a temperature parameter $\tau$ that controls similarity sharpness, and uses the pretrained PeskaVLP encoders as initialization for fine-tuning on Endoscapes-CVS201 (Baby et al., 7 Jul 2025).

3. Hierarchical knowledge augmentation

A defining element of PeskaVLP is hierarchical knowledge augmentation with LLMs. The objective is to mitigate textual information loss at three levels of supervision: narration, keystep, and abstract. The augmentation is organized around the hierarchy procedure $\rightarrow$ phase $\rightarrow$ step $\rightarrow$ instrument/organ/action, and it uses role-specific prompting to elicit richer supervisory text (Yuan et al., 2024).

At the narration level, the LLM is assigned a “recipe” role. Given a video title, it generates a pseudo step list covering the canonical ordered steps of the procedure. Each ASR narration snippet is then embedded with ClinicalBERT and assigned to the most similar pseudo step by cosine similarity. This serves two purposes: implicit typo correction and semantic enrichment of fragmentary narration. At the keystep level, the LLM is assigned a “dictionary” role, expanding terse labels such as “clipping and cutting” into descriptions that include anatomy, instruments, and events. At the abstract level, the LLM is assigned a “summarizer” role, compressing long abstracts by retaining surgical type, key anatomies, and goals while removing noise such as author and citation metadata.

The textual preprocessing stack is correspondingly domain specific. ASR cleaning uses a 51,640-word vocabulary built from textbooks, phase definitions, and curated terminology, together with a spell-checker constrained by Levenshtein distance $\leq 2$ edits and frequency priors. During training, original and augmented text are randomly mixed, with $10^6$0 for using augmented text. This is intended to reduce overfitting to synthetic reformulations while still densifying supervision.

The knowledge augmentation mechanism also anticipates later findings from CVS-AdaptNet. In the CVS setting, prompt quality directly affects performance: random text with ResNet50+BioClinicalBERT yielded mAP 45.09, generic surgical text 49.31, fixed class prompts with PeskaVLP 54.26, detailed anatomical text 53.20, and medium-detailed, discriminative text with PeskaVLP 57.54 (Baby et al., 7 Jul 2025). A plausible implication is that PeskaVLP’s pretraining benefits not merely from having more text, but from obtaining clinically discriminative text that improves cross-modal alignment.

4. Procedure-aware objectives and optimization

PeskaVLP combines cross-modal contrastive learning, clip-level visual self-supervision, and DTW-based procedural alignment into a hierarchy-specific objective called LecNCE (Yuan et al., 2024). At all levels, the core similarity is cosine similarity $10^6$1 between normalized video and text embeddings, with temperature $10^6$2 following standard CLIP practice.

The symmetric cross-modal contrastive loss is

$10^6$3

where $10^6$4 and $10^6$5 are standard InfoNCE terms over in-batch video–text pairs. At the clip level, PeskaVLP supplements language supervision with a visual self-supervision branch. Two stochastic augmentations $10^6$6 and $10^6$7 of the same clip are treated as a positive pair, producing a clip-level visual loss $10^6$8. The clip objective is then

$10^6$9

At the phase and video levels, PeskaVLP adds a procedure-aware DTW term. Given video-frame embeddings and child text-sequence embeddings, it constructs a cost matrix $f$0 with entries

$f$1

then computes a monotonic alignment path via differentiable soft-DTW. A reversed child sequence $f$2 is used as a procedure-aware hard negative, producing a hinge-style loss

$f$3

The phase/video objective is

$f$4

and the total training objective is

$f$5

The training schedule alternates mini-batch blocks across temporal hierarchies: 25 batches at clip level, 15 at phase level, and 115 at video level. Sampling uses 4, 16, and 64 frames for clip-, phase-, and video-level batches respectively, with batch sizes 120, 80, and 25. Optimization uses AdamW with base learning rate $f$6, training for 200 epochs on 4 $f$7 80 GB NVIDIA A100 GPUs. The DTW parameters are $f$8, $f$9, and $g$0. No external memory bank is used; hard negatives arise from in-batch mismatches and sequence reversal.

5. Pretraining corpus and empirical transfer

PeskaVLP is pretrained on the SVL dataset, described as 1,326 lecture videos spanning approximately 300 hours and yielding about 26k clip–narration pairs via AWS and Whisper ASR (Yuan et al., 2024). Metadata mined from public platforms supply keysteps and abstracts. After requiring parent–child correspondences, 1,007 videos remain for phase-level training and 920 for video-level training. Retrieval evaluation uses SVL-Retrieval holdout splits with 537 clip–narration pairs, 746 phase–keystep pairs, and 86 video–abstract pairs.

The reported empirical results position PeskaVLP as the strongest model among the compared multimodal baselines for zero-shot surgical scene understanding. Averaged across Cholec80, AutoLaparo, StrasBypass70, and BernBypass70, zero-shot phase recognition reaches 41.0 Accuracy and 27.1 F1 for PeskaVLP, compared with 28.7 and 19.3 for HecVL, 21.6 and 12.9 for CLIP trained on SVL, 19.9 and 8.0 for CLIP-400M, and 19.6 and 13.8 for SurgVLP. The per-dataset breakdown for PeskaVLP is 45.1/34.2 on Cholec80, 26.5/23.6 on AutoLaparo, 46.7/28.6 on StrasBypass70, and 45.7/22.6 on BernBypass70.

Cross-modal retrieval results show the same pattern. At $g$1, PeskaVLP obtains 23.3/21.3 for clip–narration text$g$2image / image$g$3text retrieval, 35.4/24.8 for phase–keystep, and 85.9/81.1 for video–abstract, all higher than the corresponding HecVL results of 17.2/16.2, 21.3/14.8, and 82.3/71.8. In few-shot linear probing with 10% of training videos, PeskaVLP reaches 61.9/50.6 on Cholec80, 53.1/36.8 on AutoLaparo, 63.8/50.4 on StrasBypass70, and 62.9/32.7 on BernBypass70.

Ablation results attribute these gains to the combination of knowledge augmentation, clip-level visual self-supervision, and DTW-based procedure-aware loss. Starting from HecVL without these components, adding knowledge augmentation improves zero-shot and linear probing across datasets; adding clip-level visual SSL further boosts both; and adding the DTW-based loss gives the best overall performance on most datasets, with a slight tradeoff on AutoLaparo zero-shot.

Task	Setting	Result
Zero-shot phase recognition	PeskaVLP average over four datasets	41.0 Accuracy / 27.1 F1
Cross-modal retrieval	Phase–Keystep $g$4	35.4 / 24.8
CVS recognition	CVS-AdaptNet, standard inference, PeskaVLP init	57.6±0.2 mAP
CVS recognition	ResNet50 image-only baseline	51.5 mAP
CVS recognition upper baseline	SurgLatentGraph (LG-CVS)	67.3 mAP

The reproducibility record contains a notable ambiguity regarding software release. The abstract of the PeskaVLP paper states that code is available at https://github.com/CAMMA-public/SurgVLP, whereas the detailed reproducibility notes state that the paper does not provide a public URL (Yuan et al., 2024). This is best treated as a documentation inconsistency rather than a technical property of the model.

6. Adaptation to Critical View of Safety recognition

In later work, PeskaVLP serves as the multimodal initialization for CVS-AdaptNet, which reformulates adaptation to Critical View of Safety recognition as a multi-label problem rather than a multi-class problem (Baby et al., 7 Jul 2025). The task is defined on Endoscapes-CVS201, which contains 11,090 frames of laparoscopic cholecystectomy with three binary CVS criteria per frame, split into 6,960 train, 2,331 validation, and 1,799 test frames. The dataset reflects clinical ambiguity, with three-expert Cohen’s $g$5, and annotator confidence scores are rounded to 0/1 for comparison with image-only baselines.

The three aligned criterion texts are the Strasberg criteria used as textual anchors:

1. “the cystic duct and the cystic artery, connected to the gallbladder”
2. “a hepatocystic triangle cleared from fat and connective tissues”
3. “the lower part of the gallbladder separated from the liver bed”

CVS-AdaptNet augments these anchors with diverse positive and negative paraphrases generated by an LLM. Positive prompts describe presence of a criterion, such as “Seeing two structures cystic duct and cystic artery in the image” for Criterion 1; negative prompts describe absence, such as “A general medical image not showing cystic duct or cystic artery.” For each image–criterion pair, the method samples a prompt from the positive set if the label is 1 and from the negative set if the label is 0, then aligns image and text embeddings through a KL-based contrastive loss over the batch. The adaptation uses the PeskaVLP vision encoder $g$6 and text encoder $g$7, with embeddings

$g$8

and cosine similarity

$g$9

For each criterion, the loss is

$L_2$0

and the total loss sums over the three criteria and averages over the batch:

$L_2$1

The rationale for this KL-based contrastive formulation is that CVS labels are subjective and noisy, and multiple prompts per criterion may legitimately match an image. The study explicitly contrasts this with binary cross-entropy, which assumes independent labels, and strict InfoNCE, which assumes a 1:$L_2$2 matching structure.

Fine-tuning is joint rather than frozen: both the PeskaVLP vision and text encoders are updated on Endoscapes-CVS201. Training uses Adam, initial learning rate $L_2$3 with cosine annealing, a separate $L_2$4 learning rate of $L_2$5, batch size 64, 20 epochs, image size $L_2$6 with center crop 224, RandAugment, and a single RTX A5500 for approximately 4 hours. The temperature is learnable and initialized at 2.6593. Implementation is in PyTorch, and the CVS-AdaptNet code is provided at https://github.com/CAMMA-public/CVS-AdaptNet.

Three inference strategies are reported. Standard inference uses one clinician-selected prompt per criterion, distinct from the training prompts, and computes criterion-wise probabilities with $L_2$7. Positive–Negative inference contrasts one positive and one negative prompt per criterion with a 2-way softmax. Multi-class inference defines eight text descriptions corresponding to all combinations of the three criteria, applies a softmax over the eight descriptions, and then aggregates probabilities back to the three binary criteria. Reported evaluation uses AP and mAP from ranked continuous scores rather than a fixed threshold.

The quantitative outcome is that CVS-AdaptNet adapted from PeskaVLP reaches 57.6$L_2$80.2 mAP with standard inference, with per-criterion AP of 54.5$L_2$90.7, 55.9$\tau$00.3, and 62.4$\tau$10.5. Positive–Negative inference reaches 56.8$\tau$20.4 mAP, and multi-class inference 57.6$\tau$30.3 mAP. The principal image-only baseline, ResNet50 initialized from ImageNet, reaches 51.5 mAP; a ResNet50-MoCov2 baseline pretrained with self-supervision on surgical videos reaches 57.4 mAP; and the vision-only encoders extracted from multimodal models reach 51.4$\tau$41.5 for SurgVLP-vision, 50.0$\tau$50.5 for HecVL-vision, and 48.9$\tau$62.3 for PeskaVLP-vision. An upper baseline using spatial annotations, SurgLatentGraph (LG-CVS), reaches 67.3 mAP with segmentation masks and graph reasoning; removing a key structure, the cystic duct, causes an 8-point mAP drop. The reported interpretation is that PeskaVLP consistently outperforms other multimodal foundations when adapted with CVS-AdaptNet, indicating stronger image–text alignment for surgical semantics.

A common misconception is that multimodal pretraining alone should solve CVS recognition. The reported zero-shot gap contradicts that view: generalist multimodal models required explicit multi-label adaptation, prompt design, and loss reformulation for this fine-grained surgical task.

7. Limitations, failure modes, and prospective directions

Several limitations are explicitly reported for PeskaVLP and its CVS adaptation. First, the quality of augmented texts depends on LLM reliability. Domain-specific inconsistencies can propagate through narration assignment, keystep expansion, or abstract summarization, although the training-time mixture of original and augmented text is intended to mitigate this (Yuan et al., 2024). Second, the compute profile is nontrivial: clip-level visual self-supervision doubles visual forward passes, and DTW introduces dynamic-programming overhead at phase and video scales.

Third, PeskaVLP’s procedure-aware monotonicity prior is not universally beneficial. For less-routinized procedures, such as some AutoLaparo cases, enforcing strong monotonic alignment may underperform, and the original paper suggests that adaptive curricula or softer alignment priors may help. Fourth, domain shift remains unresolved: despite cross-procedure and cross-center gains, hospital-specific imaging pipelines and tool sets still pose transfer challenges.

In the CVS context, the principal limitation is the performance gap relative to spatially supervised approaches. PeskaVLP plus CVS-AdaptNet reaches 57.6 mAP, whereas the segmentation/graph-based SurgLatentGraph upper baseline reaches 67.3 mAP (Baby et al., 7 Jul 2025). The CVS study also emphasizes prompt sensitivity: random or poorly matched text degrades performance substantially. In addition, evaluation is restricted to a single CVS-labeled dataset because of limited data availability.

The future directions stated across the two works are broadly compatible. For PeskaVLP, promising directions include federated or multi-center pretraining, stronger robustness to ASR and metadata noise, and additional guardrails for LLM-augmented text. For CVS-AdaptNet, the stated possibilities include stronger pretraining, clinically discriminative and standardized prompt templates, hierarchical surgical knowledge, hard-negative mining, margin-based losses, and hybrid methods that incorporate lightweight spatial cues without full segmentation (Yuan et al., 2024, Baby et al., 7 Jul 2025). This suggests that PeskaVLP is best understood not as a completed endpoint, but as a foundation for surgical multimodal transfer whose utility depends on how effectively domain structure, language supervision, and downstream task formulation are coupled.