MML-SurgAdapt: Unified Surgical Vision–Language AI

• MML-SurgAdapt is a unified CLIP-based framework that integrates surgical phase recognition, action triplet recognition, and safety assessment into a single multi-label space.
• It extends Single Positive Multi-Label learning with Hill loss to handle incomplete and noisy annotations across heterogeneous surgical datasets.
• The framework achieves competitive performance while reducing required labels by 23%, unifying diverse surgical tasks through prompt-conditioned representations.

MML-SurgAdapt is a unified, task-agnostic vision–language framework for surgical computer vision that adapts CLIP to the multi-task setting of a single procedure by expressing heterogeneous supervision as natural language prompts in a shared embedding space. Introduced in “Adaptation of Multi-modal Representation Models for Multi-task Surgical Computer Vision” (Walimbe et al., 7 Jul 2025), it was designed to address a central difficulty in surgical AI: phase recognition, action triplet recognition, and safety assessment are usually annotated in separate datasets and modeled independently, even though they are complementary aspects of the same operative workflow. Its defining methodological move is to extend Single Positive Multi-Label (SPML) learning to multi-task surgical data with incomplete or noisy annotations, thereby enabling joint training on Cholec80, Endoscapes2023, and CholecT50 while reducing the required labels by 23%.

1. Definition and problem setting

In its original formulation, MML-SurgAdapt is a CLIP-based multi-task model that treats diverse surgical computer-vision tasks as prompt-conditioned label retrieval in a single multi-label space rather than as separate classification problems. The motivating claim is that surgical workflows are inherently multi-task: the same frame may be relevant simultaneously to phase recognition, Critical View of Safety (CVS) assessment, and action triplet recognition, but conventional pipelines typically isolate these tasks into separate models and separate annotation protocols. MML-SurgAdapt was proposed to reduce that fragmentation and to replace task-specific supervision with natural language supervision grounded in a shared image–text representation (Walimbe et al., 7 Jul 2025).

A recurrent source of confusion is terminological. In (Walimbe et al., 7 Jul 2025), “MML-SurgAdapt” names a specific framework. In later papers, the label is also used more broadly as a framing for multimodal surgical adaptation problems, including surgical segmentation adaptation, vision–language adaptation, and reasoning MLLMs. This suggests that the term has acquired a wider conceptual scope than the original CLIP-based multi-task method, but the canonical method remains the one introduced in (Walimbe et al., 7 Jul 2025).

The core dataset-integration problem is partial labeling. Each source dataset supplies labels only for its own task, leaving labels for the other tasks missing. Treating all missing labels as negatives produces false negatives. MML-SurgAdapt addresses this by importing SPML into surgical multi-task learning and optimizing under the assumption that only one positive label per image may be available while other positives can remain unlabeled.

2. Unified label space, prompts, and architecture

The model follows CLIP’s two-tower design with an image encoder and a text encoder initialized from pretrained ViT-L/14 weights. All task labels are mapped to prompts, and the combined supervision is represented as a single label union of $K = 110$: seven phase classes from Cholec80, three CVS classes from Endoscapes2023, and 100 action triplets from CholecT50. For an input image $X$ and prompts $\{p_k\}_{k=1}^K$, the encoders produce

$I_f = f_i(X), \qquad T_f = \{f_t(p_k)\}_{k=1}^K,$

with $I_f \in \mathbb{R}^d$ and $T_f \in \mathbb{R}^{K \times d}$ (Walimbe et al., 7 Jul 2025).

A graph-convolutional refinement layer is then applied to the text features to model inter-label dependencies across tasks. The adjacency matrix $A \in \mathbb{R}^{K \times K}$ is constructed from cosine similarities among prompt embeddings and sparsified to retain only top-$K$ connections, with a scaling factor adjusting neighbor weights. With $H^{(0)} = T_f$, the update is

$H^{(l+1)} = \rho(A H^{(l)} W^{(l)}),$

followed by residual aggregation:

$X$0

Per-label image–text similarity is then

$X$1

and probabilities are produced by a temperature-scaled sigmoid,

$X$2

Prompt engineering is deliberately simple. The template is “a photo of [text for TASK and LABEL]”, with examples such as “a photo of the phase Calot Triangle Dissection”, “a photo of the lower part of the gallbladder divided from the liver bed to expose the cystic plate”, and “a photo of the tool grasper performing the action grasp on the target gallbladder”. No prompt tuning or learnable text tokens are reported. The GCN over prompt embeddings functions as a lightweight adapter over the text branch rather than as a replacement for CLIP’s native alignment.

This architecture places MML-SurgAdapt in a specific design regime: it is not a contrastive pretraining method, not a temporal sequence model, and not a generative VLM. It is a multi-label, prompt-conditioned surgical recognition system that uses end-to-end fine-tuning and graph-refined prompt semantics.

3. SPML extension and optimization under incomplete labels

The principal methodological contribution is the extension of SPML to integrated surgical datasets. Two training regimes are defined. In the Partial Positive setup, each image retains the positive labels provided by its source dataset while all other labels are unobserved and treated as negatives. In the Single Positive setup, one positive label is randomly selected for each image and the rest are set to zero. The latter reduces annotation requirements substantially, but only if the loss is robust to missing positives among nominal negatives (Walimbe et al., 7 Jul 2025).

MML-SurgAdapt uses Hill loss for that purpose. With margin $X$3 and focusing hyperparameter $X$4, the positive probability with margin is

$X$5

The Hill loss terms are

$X$6

$X$7

with $X$8 in the experiments. Conceptually, the objective sums positive terms over observed positives and negative terms over all remaining labels, while the negative reweighting suppresses likely false negatives.

The paper explicitly notes that, unlike standard CLIP pretraining, no InfoNCE-style contrastive loss is used. Optimization is performed directly on sigmoid probabilities with the Hill objective. This is a consequential design choice: MML-SurgAdapt is not attempting to relearn CLIP’s global contrastive geometry, but rather to adapt that geometry to a multi-task surgical label space with systematically missing annotations.

Evaluation-time masking is also central. Although the model is trained over the full $X$9 label space, test metrics for a given dataset consider only that dataset’s label subset. Thus Cholec80 evaluation uses the phase labels, Endoscapes2023 uses the CVS labels, and CholecT50 uses the triplet label configurations.

4. Data integration, training protocol, and evaluation regime

Three laparoscopic cholecystectomy datasets are integrated. Cholec80 provides phase recognition supervision; after removing overlapping videos to avoid leakage with CholecT50, the final split is 36 train, 7 val, and 15 test videos, corresponding to 78,614 train frames, 17,732 val frames, and 42,098 test frames. Endoscapes2023 provides CVS assessment with 120 train, 41 val, and 40 test videos, corresponding to 6,960 train frames, 2,331 val frames, and 1,799 test frames. CholecT50 uses the Rendezvous split with 35 train, 5 val, and 10 test videos, corresponding to 72,815 train frames, 6,797 val frames, and 21,251 test frames (Walimbe et al., 7 Jul 2025).

Dataset	Task	Label subset
Cholec80	Phase recognition	7 phase classes
Endoscapes2023	CVS assessment	C1, C2, C3
CholecT50	Action triplet recognition	100 triplets

All images are resized to $\{p_k\}_{k=1}^K$0 and normalized. In the SPML setup, the single positive label per image is randomly selected at dataset initialization and then held fixed for training and validation. The encoders are fine-tuned end-to-end from CLIP ViT-L/14 using Adam with learning rate $\{p_k\}_{k=1}^K$1, batch size 128, and 15 epochs. Model selection uses a combination of highest validation mAP, lowest loss on Partial Positive validation labels, and lowest loss on Single Positive validation labels. All experiments were run on a single NVIDIA A100 GPU.

The metric regime is task-specific. Cholec80 is evaluated with F1-score. Endoscapes2023 reports frame-level AP for C1, C2, and C3, together with mAP. CholecT50 reports AP across the I, V, T, IV, IT, and IVT configurations. Because the unified model predicts over all 110 labels, masking at evaluation is required to make cross-dataset comparison meaningful.

The label-efficiency result is a defining property of the method. The Single Positive setup reduced the number of labels required per image by 23% compared to the Partial Positive setting, while achieving nearly equivalent performance, described as approximately a 0.5% difference on average across tasks.

5. Quantitative performance and ablation structure

The reported test-set results show that MML-SurgAdapt is competitive with task-specific benchmarks while outperforming existing SPML frameworks. On Cholec80, it attains F1 = 74.2, exceeding VLPL at 69.5 and HSPNet at 68.4, and substantially surpassing task-specific CLIP ViT-L/14 at 52.6. On Endoscapes2023, it reports C1 = 55.3, C2 = 47.0, C3 = 65.4, and mAP = 55.9. The comparison note in the source is nuanced: ResNet50-MoCov2 reports 57.4 frame-level mAP, but MML-SurgAdapt improves AP on individual classes, surpasses SPML baselines, and is comparable to task-specific CLIP ViT-L/14 with 52.9 mAP. On CholecT50, it reports AP-I = 87.3, AP-V = 61.1, AP-T = 43.2, AP-IV = 37.8, AP-IT = 37.3, and AP-IVT = 30.3, exceeding SPML baselines such as HSPNet with AP-IVT = 23.5 (Walimbe et al., 7 Jul 2025).

Task	Metric	MML-SurgAdapt
Cholec80 phase recognition	F1	74.2
Endoscapes2023 CVS	mAP	55.9
CholecT50 triplets	AP-IVT	30.3

The ablations clarify where the gains arise. Under SPML, Hill loss yields Phase F1 = $\{p_k\}_{k=1}^K$2, CVS mAP = $\{p_k\}_{k=1}^K$3, and Triplet AP-IVT = $\{p_k\}_{k=1}^K$4 across five runs. WAN is described as slightly better for phase and triplet tasks but underperforming on CVS, whereas SPLC shows inconsistencies, with the source attributing this likely to pseudo-label reliance in a domain different from CLIP pretraining. The reported difference between SPML and Partial Positive is modest, with Partial Positive giving only about 0.5% average gains. The GCN over prompt embeddings yields only about a 0.5% improvement, which the paper attributes to the fact that end-to-end CLIP fine-tuning already captures strong semantics.

These results support a specific interpretation of MML-SurgAdapt’s contribution. Its empirical advantage is not primarily architectural novelty in the image backbone, nor a large gain from graph refinement alone. The central contribution is the SPML-based treatment of incompletely labeled, multi-task surgical data, together with prompt-based unification of heterogeneous labels.

6. Broader usage, limitations, and research trajectory

The original framework is limited to laparoscopic cholecystectomy, and the paper identifies several unresolved issues: class imbalance was not explicitly handled; preliminary class-weighting was inconclusive; rare triplets likely require tailored rebalancing or sampling; the GCN produced limited gains under end-to-end fine-tuning; and prompt sensitivity, contextual descriptors, and automated prompt generation were not systematically studied. The stated future directions are domain-adaptive VLM pretraining on surgical video–language corpora, temporal modeling for phase dynamics and action sequencing, and active learning to focus annotation effort on the most informative labels and frames (Walimbe et al., 7 Jul 2025).

Later work uses “MML-SurgAdapt” as a broader adaptation lens. In robotic surgical video segmentation, MDAL operationalizes one-to-many adaptation from a single source model to multiple target domains using first-frame supervision and dynamic online updates (Zhao et al., 2021). In surgical vision–language adaptation, Chain-of-Adaptation applies structured reasoning and KL-regularized RLVR to preserve pretrained multimodal priors during domain specialization (Li et al., 20 Mar 2026). In multimodal surgical data analysis, V-JEPA finetuning on unlabeled surgical video and cross-attentive fusion of OR streams are presented as a multimodal surgical adaptation paradigm (Pezold et al., 8 Sep 2025). In Surgical-VQLA, Surgery-R1 adapts a general MLLM through supervised and reinforcement fine-tuning with rule-based multimodal rewards (Hao et al., 24 Jun 2025).

This broader usage suggests a conceptual expansion from a specific CLIP-based SPML framework to an umbrella notion of surgical adaptation across segmentation, VLM specialization, multimodal fusion, and reasoning MLLMs. Even so, the original MML-SurgAdapt remains distinctive in two respects. First, it formulated surgical multi-task integration as a unified prompt-based, multi-label problem over a single label union. Second, it treated incomplete cross-dataset supervision not as a nuisance to be masked away, but as the central optimization problem, addressed by extending SPML with Hill loss.

In that sense, MML-SurgAdapt occupies a specific place in surgical AI: it is one of the clearest demonstrations that natural language supervision and label-efficient multi-label learning can consolidate heterogeneous surgical tasks within a single model, while later work indicates that the same adaptation logic is being generalized to broader multimodal and reasoning-centric settings.