SurgPIS: Part-Aware Surgical Tool Segmentation

• SurgPIS is a transformer-based framework that formulates surgical instrument segmentation as unified part-aware instance segmentation, predicting both tool and part masks jointly.
• It leverages a lightweight query transformation to derive part queries from instrument queries, enforcing structural linkage and yielding significant performance gains per ablation studies.
• The model employs a two-stage weakly supervised training protocol using PIS-labeled, IIS-only, and PSS-only datasets, achieving competitive metrics on EndoVis benchmarks.

Searching arXiv for papers on SurgPIS and closely related surgical segmentation work. SurgPIS denotes a transformer-based framework for surgical tool segmentation that formulates surgical instrument understanding as a unified part-aware instance segmentation (PIS) problem rather than treating instrument-level instance segmentation (IIS) and part-level semantic segmentation (PSS) as separate tasks. It was introduced as the first PIS model for surgical instruments and is designed to predict instrument instances together with the semantic parts belonging to each instance, while also supporting weakly supervised training from disjoint datasets labelled only for IIS or only for PSS (Wei et al., 25 Jul 2025). In the surrounding literature summarized alongside it, the same label is also used more loosely as a systems-level target platform for surveillance, perception, synthetic data generation, and perioperative decision support; however, its primary technical referent is the 2025 PIS model (Shen et al., 2018).

1. Problem formulation and scope

SurgPIS addresses a segmentation setting in which the desired output is not merely a set of instrument masks, nor merely a pixelwise part map, but a joint representation that preserves both instance identity and part semantics. The motivating observation is that existing methods only treat instrument-level instance segmentation or part-level semantic segmentation separately, without interaction between these tasks (Wei et al., 25 Jul 2025). The model therefore defines surgical tool segmentation as PIS, in which each instrument instance is associated with a structured set of part masks.

Within this formulation, an output must preserve parent–child structure: an instrument instance is the parent object, and its shaft, wrist, or other part classes are conditionally linked to that specific instance rather than emitted as globally shared semantic regions. This explicit linkage is central in scenes containing multiple tools with identical part types, where conventional semantic segmentation cannot preserve per-instrument ownership.

A common misconception is that SurgPIS is simply an IIS model with an auxiliary part head. The published formulation is stricter than that. It introduces part-specific queries derived from instrument-level object queries, and the training objective is organized around part-aware matching and aggregation into IIS or PSS views only when supervision is partial (Wei et al., 25 Jul 2025). Weak supervision in this setting does not mean absence of dense labels altogether; rather, training proceeds from a small PIS-labelled dataset and then extends to IIS-only and PSS-only datasets.

2. Architecture

SurgPIS builds on the Mask2Former transformer-based mask-classification framework. An input image $x \in \mathbb{R}^{3 \times H \times W}$ is processed by a ResNet or Swin Transformer backbone together with a Semantic-FPN pixel decoder to produce a high-resolution feature map

$F \in \mathbb{R}^{C_\epsilon \times H \times W}.$

A fixed set of learnable instrument-level queries

$Q \in \mathbb{R}^{N_q \times C_\epsilon}$

is then passed through a standard Transformer decoder with self-attention and cross-attention to $F$ (Wei et al., 25 Jul 2025).

Each processed instrument query produces two outputs: a class probability vector for instrument identity, including “no-object” and background, and a soft binary mask for the corresponding instrument instance. The distinctive architectural step is a lightweight query transformation that projects each instrument query into $C^{\text{part}}$ part queries. These transformed queries are cross-attended with the same feature map to generate part masks $\hat m_{j,k}$ that are explicitly conditioned on the parent instrument query.

This design yields two coupled prediction sets. At the instrument level, the model outputs $\{\hat c_j,\hat y_j\}_{j=1\ldots N_q}$; at the part level, it outputs $$\{\hat m_{j,k}\}_{j=1\ldots N_q,\;k=1\ldots C^{\text{part}}}$$. The architectural claim is not merely that both levels are predicted jointly, but that the part queries are derived from the instrument queries, thereby enforcing structural dependence between part masks and parent instances.

When full PIS labels are available, predictions are aligned to ground truth by a Hungarian bipartite matching procedure with a composite cost combining instrument classification, instrument mask loss, and summed part-mask losses. The paper’s ablation results indicate that this explicit part-query mechanism is not incidental: removing the part query transform causes a large degradation in PartPQ, reported as a collapse of approximately 35 percentage points in the captioned ablation summary (Wei et al., 25 Jul 2025).

3. Weakly supervised learning from disjoint labels

The weakly supervised strategy is organized in two stages. Stage 1 is fully supervised on a small PIS-labelled dataset $\mathcal D_{\mathrm{PIS}}$. Stage 2 adds IIS-only data $\mathcal D_{\mathrm{IIS}}$ and PSS-only data $F \in \mathbb{R}^{C_\epsilon \times H \times W}.$0, allowing the model to learn from disjoint supervision sources rather than requiring a single large-scale dataset annotated simultaneously at both levels (Wei et al., 25 Jul 2025).

The key mechanism is aggregation. For IIS-only data, SurgPIS applies an IIS aggregation that simply ignores the part masks and trains the instrument masks and instrument classes against the available IIS annotations. For PSS-only data, it aggregates instance-conditioned part predictions into global part-semantic probability maps. The construction first defines soft part-probability maps $F \in \mathbb{R}^{C_\epsilon \times H \times W}.$1 and then renormalizes them:

$F \in \mathbb{R}^{C_\epsilon \times H \times W}.$2

These aggregated maps are compared against the true PSS supervision (Wei et al., 25 Jul 2025).

To stabilize learning on partially labelled data, SurgPIS uses a student–teacher framework in which the teacher weights are maintained by exponential moving average:

$F \in \mathbb{R}^{C_\epsilon \times H \times W}.$3

A Dice-based filter discards low-confidence teacher part masks whose overlap with the student predictions falls below a threshold. This mechanism is intended to reduce drift when missing PIS information must be inferred indirectly from partial labels.

The loss design reflects the same decomposition. Under full supervision, the model uses instrument classification, instrument mask, and part mask losses, with the total supervised objective written as

$F \in \mathbb{R}^{C_\epsilon \times H \times W}.$4

Under weak supervision, the loss switches between IIS-aggregated and PSS-aggregated terms depending on the label type available for the current sample (Wei et al., 25 Jul 2025).

4. Training protocol, datasets, and reported performance

The implementation details reported for SurgPIS specify training on 2×Nvidia A100 with batch size 16, AdamW, weight decay 0.05, initial learning rate $F \in \mathbb{R}^{C_\epsilon \times H \times W}.$5, and polynomial decay with power $F \in \mathbb{R}^{C_\epsilon \times H \times W}.$6. Stage 1 trains for 40k iterations on EndoVis2018 PIS labels with large-scale jitter in the range $F \in \mathbb{R}^{C_\epsilon \times H \times W}.$7 and random crop $F \in \mathbb{R}^{C_\epsilon \times H \times W}.$8, using loss weights $F \in \mathbb{R}^{C_\epsilon \times H \times W}.$9. Stage 2 adds EndoVis2017 as IIS-only data and SAR-RARP50 as PSS-only data, sets $Q \in \mathbb{R}^{N_q \times C_\epsilon}$0, applies weight $Q \in \mathbb{R}^{N_q \times C_\epsilon}$1 for the PSS loss and $Q \in \mathbb{R}^{N_q \times C_\epsilon}$2 for the IIS loss, and uses strong augmentation for the student alongside lighter augmentation for the teacher (Wei et al., 25 Jul 2025).

The principal benchmark table reports that SurgPIS with an RN-50 backbone trained in “Full PIS only” mode reaches EV18 PartPQ $Q \in \mathbb{R}^{N_q \times C_\epsilon}$3, EV18 PartIoU $Q \in \mathbb{R}^{N_q \times C_\epsilon}$4, EV18 PQ $Q \in \mathbb{R}^{N_q \times C_\epsilon}$5, EV17 PartPQ $Q \in \mathbb{R}^{N_q \times C_\epsilon}$6, and EV17 PQ $Q \in \mathbb{R}^{N_q \times C_\epsilon}$7. In the “Weak (all)” setting with a Swin-B backbone, it reaches EV18 PartPQ 77.92, EV18 PartIoU 85.66, EV18 PQ $Q \in \mathbb{R}^{N_q \times C_\epsilon}$8, EV17 PartPQ $Q \in \mathbb{R}^{N_q \times C_\epsilon}$9, EV17 PQ $F$0, and SR50 PartIoU 84.75 (Wei et al., 25 Jul 2025).

The same study also evaluates semantic instance segmentation derived by aggregation from the PIS outputs. On EndoVis2018 and EndoVis2017, SurgPIS aggregated to ISS yields multi-class IoU of 67.47 and 46.09, respectively, compared with listed single-task baselines such as S3Net and MATIS (Wei et al., 25 Jul 2025).

Ablation results further clarify which components matter. On the ResNet-50 ablation table, the full model reports EV18 PartPQ $F$1, EV18 PQ $F$2, EV17 PartPQ $F$3, EV17 PQ $F$4, and SR50 PartIoU $F$5. Removing the part query transform reduces these values to EV18 PartPQ $F$6, EV18 PQ $F$7, EV17 PartPQ $F$8, EV17 PQ $F$9, and SR50 PartIoU $C^{\text{part}}$0. Removing pseudo-label filtering or using the same augmentations without strong augmentation also degrades performance (Wei et al., 25 Jul 2025).

The efficiency table frames the model’s deployment trade-offs. Reported configurations include RN-50 at $C^{\text{part}}$1M parameters, $C^{\text{part}}$2B FLOPs, and $C^{\text{part}}$3 FPS; Swin-B at $C^{\text{part}}$4M parameters, $C^{\text{part}}$5B FLOPs, and $C^{\text{part}}$6 FPS; ViT-S at $C^{\text{part}}$7M parameters, $C^{\text{part}}$8B FLOPs, and $C^{\text{part}}$9 FPS; and DINOV2-ViT-L at $\hat m_{j,k}$0M parameters, $\hat m_{j,k}$1B FLOPs, and $\hat m_{j,k}$2 FPS. The accompanying interpretation is that real-time operation is achieved with RN-50 and ViT-S, while larger backbones yield modest gains at higher latency (Wei et al., 25 Jul 2025).

5. Relation to prior weak supervision and surrounding surgical perception work

SurgPIS sits within a line of surgical vision research that has progressively expanded supervision granularity. PWISeg, for example, addresses weakly supervised surgical instrument instance segmentation using an FCN-based architecture with an FPN, a point-to-box branch, and a point-to-mask branch trained with an unsupervised projection loss, a key pixel association loss, and a key pixel distribution loss (Sun et al., 2023). On its own dataset, PWISeg reports segmentation mAP $\hat m_{j,k}$3 under the full weak-supervision loss, compared with $\hat m_{j,k}$4 for projection loss alone; on HOSPI-Tools it reports segmentation mAP $\hat m_{j,k}$5 (Sun et al., 2023). Relative to that line of work, SurgPIS advances from weakly supervised instrument-level segmentation to a unified formulation in which part semantics are tied to specific instrument instances.

A second adjacent strand concerns data generation rather than model architecture. “Realistic Surgical Image Dataset Generation Based On 3D Gaussian Splatting” proposes synthetic image generation by combining Gaussian representations of backgrounds and tools, with reported image quality of PSNR $\hat m_{j,k}$6, SSIM $\hat m_{j,k}$7, and LPIPS $\hat m_{j,k}$8 for the authors’ method, and a YOLOv5 model trained on synthetic data achieving Precision $\hat m_{j,k}$9 and Recall $\{\hat c_j,\hat y_j\}_{j=1\ldots N_q}$0 on a real-world test set of 300 images (Zeng et al., 2024). In the supplied implementation notes, this synthetic-data pipeline is explicitly discussed as a way to support a SurgPIS pipeline for instrument segmentation and pose estimation. A plausible implication is that SurgPIS-style part-aware models could benefit from the same type of controllable, pose-accurate synthetic supervision, although that integration is presented as guidance rather than as a reported experiment in the SurgPIS paper itself.

These adjacent works situate SurgPIS in a broader transition from bounding-box-level or point-level weak supervision toward structured, hierarchically linked predictions, while also indicating that training-data availability remains a central systems bottleneck across surgical vision tasks.

6. Broader systems usage, limitations, and future directions

In the broader material supplied with the SurgPIS paper, the term “SurgPIS” is also used as a platform label for more general surgical information processing. In one summary, Shen et al.’s clinical-note SSI surveillance workflow is described as a candidate component for incorporation into a real-world surveillance platform, with automatically generated SSI keywords such as “wound infection,” “cellulitis,” “dressing changes,” and “packed,” and a pruned decision tree reaching an $\{\hat c_j,\hat y_j\}_{j=1\ldots N_q}$1 of $\{\hat c_j,\hat y_j\}_{j=1\ldots N_q}$2 with four features under stratified 10-fold cross-validation (Shen et al., 2018). In another summary, SURGENT is specified as a surgical multi-agent assistance system across the perioperative workflow and is explicitly mapped into a Surgical Perioperative Information System (SurgPIS), with reported performance including DC $\{\hat c_j,\hat y_j\}_{j=1\ldots N_q}$3, MAR $\{\hat c_j,\hat y_j\}_{j=1\ldots N_q}$4, PFS $\{\hat c_j,\hat y_j\}_{j=1\ldots N_q}$5, GAR $\{\hat c_j,\hat y_j\}_{j=1\ldots N_q}$6, EWS $\{\hat c_j,\hat y_j\}_{j=1\ldots N_q}$7, FAR $\{\hat c_j,\hat y_j\}_{j=1\ldots N_q}$8, Recall $\{\hat c_j,\hat y_j\}_{j=1\ldots N_q}$9, and Sim $$\{\hat m_{j,k}\}_{j=1\ldots N_q,\;k=1\ldots C^{\text{part}}}$$0 (Shi et al., 28 May 2026). This suggests that, in secondary usage, “SurgPIS” can denote a broader perioperative platform rather than only the segmentation model.

That broader usage should be distinguished from the 2025 vision model. The segmentation paper itself is concerned with weakly supervised part-aware instance segmentation for surgical instruments, not with EMR surveillance or multi-agent perioperative planning. Conflating these senses would obscure the specific methodological contribution of part-specific query transformation and disjoint-label training.

The limitations and future directions recorded for SurgPIS are primarily model- and deployment-oriented. The published future directions include domain adaptation to new instruments, lightweight query transformers for lower latency, and integration into closed-loop robotic control (Wei et al., 25 Jul 2025). The efficiency table also makes clear that claims of real-time suitability are backbone-dependent and tied to A100-class hardware.

A final interpretive point follows from the way the name circulates across the supplied literature. SurgPIS, in its strict sense, is a part-aware transformer segmentation model. In a looser systems sense, it has become a shorthand for a surgical information and perception stack spanning perception, documentation mining, synthetic-data generation, and perioperative assistance (Shen et al., 2018). This suggests a convergence of surgical AI pipelines around unified platforms, but only the segmentation-specific meaning is directly established by the titled paper (Wei et al., 25 Jul 2025).