Surg-SegFormer: Dual-Branch Surgical Segmentation
- Surg-SegFormer is a prompt-free, dual transformer model that segments both anatomical tissues and surgical instruments without manual prompts.
- It employs specialized SegFormer branches for anatomy and tool parsing, fusing predictions via a priority-weighted conditional rule and morphological refinement.
- The model demonstrates robust performance on EndoVis benchmarks, achieving up to 0.80 mIoU in holistic surgical scene segmentation.
Surg-SegFormer is a prompt-free, dual transformer-based model for holistic surgical scene segmentation in robot-assisted surgery. It extends SegFormer, which unifies a hierarchically structured Transformer encoder with a lightweight MLP decoder, by separating anatomy parsing and instrument parsing into specialized branches and then fusing their predictions through a priority-weighted conditional rule, a logical OR over foreground detections, and morphological refinement (Ahmed et al., 6 Jul 2025, Xie et al., 2021). In reported experiments, Surg-SegFormer attains a mean Intersection over Union of $0.80$ on EndoVis 2018 Task 1, $0.64$ on EndoVis 2018 Task 2, and $0.54$ on EndoVis 2017, positioning it as a prompt-free model for automated surgical scene comprehension over long video sequences (Ahmed et al., 6 Jul 2025).
1. Problem domain and target tasks
Surg-SegFormer addresses holistic surgical scene segmentation, defined as dense, pixel-level parsing of all relevant elements in endoscopic frames, including anatomical tissues, articulated instruments, and critical structures. The stated motivation is pedagogical and operational: surgeons face firm intraoperative time constraints, surgical videos commonly exceed an hour, and prompt-driven segmentation pipelines are impractical for frame-by-frame deployment over such recordings (Ahmed et al., 6 Jul 2025).
The model is evaluated on two established RAS benchmarks. EndoVis 2017 is treated as an instrument-type segmentation task with seven instrument categories: Bipolar Forceps, Prograsp Forceps, Large Needle Driver, Vessel Sealer, Grasping Retractor, Monopolar Curved Scissors, and Ultrasound Probe. EndoVis 2018 contains two tasks. Task 1 is described as holistic scene segmentation with originally 12 labels, and common practice merges instrument shaft, wrist, and clasper into a single Robotic Instrument Part, yielding seven effective labels analyzed in the model description: Background Tissue, Robotic Instrument Part, Kidney Parenchyma, Covered Kidney, Small Intestine, Suturing Needle, and Ultrasound Probe. Task 2 is instrument-type segmentation with seven categories: Bipolar Forceps, Prograsp Forceps, Large Needle Driver, Monopolar Curved Scissors, Ultrasound Probe, Suction Instrument, and Clip Applier (Ahmed et al., 6 Jul 2025).
The core practical argument for prompt-free operation is explicit. SAM-like or CLIP-guided methods can require points, boxes, or text prompts for each frame or object, which is infeasible for lengthy surgical videos. Surg-SegFormer therefore trains fully automatic segmentation branches and applies them per frame without prompt encoders (Ahmed et al., 6 Jul 2025).
2. SegFormer lineage and inherited design principles
Surg-SegFormer is grounded in SegFormer’s architectural program. SegFormer was introduced as a simple and efficient semantic segmentation framework with two defining properties: a hierarchically structured Transformer encoder that outputs multiscale features, and a lightweight MLP decoder that aggregates information from different layers (Xie et al., 2021). In its original form, the encoder produces feature maps at $1/4$, $1/8$, $1/16$, and $1/32$ of input resolution, removes positional encoding entirely, and injects locality through Mix-FFN with a depthwise convolution:
This positional-encoding-free design is important because the original SegFormer study reported that when positional encoding is used, mIoU drops going from $0.64$0 to $0.64$1, whereas Mix-FFN drops only $0.64$2, demonstrating robustness to resolution mismatch (Xie et al., 2021). Attention is also made computationally tractable via spatial reduction, reducing complexity from $0.64$3 to $0.64$4, with segmentation experiments using $0.64$5 across stages (Xie et al., 2021).
These traits are directly relevant to surgical imagery. The SegFormer design notes emphasize that endoscopic and laparoscopic data exhibit variable resolutions and aspect ratios, small and thin structures, severe class imbalance, and the need for both fine detail and scene-level context. This is the architectural background against which Surg-SegFormer specializes the baseline into anatomy- and instrument-focused branches (Xie et al., 2021). The underlying strength of the base family is also established empirically in the original paper: SegFormer-B4 reaches $0.64$6 mIoU on ADE20K with $0.64$7M parameters, and SegFormer-B5 achieves $0.64$8 mIoU on the Cityscapes validation set while showing excellent zero-shot robustness on Cityscapes-C (Xie et al., 2021).
3. Dual-branch architecture and fusion mechanism
Surg-SegFormer departs from standard single-branch SegFormer by instantiating two specialized transformer segmentation paths. The first branch, SegAnatomy, is a SegFormer-B2 instance fine-tuned for anatomical tissues and fine structures in EndoVis 2018 Task 1. The second branch, SegTool, uses a SegFormer-B5 encoder specialized for instrument segmentation and replaces the standard MLP head with a custom lightweight decoder incorporating dense skip connections to preserve high-frequency detail such as tool tips and suturing needles (Ahmed et al., 6 Jul 2025).
The encoder side remains SegFormer-like. For an input image $0.64$9, stage-$0.54$0 features are represented as
$0.54$1
The attention mechanism is given in the usual form
$0.54$2
What changes is the decoding and the inter-branch composition (Ahmed et al., 6 Jul 2025).
SegAnatomy retains SegFormer’s efficient MLP-style decoder. SegTool instead projects each multiscale feature map to a shared channel dimension, upsamples features to a common high resolution, concatenates them, and uses dense skip connections so that early-stage features explicitly inform later decoding stages. The description is explicit that this custom decoder is intended to mitigate loss of spatial detail after downsampling by factors of $0.54$3, $0.54$4, $0.54$5, and $0.54$6 (Ahmed et al., 6 Jul 2025).
The final fusion is a priority-weighted conditional rule with confidence comparison:
$0.54$7
A logical OR is then applied on foreground predictions to retain any positive detections from either branch, followed by morphological refinement (Ahmed et al., 6 Jul 2025).
A common misunderstanding is that Surg-SegFormer is merely SegFormer fine-tuned on surgical images. The architecture described in the model paper is more specific: two different SegFormer variants are specialized for different semantic regimes, one decoder remains standard, the other is custom and small-object-aware, and the final prediction is not a simple average but a conditional fusion rule followed by post-processing (Ahmed et al., 6 Jul 2025).
4. Optimization, loss design, and training protocol
The training setup combines pixel-wise cross-entropy with a Tversky term designed to emphasize false-negative control on delicate and underrepresented classes. Cross-entropy is written as
$0.54$8
the Tversky index as
$0.54$9
with $1/4$0 and $1/4$1, and the corresponding loss
$1/4$2
The total loss is a convex combination,
$1/4$3
with the paper noting that $1/4$4 is empirically tuned so that false negatives are penalized more heavily (Ahmed et al., 6 Jul 2025).
Dice and Focal losses are discussed for reference but are not used in the final model. No label smoothing or auxiliary heads are reported. The optimizer is Adam with weight decay $1/4$5, the initial learning rate is $1/4$6, a cyclic learning-rate scheduler is used, batch size is $1/4$7, and training proceeds for $1/4$8 epochs. Training and inference were run on NVIDIA RTX 4090 and NVIDIA V100-32GB hardware, with larger models trained in the cloud to reduce runtime (Ahmed et al., 6 Jul 2025).
The preprocessing description is comparatively narrow. The reported augmentations are geometric—flips, cropping, and rotations—while resizing and normalization are inherited from standard SegFormer pipelines and are not explicitly detailed beyond those transforms. The paper follows common class merges and challenge protocols but does not provide dataset sizes, frame counts, or official splits in its description (Ahmed et al., 6 Jul 2025).
5. Empirical performance, ablations, and per-class behavior
The central reported results are as follows (Ahmed et al., 6 Jul 2025).
| Benchmark | mIoU | Dice |
|---|---|---|
| EndoVis 2018 Task 1 | 0.80 | 0.89 |
| EndoVis 2018 Task 2 | 0.64 | 0.66 |
| EndoVis 2017 | 0.54 | 0.56 |
On EndoVis 2018 Task 1, Surg-SegFormer is described as state of the art in the task’s prompt-free setting, surpassing MedT at $1/4$9 and AdaptiveSAM at $1/8$0. On EndoVis 2018 Task 2, it is competitive rather than dominant: it trails SurgicalSAM at $1/8$1 mIoU and MATIS at $1/8$2 mIoU, but exceeds U-Net at $1/8$3 mIoU and a re-trained single SegFormer at $1/8$4 mIoU. On EndoVis 2017, it outperforms U-Net at $1/8$5 and a SegFormer baseline at $1/8$6, while prompt-tuned AdaptiveSAM reaches $1/8$7 mIoU (Ahmed et al., 6 Jul 2025).
The per-class analysis reinforces the purpose of the dual design. For EndoVis 2018 Task 1, reported classwise highlights include BT $1/8$8, RI $1/8$9, KP $1/16$0, CK $1/16$1, SI $1/16$2, ST $1/16$3, SN $1/16$4, Clamps $1/16$5, Suction $1/16$6, and UP $1/16$7. The strongest margins are reported on RI, CK, and SN, with nearly perfect performance on SN and Clamps. A relative weakness is noted for SI compared with MedT, attributed to specularities and rapid motion (Ahmed et al., 6 Jul 2025).
For EndoVis 2017, Surg-SegFormer performs strongly on Ultrasound Probe at $1/16$8 IoU and Monopolar Curved Scissors at $1/16$9, but much less well on Bipolar Forceps at $1/32$0 and Prograsp Forceps at $1/32$1. For EndoVis 2018 Task 2, the highest classwise IoUs are reported for Suction Instrument at $1/32$2 and Clip Applier at $1/32$3, whereas Prograsp Forceps at $1/32$4 and Large Needle Driver at $1/32$5 remain difficult (Ahmed et al., 6 Jul 2025).
The ablation results further specify why the final configuration was adopted. SegFormer-B2 was the best tested anatomy backbone with mIoU $1/32$6 and Dice $1/32$7, while B5 was best for instruments with mIoU $1/32$8 and Dice $1/32$9. The composite 0 loss achieved the best internal validation in the reported ablation table, with mIoU 1 and Dice 2. Exact fusion ablation numbers are not tabulated, but the priority-weighted fusion plus OR rule is described as central to gains in crowded frames with overlapping tools and ambiguous tissues (Ahmed et al., 6 Jul 2025).
6. Position in the literature, limitations, and practical implications
Surg-SegFormer occupies a specific position in surgical segmentation research: it is a prompt-free transformer system optimized for long-form surgical video, and it extends SegFormer not by merely scaling a backbone but by architectural specialization and late fusion. Its main comparative claim is strongest on anatomy-rich holistic scene segmentation rather than on pure instrument-type segmentation, where prompt-based methods can still score higher (Ahmed et al., 6 Jul 2025).
This distinction matters because broader surgical benchmarking shows both the promise and the limits of transformer segmentation in the operating room. In a separate benchmark on SAR-RARP50, SegFormer ranked second overall behind DeepLabV3+, with stronger global context and generalization across varying instrument sizes and orientations, but slightly weaker delineation of fine boundaries compared with DeepLabV3+ (Ameli, 10 Apr 2026). That benchmark also notes common surgical failure modes—thin sutures and clips, specular regions, heavy occlusion, and motion blur—and reports that DeepLabV3+ better preserves thin tips and boundaries, whereas SegFormer generalizes better in clutter and partial occlusion (Ameli, 10 Apr 2026). This context suggests why Surg-SegFormer places particular emphasis on dense skip connections, Tversky weighting, and morphological refinement.
Several limitations are explicit in the Surg-SegFormer report itself. Not all instrument categories achieve high IoU; visually similar graspers such as PF, BF, and LND remain challenging because of inter-class ambiguity and limited diversity. Specular highlights, smoke, and rapid tissue motion can degrade accuracy. Domain shift across hospitals or endoscopes is not explicitly analyzed. Runtime characteristics are only partially characterized: parameters, FLOPs, memory footprint, and FPS are not reported, although training on RTX 4090 and V100-32G hardware indicates practical feasibility for modern GPUs (Ahmed et al., 6 Jul 2025).
The main controversy surrounding this class of models concerns prompt-based versus prompt-free segmentation. The reported evidence does not support a universal ranking. Prompt-based methods are impractical for multi-hour videos when prompts are required at inference, yet they can outperform prompt-free models on pure instrument-type tasks. Conversely, Surg-SegFormer surpasses prompt-tuned AdaptiveSAM on EndoVis 2018 Task 1 while trailing SurgicalSAM on EndoVis 2018 Task 2 and AdaptiveSAM on EndoVis 2017 (Ahmed et al., 6 Jul 2025). The resulting picture is not that prompt-free specialization eliminates the need for all other paradigms, but that it provides a scalable alternative for automated scene understanding when continuous human interaction is infeasible.
Future directions proposed for Surg-SegFormer include dynamic loss weighting based on per-frame class proportions, incorporation of Focal Tversky Loss, improved handling of grasper-type instruments, and explicit modeling of temporal continuity. These directions are consistent with the model’s current strengths and failure modes: strong holistic parsing, high performance on several anatomy and instrument categories, and residual difficulty on visually similar tools, fast motion, smoke, and fine reflective structures (Ahmed et al., 6 Jul 2025).