Semantic Surgery in Surgical AI

• Semantic Surgery is a computational approach that models operative scenes as structured events involving devices, anatomy, and workflow.
• It integrates techniques like language-conditioned segmentation, scene graph generation, and pixel-level analysis to enhance real-time surgical decision-making.
• The framework supports applications such as instrument handoff guidance, collision avoidance, and adaptive robotic assistance through multimodal data fusion.

Semantic surgery denotes a computational view of surgery in which operative scenes are modeled not merely as pixels, detections, or isolated phase labels, but as semantically structured clinical events involving actors, devices, tools, anatomy, interactions, workflow, and, in robot-assisted settings, surgeon intent. In this usage, the term refers to surgical AI and context-aware operating-room intelligence rather than to a surgical technique. Across the literature, the common objective is clinically useful understanding: identifying what is present, which specific instance is relevant, how it relates to anatomy and other devices, how the procedure evolves over time, and what support should be provided in response (Özsoy et al., 2021, Ghamsarian, 2023, Sharma et al., 3 Aug 2025).

1. Conceptual scope and problem setting

A recurring claim in this literature is that clinically meaningful perception requires more than category recognition. GroundedSurg makes this explicit by contrasting category-level segmentation with language-conditioned resolution of a single instrument instance; EgoSurgery-Phase frames semantic understanding as open-surgery phase recognition guided toward semantically rich regions; and workload-aware intent recognition extends the notion further to context-sensitive inference of what the surgeon means to do under cognitive load (Ashraf et al., 1 Mar 2026, Fujii et al., 2024, Sharma et al., 3 Aug 2025). In this sense, semantic surgery is a family of problems rather than a single task.

The motivating applications are consistently intraoperative and workflow-dependent. GroundedSurg ties clinically reliable scene perception to instrument handoff guidance, collision avoidance, and workflow-aware robotic support. EgoSurgery-Phase associates phase recognition with real-time assistance, education, workflow analysis, and treatment evaluation. The cataract-video thesis situates the same agenda within context-aware systems that can interpret the surgical state, support decision-making, index and document video, and generate post-operative reports (Ashraf et al., 1 Mar 2026, Fujii et al., 2024, Ghamsarian, 2023).

A common misconception is that semantic surgery is equivalent to semantic segmentation. The surveyed works do not support that reduction. Segmentation is one core substrate, but the literature also includes scene graphs, language grounding, spatiotemporal pre-training, deformable 3D reconstruction, continual class-incremental adaptation, workflow recognition, relevance-aware compression, irregularity detection, and multimodal intent inference (Özsoy et al., 2021, Yang et al., 3 Jun 2025, Hindel et al., 3 Aug 2025).

2. Symbolic and relational representations of the operating room

The most explicit symbolic formulation is the Multimodal Semantic Scene Graph, which models surgery as a graph $G=(N,E)$ whose nodes include both physical entities and virtual components. In this representation, nodes may denote surgeons, patients, tools, imaging systems, monitors, C-arms, ECG streams, anesthesia values, timers, and displays; edges are organized into spatial relations, semantic relations, and virtual relations. The framework was prototyped on the MVOR dataset of 732 multi-view RGB-D frames from three cameras, and the paper further introduced a Bird’s Eye View spatio-temporal and semantic visualizer, an event change reporter, and Dynamic Time Warping over MSSG sequences for procedure synchronization (Özsoy et al., 2021).

CAT-SG brings a related idea to cataract surgery at dataset scale. Built on CATARACTS, it provides dynamic scene graph annotations over 50 videos, sampled at 5 fps, for 164,162 annotated frames, 29 unique objects, and 1,811,252 annotated relations. The abstract reports 9 unique relation types, while the detailed description enumerates 8 relation categories: Holding, Activation, Pushing, Pulling, Cutting, Inserting, Retracting, and Close to. The relation counts are highly skewed, with Close to occurring 1,677,724 times, indicating that dense spatial context is a major part of the representation (Holm et al., 26 Jun 2025).

The associated CatSGG model performs scene graph generation from instance-centric query embeddings, with a temporally enriched CatSGG+ variant aggregating same-class queries over 8-frame chunks. Reported results show Micro F1 and Macro F1 of 89.78 and 42.08 for CatSGG, and 89.78 and 43.11 for CatSGG+, exceeding ORacleSV and ORacleSVT. Downstream, temporally aware graphs improve workflow recognition, with GATv2 reaching 78.63 accuracy and 70.15 F1, and technique recognition for Stop and Chop versus Divide and Conquer, where GATv2 with 10 s at 5 fps reaches 68.75 ± 4.11 accuracy and 48.40 ± 1.72 F1 (Holm et al., 26 Jun 2025).

A plausible implication is that semantic surgery increasingly treats structured relations as first-class objects rather than as latent by-products of frame classification. The scene-graph papers make that commitment explicit: the representation is intended to remain interpretable at the level of who interacts with what, where, and under which procedural conditions (Özsoy et al., 2021, Holm et al., 26 Jun 2025).

3. Instance grounding and holistic pixel-level understanding

GroundedSurg formalizes surgical tool perception as a language-conditioned grounded segmentation problem. Each example consists of a surgical image $I \in \mathbb{R}^{H \times W \times 3}$, a natural-language query $T$ describing one target instrument instance, a bounding box $B=(x_{\min},y_{\min},x_{\max},y_{\max})$, and a center point $C=(x_c,y_c)$, with the required mapping written as $f(I,T,B,C)\rightarrow \hat{M}$. The benchmark spans ophthalmic, laparoscopic, robotic, and open surgery, drawing from InSeg1, InSeg2, SISVE, EndoVis, and CholecInstanceSeg, and contains about 612 images and 1,071 tool-level annotations, with an average of 1.6 tools per image. Queries were generated with Qwen-2.5-VL-Instruct and then reviewed by clinicians in two validation rounds (Ashraf et al., 1 Mar 2026).

Its evaluation protocol combines mask fidelity and localization quality through IoU@0.5, [email protected], mIoU, Dice, Bounding Box IoU, and Normalized Distance Error. All models are evaluated zero-shot in a unified language-conditioned instance segmentation protocol that projects localization outputs through a frozen SAM-style backend. The reported results expose large gaps: Qwen2.5-VL is among the stronger open models for coarse localization, yet under the unified protocol its Mask IoU is around 0.20 and Dice is around 0.20; VisionReasoner-7B is the strongest overall and reaches the highest BBox IoU and Dice in several settings; medical-domain VLMs do not consistently outperform general-purpose VLMs; and prompt rephrasing can sharply change IoU, BBox IoU, and NDE, although VisionReasoner is more robust (Ashraf et al., 1 Mar 2026).

Where GroundedSurg emphasizes reference resolution, Surg-SegFormer emphasizes prompt-free holistic segmentation in robot-assisted surgery. It uses a dual-transformer design with SegAnatomy, based on SegFormer-B2, and SegTool, based on SegFormer-B5 with a lightweight decoder and dense skip connections, followed by priority-weighted conditional fusion and morphological post-processing. On EndoVis2018 Task 1 it reports mIoU = 0.80 and Dice = 0.89; on EndoVis2017 it reports mIoU = 0.54 and Dice = 0.56; and on EndoVis2018 Task 2 it reports mIoU = 0.64 and Dice = 0.66. The paper’s practical argument is that prompt-free inference is more compatible with long procedures that exceed an hour than prompt-driven segmentation pipelines (Ahmed et al., 6 Jul 2025).

Together, these works define two complementary forms of pixel-level semantics. One asks which instance satisfies a linguistic reference; the other asks for autonomous dense delineation of tools, tissues, and critical structures. This suggests that semantic surgery at the image level is bifurcating into disambiguation-sensitive grounding and high-throughput prompt-free scene parsing (Ashraf et al., 1 Mar 2026, Ahmed et al., 6 Jul 2025).

4. Geometry, deformable reconstruction, and continual adaptation

Several papers extend semantic surgery from 2D labeling to geometry-aware reconstruction. Semantic-SuPer augments the SuPer framework for endoscopic tissue tracking by injecting semantic segmentation into deformable 3D reconstruction. Surgical tools are tracked kinematically, tissue is represented as a surfel map with an Embedded Deformation graph, and optimization is performed over ED node rotations, translations, and a global transform with objective $$\arg \min_{\mathbf{q}, \mathbf{b}, \mathbf{T}_g} \lambda_s \mathcal{L}_{sim} + \lambda_m \mathcal{L}_{morph} + \lambda_r \mathcal{L}_{reg}$$. The semantic component enters both through semantic-confidence-weighted ICP, where $$\rho_{i,o} = \exp^{-JSD(\mathbf{s}_i \| \mathbf{s}_o)}$$, and through a morphing loss that penalizes projected surfels falling outside their semantic region. On dVRK deformable-tissue experiments, the reported mean reprojection errors for Semantic-SuPer are 7.5 and 6.7 in Lab1, 8.6 and 9.2 in Lab2, 6.0 and 5.9 in Lab3, and 4.3 and 4.3 in Lab4 (Lin et al., 2022).

Another strand addresses the fact that deployed segmentation systems cannot remain static. TOPICS+ formulates robotic-surgery scene understanding as hierarchical class-incremental semantic segmentation. It extends TOPICS by adding Dice loss to the hierarchical loss, introducing hierarchical pseudo-labeling with thresholds $s_0=0.6$, $s_1=0.6$, and $I \in \mathbb{R}^{H \times W \times 3}$0, and using tailored surgical taxonomies. The paper contributes six CISS benchmarks over Endovis18, MM-OR, and Syn-Mediverse, with the refined Syn-Mediverse benchmark containing more than 144 classes. In disjoint incremental learning, TOPICS+ reaches overall mIoU of 49.05 on Endovis18, 42.69 on MM-OR, and 50.16 on Syn-Mediverse; in refinement learning, it reaches 35.13, 45.10, and 48.20, respectively (Hindel et al., 3 Aug 2025).

Cataract-surgery segmentation work contributes the data-centric and architectural foundations on which these more elaborate systems depend. On CaDIS, the central finding is that class imbalance handling matters more than encoder-decoder choice: Repeat Factor Sampling with $I \in \mathbb{R}^{H \times W \times 3}$1, Adaptive Sampling with $I \in \mathbb{R}^{H \times W \times 3}$2, and Lovász-type losses substantially improve rare-class performance, yielding reported test mIoU values up to 0.7909 on Task 2 and 0.7194 on Task 3 (Pissas et al., 2021). ReCal-Net then introduces region-channel-wise calibration and reports overall IoU 85.38 and Dice 91.22 for lens, pupil, iris, and instrument segmentation (Ghamsarian et al., 2021), while DeepPyramid reports 86.70% IoU and 92.03% Dice, a 3.66% overall IoU improvement over the best rival, through Pyramid View Fusion, Deformable Pyramid Reception, and Pyramid Loss (Ghamsarian et al., 2022).

A plausible implication is that semantic surgery requires both semantic richness and geometric stability. The reconstruction and continual-learning papers make clear that semantics must survive deformation, occlusion, changing class inventories, and evolving annotation protocols rather than being confined to a single static benchmark (Lin et al., 2022, Hindel et al., 3 Aug 2025).

5. Video-level semantics, workflow, and foundation models

Temporal structure is central to surgical meaning, and multiple works argue that frame-centric methods are insufficient. EgoSurgery-Phase addresses open surgery, which prior work had left understudied relative to minimally invasive surgery, by releasing an egocentric dataset with 21 videos, 15 hours after preprocessing, 10 distinct surgical types, 8 surgeons, and 27,694 labeled frames. Videos are recorded at 25 fps and 1920×1080, subsampled to 0.5 fps, and annotated into 9 phases: Disinfection, Design, Anesthesia, Incision, Dissection, Hemostasis, Irrigation, Closure, and Dressing. The distinctive addition is eye gaze, used by GGMAE as an empirical semantic richness prior for masked video modeling. With a ViT-Small VideoMAE-style backbone, 10-frame clips at 224×224, and a masking ratio of 90%, GGMAE reports 51.7 precision, 45.6 recall, and 33.9 Jaccard, improving NETE by 8.0% precision, 10.4% recall, and 6.4% Jaccard, and improving VideoMAE by 4.1% Jaccard (Fujii et al., 2024).

SurgVISTA generalizes the same idea to large-scale video foundation modeling. It is presented as the first video-level surgical pre-training framework, trained on 3,650 videos and 3,552,777 frames spanning more than 20 surgical procedures and over 10 anatomical structures. The architecture combines a shared spatiotemporal encoder, a video reconstruction decoder, and an image-level distillation decoder guided by a surgery-specific expert, with objective $I \in \mathbb{R}^{H \times W \times 3}$3, where $I \in \mathbb{R}^{H \times W \times 3}$4 and $I \in \mathbb{R}^{H \times W \times 3}$5. Evaluation covers 13 video-level datasets across workflow recognition, action recognition, triplet recognition, and skill assessment. On Cholec80, the reported scores are 91.5% image-level accuracy, 91.5% video-level accuracy, 87.3% precision, 87.7% recall, and 78.1% Jaccard, and the paper reports statistically significant gains with typically $I \in \mathbb{R}^{H \times W \times 3}$6 (Yang et al., 3 Jun 2025).

EndoARSS places temporal-semantic reasoning in a multi-task setting where activity recognition constrains segmentation. Built on DINOv2 with Low-Rank Adaptation, TESLA, and Spatially-Aware Multi-Scale Attention, it jointly predicts activity labels and segmentation masks. The paper introduces MTLESD, MTLEndovis, and MTLEndovis-Gen. On MTLESD, EndoARSS with TESLA reports AC 99.99%, F-score 99.95%, mIoU 92.16%, Dice 95.79%, HD 13.25, and SSIM 95.67%; on MTLEndovis, it reports AC 72.18%, F-score 66.77%, mIoU 72.95%, Dice 83.47%, HD 19.53, and SSIM 91.26%; and on MTLEndovis-Gen, AC 49.24%, F-score 52.78%, mIoU 49.10%, Dice 49.73%, HD 22.17, and SSIM 88.27% (Wang et al., 7 Jun 2025).

The cataract-video thesis provides a task-specific precursor to these video foundation approaches. It argues that phase recognition improves when relevant content is localized both temporally and spatially, using idle/action recognition and cornea localization within the LocalPhase framework, and it extends the same semantic logic to relevance-based compression, irregularity detection, and self-supervised representation learning (Ghamsarian, 2023). This suggests that temporal semantics in surgery are increasingly handled as a joint problem of attention, localization, representation learning, and workflow structure rather than as phase classification alone.

6. Human intent, workload, applications, and term variation

The broadest extension of semantic surgery is from scene understanding to intent understanding. The workload-oriented robot-assisted surgery paper defines semantic understanding as the ability to infer what the surgeon means to do, not merely what motion is observed. Its proposed intelligent adaptive multimodal framework has four stages—data acquisition, signal processing, feature extraction, and fusion strategy and decision-making—and uses EEG, eye tracking, EMG, and ECG as tightly coupled modalities for estimating cognitive workload and improving intent recognition. The paper is explicit that it is a proposal rather than a completed benchmark study: it provides no formal objective functions, datasets, baselines, or quantitative results, but it argues that real-time workload estimation should allow a robotic system to adjust support or autonomy under mentally demanding conditions (Sharma et al., 3 Aug 2025).

Across the literature, the application space is broad but consistent. GroundedSurg links semantic scene understanding to instrument handoff guidance, collision avoidance, and workflow-aware robotic support; MSSG targets process optimization, operating-room design, automatic report generation, and synchronization of procedures; Semantic-SuPer points toward navigation and semantically labeled 3D tissue maps; and the cataract-video thesis adds relevance-based compression, irregularity analysis, and context-aware indexing and documentation (Ashraf et al., 1 Mar 2026, Özsoy et al., 2021, Lin et al., 2022, Ghamsarian, 2023). The common claim is that surgical AI becomes clinically useful when it models relations, context, and meaning rather than isolated detections.

The literature also identifies substantial limitations. GroundedSurg shows that modern VLMs remain far from robust clinical-level performance and are sensitive to prompt wording (Ashraf et al., 1 Mar 2026). EgoSurgery-Phase emphasizes the scarcity and anonymization difficulty of open-surgery video data (Fujii et al., 2024). Semantic-SuPer notes that segmentation quality directly affects tracking quality and that depth estimation remains noisy (Lin et al., 2022). TOPICS+ highlights background shift and catastrophic forgetting in evolving surgical environments (Hindel et al., 3 Aug 2025). EndoARSS shows degradation under domain shift even when it remains best-in-comparison (Wang et al., 7 Jun 2025). These constraints indicate that semantic surgery is still a research program rather than a solved systems problem.

In a distinct, non-surgical usage, “Semantic Surgery” also names a training-free, zero-shot concept-erasure method for text-to-image diffusion models. There the term refers to calibrated subtraction in text-embedding space before diffusion, with Co-Occurrence Encoding and an optional visual feedback loop; it is unrelated to surgical AI except for the shared metaphor of precise semantic intervention (Xiong et al., 26 Oct 2025). The coexistence of these usages makes contextual disambiguation necessary in bibliographic and technical discussions.