SSG-Com Surgical Scene Graph Framework

• The SSG-Com framework is a graph-based paradigm that models spatial and semantic relationships among surgical tools, anatomical structures, and operative actions.
• It integrates tool–action–target triplets with hand identity to achieve detailed disambiguation of surgical workflows and enhance safety assessments.
• Evaluation on the Endoscapes-SG201 dataset shows marked improvements in action triplet recognition and critical view of safety metrics over traditional approaches.

The SSG-Com framework (Surgical Scene Graph for Comprehensive Understanding) is a graph-based computational paradigm developed to advance surgical scene understanding by explicitly modeling both spatial and semantic interactions among surgical tools, anatomical structures, and operative actions. SSG-Com builds upon prior scene graph representations by integrating tool–action–target triplets and hand identity—the latter denoting whether a tool is operated by the surgeon’s right or left hand, or by an assistant. This design achieves heightened granularity in capturing surgical workflows, facilitating downstream applications such as critical view of safety assessment and action triplet recognition. SSG-Com is introduced and evaluated in the context of the Endoscapes-SG201 dataset, which is specifically annotated to support comprehensive graph-based modeling of surgical scenes (Shin et al., 21 Jul 2025).

1. Motivation and Conceptual Innovations

Surgical scene understanding fundamentally requires extracting both the spatial configuration and the operational semantics from endoscopic imagery. Classical graph-based approaches, such as LG-CVS, primarily encode spatial relations (e.g., above, below, left, right) and treat all surgical tools as a homogeneous class. These methods do not account for the combinatorial richness intrinsic to nuanced tool–action–target interactions (e.g., differentiating “Hook–Dissect–Cystic Duct” from “Hook–Retract–Cystic Artery”) or for the identity of the hand performing actions, both of which are critical in operative decision-making and safety. SSG-Com extends the modeling scope by introducing action edges for tool–anatomy interactions and an explicit prediction step for hand identity, thereby enabling fine-grained disambiguation of concurrent surgical activities and operator roles.

2. Graph Formalism and Network Architecture

The SSG-Com framework utilizes a two-stage pipeline anchored in a latent scene graph $G=(N,E)$, where nodes represent tools and anatomical structures, and edges denote both spatial and action-based relationships.

• Node Set: $N = N_\text{tool} \cup N_\text{anatomy}$, where each node $v_i$ is described by its bounding box $p_i\in \mathbb{R}^4$, class probability $s_i$, and RoIAlign feature vector $f_i\in\mathbb{R}^d$ derived from image $x$.
• Edge Set: $E = E_\text{spa} \cup E_\text{act}$ includes both spatial edges (encoding geometric relations) and action edges (tool→anatomy pairs).
• Edge Representation: Action edges are encoded via $e_{ij}^{\text{act}}=1$ for valid tool–anatomy interactions.

Graph convolutional message passing is structured as:

$$h_i^{(\ell+1)} = \sigma\Big(W_0 h_i^{(\ell)} + \sum_{j: A_{ij}=1} W_1 h_j^{(\ell)} + \sum_{j: A^{\text{act}}_{ij}=1} W_2 f_{ij} + b\Big)$$

where $N = N_\text{tool} \cup N_\text{anatomy}$0, $N = N_\text{tool} \cup N_\text{anatomy}$1, $N = N_\text{tool} \cup N_\text{anatomy}$2, $N = N_\text{tool} \cup N_\text{anatomy}$3, $N = N_\text{tool} \cup N_\text{anatomy}$4, $N = N_\text{tool} \cup N_\text{anatomy}$5, and $N = N_\text{tool} \cup N_\text{anatomy}$6 is the ReLU activation. This structure enables aggregation of self-information, neighborhood context, and explicit representation of tool–anatomy action cues.

3. Component Modules and Functionality

The SSG-Com pipeline is composed of the following components:

1. Object Detection Backbone: Utilizes Faster R-CNN with a ResNet-50 backbone for extracting convolutional feature maps from surgical images and generating region proposals. The RoIAlign module yields feature vectors $N = N_\text{tool} \cup N_\text{anatomy}$7 for detected objects.
2. Latent Graph Encoder (GCN): Constructs the scene graph leveraging adjacency matrices for both spatial and action edges, and performs multi-layer graph convolutions to propagate features across the surgical scene topology.
3. Action Edge Classifier: Predicts the action type for each valid tool–anatomy edge $N = N_\text{tool} \cup N_\text{anatomy}$8 using an MLP operating on $N = N_\text{tool} \cup N_\text{anatomy}$9, with outputs interpreted via a six-way softmax (corresponding to Dissect, Retract, Grasp, Clip, Coagulate, Null_verb).
4. Hand Identity Classifier: Predicts whether each tool is operated by the surgeon’s right hand, left hand, or an assistant using another MLP acting on the node’s graph-augmented feature $v_i$0.
5. Downstream Decoders: Task-specific MLPs are used for downstream objectives (e.g., Critical View of Safety or triplet recognition), with weights fine-tuned after freezing the detection and graph encoder modules.

4. Loss Functions, Training Protocol, and Dataset

The optimization framework is multi-objective, comprising detection, graph structure, action classification, and hand identity prediction losses:

$v_i$1

where $v_i$2 is the sum of Faster R-CNN loss, edge existence (BCE), and spatial relation classification (CE), with $v_i$3 and $v_i$4.

Training Dataset

• Endoscapes-SG201: 1,933 frames from 201 cholecystectomy videos, with 1,212/409/312 frames in train/validation/test splits.
• Annotations: 6 tool classes (Hook, Grasper, Clipper, Bipolar, Irrigator, Scissors), 5 anatomy classes (including Cystic Duct, Liver), 6 action classes, and 3 hand identities.
• Hardware: Single NVIDIA RTX 3090, two-stage regimen of 50 epochs for each phase.

5. Empirical Performance and Ablation

Evaluation is conducted via mean Average Precision (mAP) for two tasks: action triplet recognition (over 34 classes) and Critical View of Safety (CVS) prediction (multi-label mAP over three criteria). Key benchmark results:

Method	Triplet mAP	CVS mAP
ResNet50-DetInit (no graph)	9.7	55.3
LG-CVS (spatial only)	18.0	63.2
SSG-Com (spatial + action)	23.5	64.6
SSG-Com (+hand identity)	24.2	–

Ablation studies indicate:

• Adding graph structure improves triplet mAP by +8.3 percentage points over no-graph baselines.
• Introducing action edges yields a further +5.5 percentage point gain.
• Incorporating hand identity contributes an additional +0.7 percentage point increase.
• Training on Endoscapes-SG201 (tool-specific labels) rather than BBox201 (generic tool class) confers ~+5 percentage point gains.

6. Scientific Insights and Limitations

Explicit representation of tool–action–target interactions confers measurable advantages in semantic scene understanding, particularly in action triplet recognition, where differentiation between superficially similar tool uses is essential. Hand identity prediction further disambiguates roles (e.g., distinguishing between primary operation and retraction), providing richer context for downstream interpretability.

Principal limitations include the restriction to single-frame reasoning and the absence of temporally localized action predictions. A plausible implication is that future integration with spatio-temporal graph neural networks could capture long-range dependencies across video frames, enabling temporally coherent surgical understanding. Additionally, the exclusive focus on tool–anatomy and spatial edges suggests that extension to anatomy–anatomy edges (e.g., “artery inside liver plate”) may yield more holistic scene graphs.

7. Data and Code Availability, and Prospective Directions

The Endoscapes-SG201 dataset, with dense annotations of tool–action–target triplets and hand identity, is publicly released. The SSG-Com source code is also available, providing a platform for reproducible research and further development in holistic surgical scene graph generation. Extensions may include temporally-aware models, joint modeling of complex anatomical relationships, and integration with other modalities relevant to computer-assisted intervention systems (Shin et al., 21 Jul 2025).