Surgical Action Triplet

• Surgical action triplet is a structured representation defined as (tool, action, target) that captures detailed surgical interactions.
• The model leverages graph-based representations and expert annotation to link tools, actions, and anatomical targets, enhancing context-aware decision support.
• Empirical results on Endoscapes-SG201 show that incorporating explicit action edges and hand identity improves recognition performance significantly.

A surgical action triplet is a structured representation used to model the fine-grained interactions taking place within a surgical scene. Formally, a surgical action triplet is defined as an ordered tuple (tool, action, target) that specifies which instrument (tool) is performing what kind of manipulation (action) on which anatomical structure (target). This schema provides a minimal yet semantically dense account of tool–tissue interaction at the frame or instance level, addressing the need for context-aware understanding and downstream decision support in computer-assisted interventions and surgical AI (Shin et al., 21 Jul 2025).

1. Formal Definition and Ontological Structure

A surgical action triplet is expressed as (tool, action, target), with domains:

• tool $t_i \in C_\mathrm{tool} = \{$Hook, Grasper, Clipper, Bipolar, Irrigator, Scissors$\}$
• action $a_k \in C_\mathrm{action} = \{$Dissect, Retract, Grasp, Clip, Coagulate, Null_verb$\}$
• target $u_j \in C_\mathrm{anatomy} = \{$Anatomy₁,…,Anatomy₅$\}$

Given sets of detected tools $N_\mathrm{tool}$ and anatomical targets $N_\mathrm{anatomy}$, the space of possible triplets is

$$T = \{\, (t_i, a_k, u_j) \mid t_i \in N_\mathrm{tool},\ u_j \in N_\mathrm{anatomy},\ a_k \in C_\mathrm{action} \,\}$$

Each object node in scene graph representations is described as $n = (p, s, f)$, where $p$ indicates the bounding-box, $s$ the class-probability, and $f \in \mathbb{R}^d$ the feature vector derived via RoIAlign.

Hand identity $h \in \{$Rt, Lt, Assi$\}$ is often included to further specify which agent (right/left hand of the surgeon, assistant) manipulates each tool, crucial for disambiguating tool usage contexts (Shin et al., 21 Jul 2025).

2. Dataset Annotation Protocols

The Endoscapes-SG201 dataset exemplifies high-fidelity annotation for action triplets:

1. Refinement of bounding boxes from broader datasets (e.g., Endoscapes-BBox201) to accurately localize tools and anatomical structures.
2. Subdivision of generic tool classes into the canonical fine-grained set of six instruments.
3. For each tool–anatomy pair within a frame, assignment of one action label from the set of six or "Null_verb" to encode explicit absence of interaction.
4. Explicit annotation of hand identity: Rt (surgeon’s right), Lt (surgeon’s left), or Assi (assistant).
5. Iterative expert review to ensure label consistency across frames (Shin et al., 21 Jul 2025).

Table: Distributional summary from Endoscapes-SG201

Statistic	Value
Annotated frames	1,933 (train 1,212/val 409/test 312)
Tool instances (total)	Hook 1,060, Grasper 1,598, Clipper 219, Bipolar 142, Irrigator 69, Scissors 4
Action triplets	Train 916, Val 349, Test 208
Action labels (total)	Dissect 916, Retract 1,429, Grasp 95, Clip 207, Coagulate 60, Null_verb 383
Hand identity (total)	Rt 1,543, Lt 1,328, Assi 221

The annotation protocol ensures not just the detection of entities but the precise modeling of their role, context, and agent.

3. Graph-Based Representation of Action Triplets

Surgical action triplets can be modeled as explicit structures within scene graphs. Within SSG-Com (Shin et al., 21 Jul 2025):

• Node types:
  • $$N_\mathrm{tool} = \{\, n_i = (p_i, s_i, f_i) \mid s_i \in C_\mathrm{tool} \,\}$$
  • $$N_\mathrm{anatomy} = \{\, n_j = (p_j, s_j, f_j) \mid s_j \in C_\mathrm{anatomy} \,\}$$
• Edge types:

1. Spatial edges (relative positions: left/right, above/below, inside/outside) 2. Surgical action edges (SAE): $e^{{action}}_{ij}=1$ iff $n_i\in N_\mathrm{tool}$ and $n_j\in N_\mathrm{anatomy}$

• Adjacency matrix: $A_{ij}=1$ if a spatial relation or $e^{{action}}_{ij}=1$; $0$ otherwise
• Features: Node features from RoIAlign; edge features from RoIAlign on the union of bounding boxes $p_i \cup p_j$

Message passing is performed with graph convolutional operations:

$$H^{(l+1)} = \sigma\left( D^{-½} A D^{-½} H^{(l)} W^{(l)} \right)$$

where $H^{(l)}$ contains node features, $D$ is the degree matrix, $A$ the adjacency encoding spatial and action relations, $W^{(l)}$ learnable weights, and $\sigma$ ReLU activation.

Classification heads enable:

1. Action edge prediction via softmax over 6 action classes
2. Hand identity prediction via softmax over {Rt, Lt, Assi}
3. Triplet recognition by exhaustively enumerating valid $|\mathcal{T}|=34$ triplets, classifying combined tool–anatomy–action edge features

4. Model Training and Loss Functions

The complete model is trained via a composite loss:

• $$L_\mathrm{action} = \sum_{i,j} \mathrm{CE}(y_{ij}^{\mathrm{action}}, \hat{y}_{ij}^{\mathrm{action}})$$
• $$L_\mathrm{hand} = \sum_{i} \mathrm{CE}(y_i^{\mathrm{hand}}, \hat{y}_{i}^{\mathrm{hand}})$$
• Graph-structural losses $L_\mathrm{LG}$ (Murali et al.):
  • Detection: standard Faster R-CNN loss (classification + box regression)
  • Edge existence: BCE for meaningful edge prediction
  • Spatial: CE for spatial relation classification

The total loss is

$$L_\mathrm{total} = L_\mathrm{LG} + \lambda_\mathrm{action} L_\mathrm{action} + \lambda_\mathrm{hand} L_\mathrm{hand}$$

with weights $\lambda_\mathrm{action} = 0.6$, $\lambda_\mathrm{hand} = 0.001$.

Training proceeds for 50 epochs per stage, using Faster R-CNN batch and optimizer defaults (weight decay $1 \times 10^{-4}$, learning rate $\approx 0.02$, batch size $\sim 2$ per GPU), on NVIDIA RTX 3090 hardware.

5. Quantitative Performance and Empirical Findings

On the Endoscapes-SG201 dataset (Shin et al., 21 Jul 2025):

Model	Graph Components	mAP
ResNet50-DetInit	None	9.7
LG-CVS (spatial only)	Yes (spatial)	18.0
SSG-Com (spa + act)	Spatial + action	23.5
SSG-Com (spa + act + hand)	+ hand role	24.2

Incorporating explicit action edges yields a +5.5 mAP increase over spatial-only graphs, and annotating hand identity adds an additional +0.7 mAP. The factorized graph structure (tools as nodes, tool–action–target as typed edges, hand as node attributes) systematically outperforms both naïve image-based recognition and spatial-only graph models.

The explicit modeling of which tool performs which action on which anatomy, and which agent operates the tool, is shown to be a decisive factor in achieving high-precision recognition, especially in the context of downstream tasks (e.g., critical view of safety assessment).

6. Modeling Considerations and Broader Implications

The SSG-Com framework demonstrates that surgical activity understanding benefits from a holistic graph embedding that makes triplet associations and operator roles first-class components. Key observations:

• Data representation: Segregation of component entities (tools/anatomy) as nodes, and contextual relations (action, spatial, hand) as edges, supports compositional reasoning and modular graph updates.
• Association mechanism: Triplet recognition accuracy is tightly coupled to the explicit encoding of action edges—models lacking this capacity exhibit severe confusion and poor recall for complex scenes.
• Human operator context: Hand identity annotation differentiates otherwise ambiguous tool usage, critical for workflow audit and skill assessment.
• Dataset dependency: Performance is sensitive to the granularity and consistency of bounding-box and interaction annotations; systematic, expert-led refinement is indispensable.

7. Future Directions and Open Issues

Despite substantial improvements, SSG-Com’s top mAP of 24.2 indicates persistent challenges:

• Interaction complexity: Surgical fields with overlapping instruments, uncommon tool–action–target combinations, or rapid exchanges between agents remain underexplored.
• Spatial grounding: The method leverages bounding-box annotations for tools and anatomy but does not explicitly handle pixel-level segmentation; finer spatial grounding would benefit from integration with state-of-the-art segmentation architectures.
• Temporal consistency: SSG-Com operates at the frame level; coherent action-recognition across sequences, or temporal scene graphs, may further advance the field.
• Annotation burden: High-quality action triplet and hand identity annotation is resource-intensive; automated or semi-automated labeling tools are required for scalability.

The graph-based triplet paradigm is central to modern surgical scene understanding, providing a robust substrate for downstream clinical analytics, skill evaluation, and intelligent robotic assistance (Shin et al., 21 Jul 2025).