Endoscapes-SG201: Surgical Scene Graph Dataset

• The dataset offers a detailed annotated framework including tool, anatomy, and action labels to enable advanced surgical scene graph analysis.
• It provides uniformly sampled video frames from 201 laparoscopic cholecystectomy procedures, supporting temporal video analysis and benchmark evaluations.
• The resource facilitates CVS assessment and fine-grained segmentation with multi-level annotations, aiding research in graph-based surgical data science.

Endoscapes-SG201 is a structured, sparsely-annotated dataset of endoscopic video frames devised for advanced research in surgical scene understanding, particularly the modeling of context-rich scene graphs and semantic segmentation in laparoscopic cholecystectomy. Drawing upon 201 real-world surgical procedures, Endoscapes-SG201 provides multiple annotation levels (object, action, hand identity, semantic mask) and is tightly linked to the Endoscapes suite of datasets. Its schema and benchmark protocols are specifically tailored for graph-based visual workspace modeling and critical view of safety assessment, offering resources for detection, relation modeling, and temporal video analysis (Shin et al., 21 Jul 2025, Murali et al., 2023, Alapatt et al., 2021).

1. Dataset Constitution and Modalities

Endoscapes-SG201 comprises 1,933 high-fidelity RGB frames sampled from the gallbladder dissection phase of 201 laparoscopic cholecystectomy (LC) procedures (drawn from the Endoscapes-BBox201 subset). Sampling is uniform within the window relevant for critical view of safety (CVS) assessment, with one annotated frame every 30 seconds, resulting in approximately 9.6 annotated frames per video. Original videos are acquired at ≈854×480 pixels and 25–30 fps, with all annotated frames rescaled to 854×480 and provided at 1 fps. Data is split by patient into training (1,212 frames from 120 videos), validation (409 frames from 41 videos), and test (312 frames from 40 videos) sets with strict subject-level separation (Shin et al., 21 Jul 2025, Alapatt et al., 2021, Murali et al., 2023).

2. Annotation Framework and Topology

Each frame is encoded as a scene graph $\mathcal{G} = (\mathcal{N}, \mathcal{E})$, unifying both spatial and surgical-action-based relations:

• Nodes ($\mathcal{N}$) represent detected tool and anatomy instances:
  • $\mathcal{N}_\text{tool} = \{(p_i, s_i, f_i)\}$, $s_i \in \mathcal{C}_\text{tool}$
  • $\mathcal{N}_\text{anatomy} = \{(p_j, s_j, f_j)\}$, $s_j \in \mathcal{C}_\text{anatomy}$
• Class vocabularies:
  • Tools $V_\text{tool} =$ {Hook, Grasper, Clipper, Bipolar, Irrigator, Scissors}
  • Anatomy $V_\text{anatomy} =$ five fine-grained structures following Endoscapes-BBox201 nomenclature (e.g., gallbladder, cystic duct, cystic artery, liver bed, surrounding tissue)
  • Actions $A =$ {Dissect, Retract, Grasp, Clip, Coagulate, Null_verb}
  • Hands $H =$ {Rt (surgeon right), Lt (surgeon left), Assi (assistant hand)}
• Edges ($\mathcal{N}$0) are twofold:
  • Spatial: Directed, derived from bounding box geometry, encoding spatial arrangements (left-right, above-below, inside-outside)
  • Action: Surgical Action Edges (SAE); binary indicator $\mathcal{N}$1 when $\mathcal{N}$2 and $\mathcal{N}$3, otherwise 0
• Annotation format: JSON per frame with nodes (id, class_label, bbox coordinates, hand_id for tools, RoI feature index) and edges (subject_id, object_id, relation_type). Flat tables encode all tool–action–target triplets, facilitating downstream multi-relation tasks.

For semantic segmentation experiments, Endoscapes-SG201 further provides 29 fine-grained semantic mask classes, including detailed anatomical and instrument subclasses, which are used for supervised and temporally-consistent training (Shin et al., 21 Jul 2025, Murali et al., 2023, Alapatt et al., 2021).

3. Detailed Statistics and Taxonomy

A statistical summary quantifies the object and relation distributions within the annotated corpus:

Category	Class / Label	Count (Frames/Instances)	Share / Type
Tool instances	Grasper	1598	51.7%
	Hook	1060	34.3%
	Clipper	219	7.1%
	Bipolar	142	4.6%
	Irrigator	69	2.2%
	Scissors	4	0.1%
Action triplets	Retract	1,429	46.3%
	Dissect	916	29.6%
	Null_verb	383	12.4%
	Clip	207	6.7%
	Grasp	95	3.1%
	Coagulate	60	1.9%
Hand identity (tools)	Surgeon Rt	1,543	49.9%
	Surgeon Lt	1,328	42.9%
	Assistant	221	7.2%

Average per-frame complexity: ≈2.6 nodes (1.6 tools + 1.0 anatomy), ≈4.6 edges (≈3 spatial + ≈1.6 action) (Shin et al., 21 Jul 2025). The semantic segmentation annotations span 29 classes, but instrument subclasses are typically collapsed into a superclass for robust training due to class imbalance (Alapatt et al., 2021).

4. Access, Licensing, and Data Organization

The Endoscapes-SG201 annotation suites and supporting code are accessible at https://github.com/ailab-kyunghee/SSG-Com (for graph annotations) and https://github.com/CAMMA-public/Endoscapes (for semantic masks and detection splits). Dataset directories are organized with standardized splits (train/val/test), COCO-style JSONs for detection and graph tasks, and PNG-encoded per-pixel labels for segmentation. Licensing for SSG-Com is not explicit in the original paper; citation is required for research use, and the GitHub repository provides the latest terms. The broader Endoscapes datasets are distributed under CC BY-NC-SA for non-commercial academic use; IRB and patient-privacy protections are strictly enforced (Shin et al., 21 Jul 2025, Murali et al., 2023, Alapatt et al., 2021).

5. Benchmark Tasks and Results

Endoscapes-SG201 is the primary benchmark for scene-graph-based action and safety assessment. Key evaluation protocols—deployed in both (Shin et al., 21 Jul 2025) and (Murali et al., 2023)—are:

• Triplet Recognition: Multi-class classification of (tool, action, target) relations (34 possible triplets); mAP on test:
  • ResNet50-DetInit (no graphs): 9.7%
  • LG-CVS (spatial graphs): 18.0%
  • SSG-Com (spatial+action): 23.5%
  • SSG-Com (spatial+action+hand): 24.2%
• Critical View of Safety (CVS) Assessment: Multi-label (three criteria) classification; mAP on Endoscapes-CVS201:
  • ResNet50-DetInit: 55.3%
  • LG-CVS: 63.2%
  • SSG-Com: 64.6%

Detection and segmentation model performance on Endoscapes-BBox201 (object detection) and Endoscapes-Seg50 (instance segmentation) provide comparable benchmarks with mAP ranging from 20–32% depending on model architecture and task (Shin et al., 21 Jul 2025, Murali et al., 2023).

6. Methodological Innovations and Use Case Scenarios

The graph schema in Endoscapes-SG201 explicitly models tool–action–target–hand quadruples, surpassing previously studied scene graphs in surgical vision by encoding operator role and fine-grained action semantics. Scene-graph-based models (e.g., SSG-Com) operationalize this representation with a structured loss:

$\mathcal{N}$4

where each component targets either spatial, functional, or operator-identity parsing. Semantic segmentation experiments additionally leverage temporally-constrained learning with autoencoder-based latent spaces (as in TCNN (Alapatt et al., 2021)), exploiting proxy supervision with sparsely labeled frames to enforce temporal and global shape consistency.

Endoscapes-SG201 directly enables research in:

• Surgical action recognition and triplet extraction
• Automated CVS assessment pipelines
• Graph-based multitask learning (spatial+temporal+functional)
• Semi-supervised segmentation with temporal constraints

7. Impact and Relationship to Broader Surgical Data Science

Endoscapes-SG201 fills a methodological and practical gap in surgical scene understanding by combining fine-grained spatial, relational, and temporal cues within an extensible, high-quality annotation protocol. Unlike larger-yet-noisier surgical video datasets, Endoscapes-SG201 emphasizes annotation fidelity, graph-based context, and benchmark repeatability, making it a de facto standard for new methods in surgical scene graphs, relation comprehension, and CVS automation. The explicit modeling of hand identity, tool-action-target relations, and spatial layout supports granular analysis required for next-generation computer-assisted surgical interventions (Shin et al., 21 Jul 2025, Murali et al., 2023, Alapatt et al., 2021).