PhaKIR Sub-Challenge: Surgical Scene Analysis

• PhaKIR Sub-challenge is a unified benchmark integrating surgical phase recognition, instrument keypoint estimation, and instrument segmentation on full-length laparoscopic videos.
• It leverages multi-center clinical data and advanced temporal modeling to assess algorithm robustness under challenges like occlusion, blur, and domain shift.
• The benchmark supports both single-task and multi-task methods, aiming to improve workflow understanding and precise instrument localization in computer-assisted surgery.

Searching arXiv for the cited PhaKIR papers and closely related challenge benchmarks to ground the article. The PhaKIR sub-challenge—“Surgical Procedure Phase, Keypoint, and Instrument Recognition”—was organized as part of the Endoscopic Vision (EndoVis) Challenge at MICCAI 2024 to benchmark three tightly related perception problems in laparoscopic surgery on a single, unified, real-world dataset: surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation. Its central premise is that robust endoscopic scene understanding should not isolate workflow understanding from instrument understanding, but should instead study procedural context together with instrument geometry and localization on full-length videos under clinically realistic variability. The benchmark is grounded in routine laparoscopic cholecystectomy and was later accompanied by a dataset publication describing the public training release (Rueckert et al., 22 Jul 2025, Rueckert et al., 9 Nov 2025).

1. Benchmark rationale and historical position

PhaKIR was created to address a gap in publicly available surgical video benchmarks. Existing resources typically covered only one task at a time—such as phase recognition, instrument segmentation, or keypoint estimation—and often lacked one or more of the following properties: complete procedures rather than short clips, multi-center acquisition, instance-aware and class-aware instrument annotation, human clinical data, and support for temporal modeling over full surgeries. The benchmark therefore unified three interrelated tasks in one dataset: surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation, with the explicit goal of supporting both single-task and multi-task methods (Rueckert et al., 9 Nov 2025).

The clinical motivation is correspondingly broad. Surgical phase recognition can support context-aware warnings, operating room management, and procedure time prediction. Instrument segmentation can enable navigation, skill assessment, and autonomous endoscope guidance. Instrument keypoints can support pose estimation, distance-to-risk-structure assessment, and more granular analysis of surgical actions and skill. Technically, the benchmark was designed to test robustness under smoke, blur, rapid motion, occlusion, tissue overlap, and bleeding, and to enable algorithmic use of long-range temporal context across entire procedures (Rueckert et al., 22 Jul 2025).

In historical context, PhaKIR extends the trajectory of EndoVis benchmarking beyond task-isolated designs. Earlier challenges such as the 2017 robotic instrument segmentation challenge focused on binary, parts, and type-based segmentation of articulated da Vinci robotic instruments on 10 porcine procedures, with mean IoU as the primary ranking metric (Allan et al., 2019). By contrast, PhaKIR targets full-length human laparoscopic cholecystectomy videos, incorporates procedural context, and couples workflow labels with instance-aware segmentation and instrument-specific keypoints on the same cases (Rueckert et al., 22 Jul 2025).

2. Dataset composition and annotation protocol

The challenge dataset comprised 13 full-length human laparoscopic cholecystectomy videos from three German surgical centers, recorded with different endoscope systems: TUM University Hospital Rechts der Isar (hospital 1, $n=9$), Heidelberg University Hospital (hospital 2, $n=2$), and Weilheim-Schongau Hospital (hospital 3, $n=2$). All recordings were monocular laparoscopic videos at $1920 \times 1080$ and 25 FPS. Each video begins when the camera is first inserted into the abdomen and ends when it is finally removed. Durations ranged from 23 to 60 minutes. The data split was video-wise, not frame-wise, to avoid leakage across train and test; the held-out test videos were Video_06, Video_08, Video_09, Video_12, and Video_14, while Video_10 was excluded before the challenge because its instrumentation differed too strongly from the rest (Rueckert et al., 22 Jul 2025).

Participants were given the full-length training videos, while test frames and labels were hidden. Frames captured outside the abdomen, for example during laparoscope cleaning, were removed for anonymization, and participants were provided with the frame numbers after which those removals occurred. The training data contained 485,875 frames total, with phase labels on all 485,875 frames and segmentation/keypoint labels on 19,435 frames at 1 fps. The hidden test data contained 323,025 frames, with corresponding hidden phase labels for all frames and segmentation/keypoint labels for 12,921 frames at 1 fps (Rueckert et al., 22 Jul 2025).

A later dataset publication described the public release of the training resource as eight complete videos from three medical centers, with the challenge test set remaining non-public. That release reports a total duration of 323:55 minutes, 486,875 raw frames, 486,875 phase-labeled frames, and 19,475 segmentation/keypoint-labeled frames in Table 1, while also noting a minor inconsistency elsewhere in the paper, where the text states 485,875 phase frames and 19,435 sampled frames. The paper therefore advises the safest reading as roughly 486k full-resolution phase-labeled frames and roughly 19.4k sampled frames for segmentation and keypoints (Rueckert et al., 9 Nov 2025).

Annotation quality was a major design point. Four annotators with medical expertise—one senior surgeon and three medically trained students—followed predefined rules and a multi-stage quality-control process. The challenge paper describes a three-stage process consisting of initial annotation plus two independent review stages by different annotators; the dataset paper further states that one annotator performed the initial annotation, the same annotator reviewed and corrected the result, and two additional team members verified the annotations sequentially. The benchmark was explicitly documented in accordance with the BIAS recommendations, including design, organization, data, assessment, participant policies, and supplementary challenge design documents (Rueckert et al., 22 Jul 2025, Rueckert et al., 9 Nov 2025).

3. Task definitions and label structures

PhaKIR benchmarked three tasks on aligned videos.

Surgical phase recognition was formulated as multi-class classification of the current surgical phase at each frame of full-length laparoscopic cholecystectomy videos. Every frame was assigned one of seven standard cholecystectomy phases adopted from Cholec80, plus an eighth “undefined” transition category used for ambiguous transition periods. The seven operative phases were preparation (P), calot triangle dissection (CTD), clipping and cutting (ClCu), gallbladder dissection (GD), gallbladder packaging (GP), cleaning and coagulation (ClCo), and gallbladder retraction (GR). Ground truth was created by timestamping intervals during video review and assigning the corresponding phase to every frame in that interval; undefined frames were not evaluated (Rueckert et al., 22 Jul 2025).

Instrument instance segmentation was defined as multi-class, multi-instance, pixel-wise segmentation of visible surgical instruments, with correct distinction both between classes and between multiple instances of the same class. Annotations were produced in CVAT using manually drawn contours over visible instrument regions only; no inferred occluded parts were labeled. Nineteen instrument classes were annotated:

• Argonbeamer
• Bipolar-Clamp
• Blunt-Grasper
• Blunt-Grasper-Curved
• Blunt-Grasper-Spec
• Clip-Applicator
• Dissection-Hook
• Drainage
• Grasper
• HF-Coag.-Probe
• Hook-Clamp
• Needle-Probe
• Overholt
• Palpation-Probe
• PE-Forceps
• Scissor
• Sponge-Clamp
• Suction-Rod
• Trocar-Tip

Masks were encoded so that red and green channels specified class identity and blue specified the instance within class; thus two instruments of the same class in one frame had the same $R,G$ values but different $B$ values. One instrument class appeared in the test set but not in training and was excluded from evaluation for fairness (Rueckert et al., 22 Jul 2025, Rueckert et al., 9 Nov 2025).

Instrument keypoint estimation required localization of tool-specific keypoints together with instrument class and instance discrimination under variable keypoint cardinality and partial visibility. Depending on the tool type, between two and four keypoints were annotated. The semantic points were EP (EndPoint), where the instrument enters the image border; SP (ShaftPoint), the shaft-to-tip junction or transition point; and T1/T2, the tip keypoints. Visibility followed the COCO protocol and was encoded as visible, hidden or outside-image/unannotated in the challenge paper, and as visible, occluded, or not available in the dataset paper. Unlike segmentation, keypoint annotation permitted inference from temporal continuity, so annotators could use neighboring frames to estimate hidden locations (Rueckert et al., 22 Jul 2025, Rueckert et al., 9 Nov 2025).

The keypoint schema was instrument-specific. Four keypoints $(T1, T2, SP, EP)$ were used for Bipolar-Clamp, Blunt-Grasper, Blunt-Grasper-Curved, Blunt-Grasper-Spec., Clip-Applicator, Grasper, Hook-Clamp, Overholt, PE-Forceps, Scissor, and Sponge-Clamp. Three keypoints $(T1, SP, EP)$ were used for Argonbeamer, Dissection-Hook, HFcoag-Probe, and Suction-Rod. Two keypoints $(T1, EP)$ were used for Drainage, Needle-Probe, Palpation-Probe, and Trocar-Tip (Rueckert et al., 9 Nov 2025).

4. Assessment design and ranking methodology

The assessment protocol was explicitly designed to address class imbalance and variable video length. Metrics were computed per surgical phase or instrument class for each annotated frame. Because instrument frequencies were highly imbalanced, metric values were first averaged across all frames of a video for each phase or class, then these class-specific values were averaged with equal weighting to obtain a video-level score, and video-level scores were then averaged equally across test videos. To increase robustness, bootstrapping with 10,000 iterations was applied; each bootstrap drew as many samples as there were annotated test frames, with replacement, and final metric values were taken as the average over bootstrap runs (Rueckert et al., 22 Jul 2025).

Because the three tasks used different metrics, the challenge did not aggregate raw scores directly. Instead, for each metric, teams received a metric-specific rank; these ranks were averaged equally to produce an overall ranking value, and final task ranking was based on this mean rank. This ranking rule favored consistency across metrics rather than isolated excellence on a single score (Rueckert et al., 22 Jul 2025).

For Task 1, the evaluated labels were the seven named operative phases, with “undefined” frames ignored in scoring. The metrics were per-class F1-score and Balanced Accuracy (BA) as an overall multi-class measure. The rationale given by the organizers was that F1 balances false positives and false negatives, whereas BA gives equal importance to short and long phases (Rueckert et al., 22 Jul 2025).

For Task 2, the formal task was multi-class, multi-instance, pixel-wise segmentation of visible instruments. The metrics were Dice Similarity Coefficient (DSC), mean Average Precision over IoU thresholds $\mathrm{mAP}_{\mathrm{IoU}}$, and 95% Hausdorff Distance. Predicted and ground-truth instances were matched using the Hungarian maximum matching algorithm based on IoU (Rueckert et al., 22 Jul 2025).

For Task 3, evaluation used COCO-style $n=2$0 based on Object Keypoint Similarity. Only visible ground-truth keypoints were included in OKS via the visibility indicator. If predictions had too many keypoints, only those corresponding to the ground-truth class were evaluated; if too few, placeholder coordinates $n=2$1 were appended. For tools with two tip keypoints, the Hungarian algorithm determined the optimal one-to-one correspondence; if too many tip keypoints were predicted, only the first predicted tip keypoint was evaluated (Rueckert et al., 22 Jul 2025).

BIAS-aligned challenge operations included explicit video-level train/test separation, hidden test labels, specification of allowed training resources, Docker-based submission and execution, metric justification using the Metrics Reloaded framework, bootstrap-based robustness analysis, and transparent reporting of timeline, ethics approval, data access restrictions, and code/data availability (Rueckert et al., 22 Jul 2025).

5. Participation and empirical results

Across all tasks, the benchmark attracted the highest participation among EndoVis 2024 sub-challenges, and the later dataset paper reports that 66 registered teams worldwide downloaded and inspected the data; rare annotation errors reported by participants were corrected, and no systematic errors were identified during challenge use (Rueckert et al., 22 Jul 2025, Rueckert et al., 9 Nov 2025).

For phase recognition, seven teams participated: augi, hanglok, jmees_inc., ryze, smartlab_hkust, uniandes24, and yipingli. All seven incorporated temporal information. The winning method, uniandes24, used MuST (Multi-Scale Transformers for Surgical Phase Recognition), a specialized two-stage architecture combining a Multi-Term Frame Encoder with a Temporal Consistency Module and an MViT backbone initialized from Kinetics-400 and Cholec80. It achieved the best results on both metrics: F1-score $n=2$2 with 95% CI $n=2$3, and BA $n=2$4 with 95% CI $n=2$5. Second place, jmees_inc., used EVA-02 with temporal heuristics including EMA weight updates and rule-based post-processing, reaching F1 $n=2$6 and BA $n=2$7. Third place was shared by yipingli and smartlab_hkust, each with mean rank 3.5 (Rueckert et al., 22 Jul 2025).

For instrument instance segmentation, nine teams participated: augi, goncalo, hanglok, jmees_inc., kist_harilab, floor9, recogna, sk, and uniandes24. The winning system, jmees_inc., used Mask2Former with a Swin-Base backbone and pseudo-labels generated from unlabeled PhaKIR video frames using SAM 2. It ranked first overall with average rank 1.67 and obtained DSC $n=2$8 $n=2$9, $n=2$0 $n=2$1, and 95% HD $n=2$2 $n=2$3. uniandes24 ranked second with DSC $n=2$4, $n=2$5, and the best 95% HD at $n=2$6, using a MATIS-derived Mask2Former-style pipeline augmented with a Temporal Consistency Module inspired by TAPIS. augi ranked third with DSC $n=2$7, $n=2$8, and 95% HD $n=2$9, using Mask2Former with a Swin-L backbone and a Presence-aware Instrumental Segmentation module for instrument-free frame detection (Rueckert et al., 22 Jul 2025).

For instrument keypoint estimation, only two teams submitted. sds-hd won clearly with $1920 \times 1080$0 and 95% CI $1920 \times 1080$1, using yolov8x-pose-p6 and padding annotations to standardize keypoint count across tool classes before filtering padded keypoints during postprocessing. alvaro scored $1920 \times 1080$2 with CI $1920 \times 1080$3, using YOLOv8-x extended to jointly predict boxes, classes, and keypoints with uncertainty estimation via Deep Deterministic Uncertainty ideas and spectral normalization in each convolutional block (Rueckert et al., 22 Jul 2025).

Several task-specific empirical patterns were prominent. In phase recognition, clipping and cutting (ClCu) was hardest, with the lowest average phase-wise accuracy, $1920 \times 1080$4, whereas preparation was easiest, with average accuracy $1920 \times 1080$5. Common phase confusions included CTD $1920 \times 1080$6 GD, ClCu $1920 \times 1080$7 CTD and GD, GD $1920 \times 1080$8 ClCo and GP, GP $1920 \times 1080$9 GR, ClCo $R,G$0 GD, and GR $R,G$1 GP or Preparation. Test-video analysis showed domain shift: video 9 was easiest overall and video 14 was hardest; performance was best on hospital 1 videos, which also dominated the training data (Rueckert et al., 22 Jul 2025).

6. Methodological interpretation, limitations, and legacy

The challenge results gave a nuanced answer to the benchmark’s central hypothesis about temporal and contextual modeling. In phase recognition, temporal context was universally used and clearly beneficial, and the best results came from methods with sophisticated, task-specific temporal modeling. In segmentation, temporal context was rare and not clearly advantageous; only two teams used temporal information meaningfully, and one of the winning teams explicitly removed temporal logic from the final system after ablation showed harm. In keypoint estimation, no participating method used temporal context at all. This suggests that the benchmark succeeded in enabling temporally aware study, but the submitted methods did not demonstrate a general gain from jointly modeling phase and instrument information across tasks (Rueckert et al., 22 Jul 2025).

A related misconception is that PhaKIR already established the superiority of joint multi-task learning. The challenge paper explicitly states that the submitted methods were almost entirely task-specific, and it does not report a submission that jointly learned phase and instrument outputs from the same model. Direct empirical evidence that joint phase–instrument modeling improved robustness, interpretability, or generalization is therefore not established by the competition results. The significance lies instead in the benchmark design: to the organizers, PhaKIR is one of the first datasets to make such integrated analysis possible on the same full-length, multi-center surgical cases (Rueckert et al., 22 Jul 2025).

Architecturally, the benchmark produced clear but task-dependent trends. For phase recognition, the organizers interpreted the results as showing that highly specialized phase-recognition architectures with advanced temporal modeling outperform more generic image classifiers or simpler recurrent add-ons, and they noted an architectural shift relative to earlier challenges: whereas older phase-recognition challenges were dominated by CNNs, here all but one method were transformer-based. For segmentation, transformer-based Mask2Former variants dominated the leaderboard, while CNN-based U-Net and Mask R-CNN solutions lagged behind. For keypoint estimation, the organizers concluded that the overall low scores from both teams reflected a mismatch between generic human pose estimation architectures and the more complex challenge setting of multiple instrument classes with different keypoint schemas (Rueckert et al., 22 Jul 2025).

The benchmark also exposed persistent failure modes. True instance separation remained difficult: even top segmentation methods often produced correct class masks for multiple tools but failed to distinguish separate instances of the same class, particularly for “blunt-grasper-spec.” Rare instruments were much more likely to be missed or misclassified. Smoke, blur, bleeding, occlusions, and center shift degraded performance substantially, and none of the submissions generalized robustly across all hospitals. The inter-center performance gap was one of the clearest signs of domain shift, and the center distribution was unbalanced, with hospital 1 contributing most training and test cases (Rueckert et al., 22 Jul 2025).

Several limitations were acknowledged explicitly. Despite many frames, there were only 13 patient cases in the challenge setup, and only 8 surgeries in the public training release. The benchmark focused exclusively on laparoscopic cholecystectomy, all videos shared the same recording resolution and frame rate, and all data came from German medical centers. Structural metadata for more systematic analysis of difficult sequences were lacking, and segmentation/keypoint annotation was sparse at 1 fps rather than dense at every frame. The public dataset release is hosted on Zenodo, is publicly available upon request, and is distributed under CC-BY-NC-SA (Rueckert et al., 22 Jul 2025, Rueckert et al., 9 Nov 2025).

The organizers’ recommendations for future work were correspondingly concrete: improve robustness and domain generalization across institutions; collect larger and more diverse datasets spanning more centers, surgeries, and equipment; explore temporal modeling more deeply for segmentation and keypoint estimation; develop instrument-specific keypoint methods rather than relying on generic human pose estimators; improve handling of rare instruments; and perform more systematic analysis of failure modes. A plausible implication is that PhaKIR’s principal contribution is infrastructural rather than conclusively algorithmic: it establishes a benchmark substrate for context-aware, temporally informed surgical scene understanding, while simultaneously showing that robust real-world deployment in RAMIS remains unresolved (Rueckert et al., 22 Jul 2025).