CholecTriplet2022 Challenge

Updated 15 December 2025

CholecTriplet2022 is a challenge focused on detecting surgical action triplets that capture instrument, verb, and target associations in laparoscopic cholecystectomy videos.
The benchmark leverages 50 annotated videos with 151,000 triplet instances under weak supervision, emphasizing spatial instrument localization.
The challenge drives advances in graph-based association, multi-task learning, and real-time deployment, enhancing clinical precision in surgical analytics.

CholecTriplet2022 Challenge centers on the fine-grained recognition and localization of surgical action triplets—combinations of instrument, verb, and target—that describe instrument-tissue interactions in laparoscopic cholecystectomy procedures. The challenge introduces a benchmark (CholecT50) with 50 fully annotated videos and demands both triplet recognition and spatial localization of the instruments responsible for each action under weak supervision. This framework pushes research in surgical scene understanding toward higher clinical precision, real-time deployment, and robust association of visual, semantic, and workflow cues.

1. Problem Definition and Task Structure

CholecTriplet2022 formalizes surgical activity in endoscopic video as the detection of action triplets $\langle$ instrument, verb, target $\rangle$ for every frame, requiring both semantic association and spatial instrument localization. Formally, the model predicts for a given frame $x$ the presence of triplets $T(x) = \{\langle i,v,t\rangle\}$ with $i \in I$ (6 instrument classes), $v \in V$ (10 verb classes), and $t \in T$ (15 target anatomies). Crucially, the challenge extends recognition to detection, requiring bounding boxes around instrument tips and correct association of each box with a triplet label. Unlike standard action recognition tasks, CholecTriplet2022 adopts a weak supervision paradigm: only binary presence labels for triplet components in training, with spatial instrument annotations available exclusively on the private test set and a small validation set (Nwoye et al., 2023). This experimental design enforces the development of localization and association techniques compatible with practical annotation constraints.

2. Data Resources and Annotation Protocol

The core dataset, CholecT50, comprises 50 laparoscopic cholecystectomy videos (sampled at 1 fps, totaling ~100,900 frames and ~151,000 triplet instances), encoded as 100 valid triplet classes drawn from the full cross product of 6 instruments, 10 verbs, and 15 targets. Annotation was performed by expert clinicians using tool timelines (start/end), and action labels, with multi-stage class curation for clinical relevance and disambiguation (Nwoye et al., 2021).

The challenge split consists of 45 public trainval videos (CholecT45) with only frame-wise binary labels for triplets and components, and 5 private test videos containing both presence labels and spatial bounding box annotations for instrument tips (≈13,000 boxes), together with mappings from boxes to action triplets (Nwoye et al., 2022). Instrument tip boxes are tightly drawn, and target spatial annotation is absent in training, reflecting clinical annotation feasibility.

Recording protocols, data preprocessing (resizing to 256×448, color normalization), and augmentation strategies (random flip, color jitter, patch masking) are standardized to facilitate benchmarking and reproducibility.

3. Evaluation Protocols and Metrics

Performance is measured using established precision-recall methodologies, operating at both recognition and detection levels. The key metric is average precision for triplet detection ( $\mathrm{AP}_{IVT}$ ) with an IoU threshold of 0.5 between the predicted instrument box and the ground-truth, provided the verb and target are also correct (Nwoye et al., 2023). Additional metrics include:

$\mathrm{AP}_I$ , $\mathrm{AP}_V$ , $\mathrm{AP}_T$ : Component-wise APs.
$\mathrm{AP}_{IV}$ , $\mathrm{AP}_{IT}$ : Pairwise APs.
$\mathrm{AR}_{IVT}$ : Average recall at IoU=0.5.
Triplet Association Scores (TAS): Error breakdown—including Localize & Match (LM), partial LM, Identity Switch (IDS), Identity Miss (IDM), Miss Localization (MIL), and residual FP/FN (Nwoye et al., 2022, Nwoye et al., 2023).
Top- $K$ accuracy (fraction of ground-truth triplets ranked in top $K$ scoring predictions).

A Python library (ivtmetrics) is provided for standardized computation. Metrics are reported per video and averaged, ignoring absent classes to account for long-tailed distributions.

4. Methodological Landscape and Model Architectures

The CholecTriplet2022 challenge catalyzed the development of diverse approaches, from single-frame CNNs with weak localization to hybrid architectures leveraging detection models pre-trained on external datasets. Key methodologies include:

Weakly-Supervised Localisation: Class activation maps (CAM) for instrument tip localization under absence of bounding box labels, used by methods like Tripnet (Nwoye et al., 2020), Rendezvous (RDV) (Nwoye et al., 2021), and others.
Multi-Task Learning and Attention: Architectures such as RDV, which combine Class Activation Guided Attention Mechanism (CAGAM—channel/position attention) for verb/target recognition and Multi-Head of Mixed Attention (MHMA) for semantic triplet association.
Graph-based Approaches: MCIT-IG combines a Multi-Class Instrument-aware Transformer for per-class target embeddings with an Interaction Graph module (complete bipartite graph, GAT-based message passing) for explicit modeling of instrument-target-verb associations (Sharma et al., 2023).
Mixed-Supervision: Approaches—such as MCIT-IG—train with weak target labels for transformer stages and pseudo triplet labels for graph stages, leveraging minimal spatial annotation to maximize triplet detection.
Localization via Detection: High-performing submissions, such as ResNet-CAM-YOLOv5, employ object detectors (YOLOv5) pre-trained on cholecystectomy segmentation datasets to pseudo-label or fine-tune for precise box regression (Nwoye et al., 2023).
Distillation and Temporal Modeling: Self-distilled transformer ensembles and phase recognition as auxiliary tasks (e.g., Distilled-Swin-YOLO, MTTT, SurgNet) are incorporated to exploit temporal context and mitigate label noise.

Methods are benchmarked on publicly released codebases (PyTorch, TensorFlow), with strict adherence to provided data splits to ensure fair competition (Nwoye et al., 2022).

5. Benchmark Results and Comparative Analysis

The main results on the CholecT50 private test set are summarized as follows (Nwoye et al., 2023, Sharma et al., 2023):

Method	Supervision	AP_I (%)	AP_{IVT} (%)
ResNet-CAM-YOLOv5	Full	41.9	4.49
Distilled-Swin-YOLO	Full	17.3	2.74
Rendezvous (RDV, baseline)	Weak	60.1	6.43
MCIT+IG (ours)	Mixed (weak+pseudo)	60.1	7.32

MCIT+IG achieves the highest AP_I (60.1%) and AP_{IVT} (7.32%), outperforming both fully-supervised and weakly-supervised baselines by substantial margins, with a +13.8% relative improvement in triplet detection AP over the strong RDV baseline. Triplet-Association Scores indicate improved LM (tools localized and matched: 29.6%, +5.7% over next best) and minimized identity errors (IDS, IDM) (Sharma et al., 2023). Notably, MCIT-IG delivers this performance using fewer annotated frames than traditional detectors, demonstrating the benefit of combining minimal spatial supervision with rich class embedding and graph association.

Baseline recognition-only methods achieve higher AP_{IVT} (e.g., Distilled-Swin-YOLO at 35.0%), but performance drops sharply for detection due to localization bottlenecks—highlighting the challenge of translating semantic triplet presence into precise spatio-association under weak supervision (Nwoye et al., 2023).

6. Error Modes, Limitations, and Future Directions

Major limitations observed include:

Localization Under Weak Supervision: Most failures arise from low IoU box predictions, false associations, or missing triplets due to occlusion, poor boundary definition, and scene complexity (crowding, visual artifacts) (Nwoye et al., 2023).
Class Imbalance and Rare Triplets: Long-tailed distribution yields low AP for rare targets (e.g., falciform ligament, suture) and increases false negatives.
Temporal Consistency: Per-frame models suffer from label flicker due to absence of temporal aggregation; integrating ConvLSTM or Transformer temporal modules is suggested.
Target Localization: Absence of spatial annotation for targets limits performance; weak anatomical segmentation or multi-view geometric priors are proposed as future enablers (Nwoye et al., 2023).

Advances in semi-supervised/self-supervised box refinement, high-resolution CAMs, and fusion of context (surgical phase, multi-task learning) are anticipated to further improve both recognition and detection components (Nwoye et al., 2023, Sharma et al., 2023). The challenge also emphasizes the importance of standard reference implementations, curated splits, and open-source libraries (ivtmetrics) for consistent longitudinal benchmarking (Nwoye et al., 2022).

7. Significance and Clinical Impact

CholecTriplet2022 defines and operationalizes a new standard for surgical video scene understanding: holistic, frame-level, and spatially-grounded surgical activity modeling. Accurate triplet detection supports real-time decision support, including contextual awareness (e.g., anticipatory safety alerts, phase/step tracking), workflow analytics, intraoperative guidance, and AI-assisted skill assessment (Nwoye et al., 2023). The challenge uncovers methodological bottlenecks and catalyzes the broader adoption of robust, annotation-efficient frameworks, accelerating translation toward safety-critical integration in operating rooms.