---
title: 'CholecT50: Surgical Action Triplet Benchmark'
url: https://www.emergentmind.com/topics/cholect50-dataset
type: topic
---

# CholecT50: Surgical Action Triplet Benchmark

The CholecT50 dataset is a large-scale, clinically acquired corpus of laparoscopic cholecystectomy videos comprehensively annotated at the frame level with fine-grained “surgical action triplet” labels. Each annotation captures the fundamental interaction between an instrument, a surgical verb or action, and the anatomical target—formalized as $\langle \text{instrument}, \text{verb}, \text{target} \rangle$. Designed as a benchmark dataset for computer vision research in surgical workflow analysis, CholecT50 has become the reference resource for robust, fine-grained activity recognition, zero-shot reasoning, and, via extensions, instance segmentation and generative modeling in surgical environments [2109.03223][2204.04746][2204.05235][2406.16039][2407.09230][2503.19670].

## 1. Dataset Composition and Annotation Schema

CholecT50 consists of fifty full-procedure laparoscopic cholecystectomy videos, primarily sourced from the Cholec80 dataset (45 videos) and supplemented with five additional in-house procedures. Video resolutions range from 480×854 to 1920×1080 pixels and are down-sampled to 1 frame per second (fps) for annotation. The total dataset comprises approximately 100,900–101,000 annotated frames, corresponding to roughly 28 hours of recorded operative time [2109.03223][2204.04746][2407.09230].

### Annotation Structure

Each frame is labeled with one or more triplets capturing fine-grained tool–anatomy interactions:

- **Instrument**: e.g., grasper, hook, scissors, clipper, bipolar, irrigator
- **Verb (Action)**: e.g., grasp, dissect, retract, coagulate, clip, cut, aspirate, irrigate, pack, null
- **Target (Tissue/Structure)**: e.g., gallbladder, cystic duct, cystic artery, blood vessel, fluid, liver, omentum, peritoneum, abdominal wall/cavity, gut, specimen bag, adhesion, null

After clinical curation, 100 distinct $\langle \text{instrument}, \text{verb}, \text{target} \rangle$ triplet classes are retained, representing plausible operative actions. The annotation format encodes instruments, verbs, targets, and triplet classes as multi-hot binary vectors. On average, each frame contains $1.6$ annotated triplets [2109.03223][2204.04746].

### Class Balance

CholecT50 exhibits heavy class imbalance. For example, “grasper” appears in roughly $71\%$ of frames and “specimen bag” in roughly $6\%$; the most frequent triplet (“grasper, retract, gallbladder”) occurs over $48,000$ times, while others are rare, producing a Gini coefficient of $\approx 0.76$ on triplet frequencies [2204.04746].

## 2. Data Splitting Protocols and Accessibility

Standardized data splits have been established to facilitate rigorous benchmarking and reproducibility. The splits include [2204.05235][2204.04746]:

| Split Type        | Train | Val | Test | Purpose                                 |
|-------------------|-------|-----|------|-----------------------------------------|
| Rendezvous (RDV)  | 35    | 5   | 10   | Primary canonical split                 |
| Challenge Split   | 45    | -   | 5    | CholecTriplet2021 challenge (+ val)     |
| 5-Fold CV         | 5×10  | -   | -    | Cross-validation over the full 50 videos|

Splits are stratified by procedure length to ensure balanced distribution of case complexity. Annotations, preprocessing scripts, and evaluation code are available through the CAMMA team and Grand Challenge web portals, subject to a research-only license [2204.04746].

## 3. Benchmarking Protocols and Evaluation Metrics

CholecT50 underpins the evaluation of fine-grained recognition systems, with all principal metrics formalized and implemented in the ivtmetrics Python library [2204.05235].

### Recognition Metrics

- **Average Precision (AP)**: Calculated per class as the area under the precision-recall curve. 
    - $\mathrm{AP}_I$, $\mathrm{AP}_V$, $\mathrm{AP}_T$: for instrument, verb, and target, respectively.
    - $\mathrm{AP}_{IVT}$: triplet AP, the main metric.
- **Mean Average Precision (mAP)**: Mean of AP across all triplet classes: 
  $$
  \mathrm{mAP} = \frac{1}{N} \sum_{i=1}^N \mathrm{AP}_i
  $$
- **F1-Score**: 
  $$
  F_1 = \frac{2pr}{p + r}
  $$
  where $p$ is precision and $r$ is recall.

### Association and Localization Metrics

- **Triplet Association Scores (TAS)**: Quantifies joint recognition and localization errors (including LM, pLM, IDS, IDM, MIL, RFP, RFN categories) [2204.05235].
- **Detection AP and IoU**: For segmentation/localization extensions, AP is computed over predicted bounding boxes or masks with standard IoU thresholds.

## 4. Specialized Zero-Shot and Base-to-Novel Benchmarks

A key challenge addressed using CholecT50 is zero-shot recognition of novel triplets. The “base-to-novel” protocol divides observed triplets into exclusive train (base) and test (novel) sets. Two established splits, used especially for vision-language modeling studies, are [2503.19670]:

- **Unseen-Target (UT)**: Novel triplets have previously unseen targets (e.g., cystic artery, peritoneum).
    - Base: $36$ triplets, $\sim45$k train, $\sim3.5$k val, $\sim12$k test frames.
    - Novel: $18$ triplets, $\sim700$ val, $\sim1.9$k test frames.
- **Unseen-Instrument-Verb (UIV)**: Novel triplets have unobserved instrument-verb pairs.
    - Base: $28$ triplets, $\sim18$k train, $\sim1.7$k val, $\sim4.3$k test frames.
    - Novel: $21$ triplets, $\sim1.0$k val, $\sim3.7$k test frames.

For both, any frame containing both base and novel triplets is excluded to prevent data leakage. Models are trained exclusively on base triplets and evaluated zero-shot on novel triplets. Performance is reported as F1@3 (top-3 predictions per frame) and mAP for base, novel, and harmonic mean (HM) scores [2503.19670].

#### Example Reported Results (fine-CLIP, Unseen-Target setting):

| Metric      | Base   | Novel  | HM     |
|-------------|--------|--------|--------|
| F1@3 (%)    | 61.71  | 39.78  | 48.38  |
| mAP (%)     | 31.72  | 32.17  | 31.95  |

## 5. Extensions: Segmentation and Generative Modeling

While the original CholecT50 release does not include segmentation labels, CholecInstanceSeg augments a subset of CholecT50 with pixel-precise mask annotations for tool instance segmentation [2406.16039]. CholecInstanceSeg provides over 30,998 frames from CholecT50, annotated for up to seven tool classes (including “Snare”) and includes rigorous quality controls (manual review, human-in-the-loop correction, inter-annotator PQ $>90$).

Instance segmentation methods such as Mask R-CNN and Mask2Former establish baseline reference points on this extension (e.g., Mask2Former achieves $0.682$ mAP). This enables research into simultaneous triplet recognition, spatial localization, and object segmentation within surgical video scenes.

The dataset is also used for text-to-image modeling in the surgical domain. Using CholecT50’s action triplet annotations, studies have formulated conditional image synthesis setups with T5-based triplet text embeddings and developed instrument-centric class balancing methods to address heavy instrument imbalance [2407.09230]. For instance, “Surgical Imagen” achieves FID $3.7$ and CLIP $26.8\%$ in photorealistic, action-aligned sample generation conditioned on triplet prompts.

## 6. Dataset Challenges and Limitations

CholecT50 presents multiple challenges:

- **Long-Tailed Distribution**: A handful of common triplets dominate, while many are rare or absent. Instrument frequency varies drastically (e.g., “grasper” $\approx42\%$, “specimen bag” $\approx0.2\%$) [2407.09230]. Class balancing via log-inverse weighting addresses some model training biases:
  $$
  w_i = \frac{1}{\log(1 + N_i)}
  $$
- **Ethics and Scope**: Patient privacy and the necessity for expert annotation restrict expansion and broader distribution [2407.09230]. Annotations cover only cholecystectomy, with rare or complex events underrepresented.
- **Modeling Complexity**: Fine-grained triplet recognition is challenging for vision-language and end-to-end models due to contextual ambiguity, unbalanced classes, and strong combinatorial novelty in real-world settings [2503.19670].

Methods that exploit the two-level annotation hierarchy ($\langle \text{instrument}, \text{target} \rangle$ as root; $\langle \text{instrument}, \text{verb}, \text{target} \rangle$ as leaf) and leverage semantic clustering, specialized adaptation, and hierarchical loss functions demonstrate improved generalization and discrimination [2503.19670].

## 7. Impact and Community Adoption

CholecT50 constitutes the foundation for major surgical action recognition challenges, including CholecTriplet2021 and the benchmarking of models such as Tripnet, Attention Tripnet, and Rendezvous architectures. These have enabled fair comparison and global progress tracking in surgical workflow analysis [2204.04746][2109.03223]. The dataset’s structure and benchmarking tools (ivtmetrics) have set a standard for rigorous, component-disentangled, and association-aware evaluation in fine-grained visual scene understanding for the operating room environment.

The open-access availability of extension datasets (e.g., CholecInstanceSeg), comprehensive metrics, and wide adoption across multiple vision-language and generative modeling paradigms affirm CholecT50’s status as an indispensable resource in surgical data science [2406.16039][2503.19670][2407.09230].

Source: https://www.emergentmind.com/topics/cholect50-dataset