---
title: 'CholecT50: Fine-Grained Laparoscopic Triplet Dataset'
url: https://www.emergentmind.com/topics/cholect50
type: topic
---

# CholecT50: Fine-Grained Laparoscopic Triplet Dataset

CholecT50 is a fine-grained laparoscopic cholecystectomy video dataset in which surgical activity is formalized as frame-level, multi-label triplets of the form \(\langle \text{instrument}, \text{verb}, \text{target} \rangle\). It was introduced to move surgical workflow analysis beyond coarse labels such as phases or steps and toward direct modeling of tool–tissue interactions, and later work characterizes it as the most widely adopted dataset for surgical triplet research [2109.03223] [2506.01130]. Within the surgical computer-vision literature, CholecT50 has served both as a canonical benchmark for triplet recognition and as a substrate for challenge protocols, detection extensions, multi-task learning, vision-language evaluation, text-to-image generation, question answering, and action planning [2204.05235].

## 1. Origins, scope, and semantic ontology

CholecT50 was introduced alongside the Rendezvous model as an extension and refinement of the earlier CholecT40 dataset [2109.03223]. It contains **50 videos** of **laparoscopic cholecystectomy**, with the original paper describing the corpus as comprising **45 videos from Cholec80** and **5 videos from an in-house dataset**, all from the same procedure type [2109.03223]. The later benchmarking paper further clarifies that **CholecT45** is the first public release of CholecT50 and contains the **45 of the 50 videos** that were publicly available, while the remaining five videos corresponded to the held-out challenge test set [2204.05235].

The dataset’s central abstraction is the surgical action triplet,
\[
\langle \text{instrument}, \text{verb}, \text{target} \rangle,
\]
intended to capture the acting tool, the action being performed, and the anatomical or operative target [2109.03223]. The component vocabularies reported for CholecT50 are **6 instrument classes**, **10 verb classes**, and **15 target classes**, combined into **100 valid triplet classes** [2109.03223] [2204.05235]. The benchmarking paper also reports **7 phase classes** in the dataset statistics, reflecting the dataset’s overlap with broader laparoscopic cholecystectomy annotation ecosystems [2204.05235].

The triplet vocabulary was not obtained by taking the full combinatorial product of instruments, verbs, and targets. The original paper states that, although the raw component spaces would permit \(6\times10\times15=900\) combinations and roughly 300 combinations were observed in the raw annotations, the final **100 triplet classes** were clinically curated in two stages: semantically equivalent labels were grouped into super-classes, and candidate triplets were ranked by clinicians according to plausibility and utility [2109.03223]. This curation makes CholecT50 more than a naïve label product: it is a clinically filtered ontology of laparoscopic tool–tissue interactions.

The dataset was explicitly motivated by the limitations of coarser workflow tasks. In the original framing, triplet recognition is the only workflow-analysis formulation that aims at “truly fine-grained and comprehensive information” about surgical activity [2109.03223]. That claim explains why CholecT50 became a reference point not only for triplet recognition, but also for later work seeking semantically richer surgical representations.

## 2. Annotation regime and reported corpus statistics

CholecT50 is annotated at **1 frame per second**, and its labels are **frame-level** and **multi-label**: a frame may contain multiple simultaneous action triplets [2109.03223] [2204.04746]. The original paper describes annotation as being performed by **two surgeons** using **Surgery Workflow Toolbox-Annotate**, with action intervals marked on the timeline and labeled by instrument, verb, and target [2109.03223]. An action ended when the instrument left the frame or when either the verb or the target changed; out-of-frame actions were not annotated, and frames recorded outside the patient’s body were zeroed out [2109.03223].

A crucial characteristic of the original release is that it is a **frame-level weak-label dataset** rather than a densely localized interaction dataset. Each frame has binary presence labels for one or more triplet classes, but the original paper notes that CholecT50 does **not** distinguish multiple instances of the same triplet class within one frame [2109.03223]. This weakly supervised spatial regime later became a major source of methodological difficulty for triplet detection work.

Reported corpus totals differ across papers.

| Source | Reported scale | Notes |
|---|---|---|
| Rendezvous [2109.03223] | 50 videos, 100,863 annotated frames, 161,005 triplet instances | Extension of CholecT40 |
| Benchmarking protocol [2204.05235] | 50 videos, 100.9K frames, 151.0K triplet instances, 13.0K bounding boxes | Also reports 7 phase classes |
| CholecTriplet2021 [2204.04746] | 50 videos, 100.9K frames, 161K triplet instance labels | Challenge-scale summary |

These discrepancies are part of the published record rather than a single reconciled canonical count. A plausible implication is that different papers are summarizing slightly different annotation exports or benchmark views of the same underlying corpus.

Beyond full triplets, the challenge literature emphasizes that CholecT50 also exposes component-wise labels for instruments, verbs, and targets, enabling component and association evaluation in addition to full-triplet scoring [2204.04746]. At the same time, target recognition emerges repeatedly as the hardest component. The CholecTriplet2021 analysis reports that target recognition had the lowest overall performance, and identifies cystic artery, blood vessels, peritoneum, omentum, and gut as especially difficult targets [2204.04746]. This establishes one of the dataset’s enduring empirical signatures: instruments are comparatively easy, verbs intermediate, and targets the main bottleneck.

## 3. Benchmark protocols and evaluation methodology

A substantial part of CholecT50’s significance comes from the effort to standardize how it is used. The paper “Data Splits and Metrics for Method Benchmarking on Surgical Action Triplet Datasets” defines three important split families for CholecT50 and recommends **5-fold cross-validation** as the primary research protocol [2204.05235].

| Protocol | Structure | Purpose |
|---|---|---|
| RDV split | 35 train / 5 val / 10 test videos | Reproducibility with Rendezvous |
| CholecTriplet challenge split | 45 public trainval videos / 5 hidden test videos | Challenge evaluation |
| Official 5-fold CV | 5 folds over all 50 videos | Recommended research benchmark |

The **RDV split** preserves the train/validation/test setup used in the original Rendezvous paper [2109.03223]. The **challenge split** underlies the CholecTriplet2021 and CholecTriplet2022 competitions, where **45 public videos** form the development set and **5 hidden videos** are reserved for official testing [2204.04746] [2302.06294]. The **official 5-fold cross-validation split** was introduced to reduce bias from class imbalance and missing classes in single held-out test sets; the folds were constructed by first sorting videos by procedure duration as a proxy for difficulty and then distributing videos across folds so that difficulty was more balanced [2204.05235].

Evaluation is centered on **Average Precision (AP)**. The benchmark papers define component metrics
\[
AP_I,\; AP_V,\; AP_T,
\]
association metrics
\[
AP_{IV},\; AP_{IT},
\]
and full-triplet recognition
\[
AP_{IVT},
\]
with \(AP_{IVT}\) treated as the primary metric [2204.04746] [2204.05235]. Recognition AP is computed as the area under the precision–recall curve, and the standardized tooling computes both video-level and global variants, with the benchmark tables using video-level aggregation [2204.05235].

For methods that output only triplet scores, component predictions are derived by filtering the triplet outputs: the score of an instrument, verb, or target is obtained by taking the maximum triplet score over all triplets containing that component [2109.03223] [2204.05235]. This disentangling procedure became formalized in the released **`ivtmetrics`** library, which supports recognition, detection, and component extraction for CholecT50-style evaluation [2204.05235].

Challenge-style reporting introduces one further convention: only **94 valid triplet classes** are scored, because **6 null triplet classes** are excluded from ranking [2204.04746] [2204.05235]. This matters when comparing numbers across papers, since some works report over the nominal 100-class space and others over the challenge-effective 94-class space.

The 2022 detection challenge extends this protocol by adding spatial evaluation. It uses **45 full training videos**, **5 short validation clips**, and **5 full hidden test videos**, with **instrument-tip bounding boxes** provided for validation and hidden test evaluation but not for the full training split [2302.06294]. That design operationalizes triplet detection as weakly supervised localization and association rather than fully supervised detection training.

## 4. Recognition and detection performance landscape

CholecT50 has been used to chart the progression from early triplet recognition baselines to more structured association models, pretraining-based approaches, and detection systems. The CholecTriplet2021 challenge paper remains the clearest benchmark snapshot of the original recognition era: across **24 evaluated models**, reported \(AP_{IVT}\) ranged from **4.2% to 38.1%**, with a mean triplet performance of **\(23.3 \pm 9.9\%\)** [2204.04746]. The official winner, **Trequartista**, reached **38.1% \(AP_{IVT}\)**, while the strongest organizer baseline, **Rendezvous**, achieved **32.7%**; an ensemble of top methods later pushed performance to **42.4% \(AP_{IVT}\)** [2204.04746].

Under the later standardized **official 5-fold cross-validation** protocol, reproduced baselines are lower but more comparable across studies. The benchmarking paper reports **Tripnet** at **\(25.3 \pm 2.4\) \(AP_{IVT}\)**, **Attention Tripnet** at **\(27.2 \pm 2.9\)**, and **Rendezvous** at **\(29.4 \pm 2.5\)**, with Rendezvous also leading on \(AP_{IV}\) and \(AP_{IT}\) [2204.05235]. This standardized CV view is important because it reduces the optimism of favorable single-split results and exposes nontrivial fold variance.

Later work improved substantially on this baseline regime. **DiffTriplet** recast triplet recognition as conditional denoising in a joint space of triplets and components, and on CholecT50 cross-validation reported **\(35.1 \pm 2.7\) \(AP_{IVT}\)** using Rendezvous features and **\(40.3 \pm 2.5\)** using SDSwin features [2406.13210]. Its ablations show that joint-space learning and association guidance both contribute measurable gains over a diffusion model operating only in triplet space [2406.13210]. **UniSurg**, a V-JEPA-based video-native foundation model evaluated with attentive probing on a frozen backbone, reported **AP-I 91.55**, **AP-V 57.72**, **AP-T 48.18**, **AP-IV 40.39**, **AP-IT 43.47**, and **AP-IVT 39.54**, leading all six metrics in its comparison table [2602.05638]. **SurgFM**, pretrained on Surg-3M, reported **\(64.2 \pm 1.7\) mAP** for CholecT50 action recognition in a frame-wise protocol and outperformed Tripnet, Rendezvous, and Attention Tripnet in that setting [2503.19740]. On a separate fixed **40/5/5** CholecT50 split, **MEJO-B** is reported to outperform **CoT** by **1.6% in \(AP_{IVT}\)** and to achieve the best results across **\(AP_I, AP_V, AP_T, AP_{IV}, AP_{IT}, AP_{IVT}\)** [2509.12893].

Detection remained much harder. The CholecTriplet2022 challenge extended the problem from recognition to **triplet detection**, requiring instrument-tip localization plus triplet assignment [2302.06294]. In that leaderboard, the weakly supervised **RDV-Det** baseline achieved only **0.24 \(AP_{IVT}\)**, while the strongest challenge method, **ResNet-CAM-YOLOv5**, reached **4.49 \(AP_{IVT}\)** [2302.06294]. The follow-up **MCIT-IG** method, which combines instrument-aware target embeddings with a bipartite interaction graph, reported **60.1 \(AP_I\)**, **66.6 \(AR_I\)**, **7.32 \(AP_{IVT}\)**, and **10.26 \(AR_{IVT}\)** on the private challenge test set, and the authors state that it would have ranked first on the CholecTriplet2022 leaderboard [2307.09548]. This gap between recognition and detection is a persistent characteristic of CholecT50: recognizing that a triplet is present is materially easier than localizing and associating the operative actor under weak supervision.

## 5. Derivative resources and repurposed task formulations

CholecT50 has generated a large family of derivative resources and alternative task formulations. Some extend the dataset itself; others repackage its labels into new modalities or learning problems.

| Resource or protocol | Transformation of CholecT50 | Resulting task |
|---|---|---|
| CholecT45 [2204.05235] | Public 45-video subset of CholecT50 | Public benchmarking |
| CholecInstanceSeg [2406.16039] | New instance masks and IDs on CholecT50/Cholec80 frames | Tool instance segmentation |
| CholecT50-SAP [2503.18296] | Consecutive same-action frames grouped into clips; 5-action label space | Surgical action planning |
| CholecT50-Caption-VQA [2603.29962] | Action-consistent clips with clinician captions converted to QA | Video question answering |
| Surgical Imagen training set [2407.09230] | Triplets linearized into text prompts | Surgical text-to-image generation |

**CholecInstanceSeg** is the most direct annotation extension. It reuses CholecT50 frames and adds **semantic masks** and **instance IDs** for tool instance segmentation. The CholecT50-derived portion comprises **15 fully annotated sequences** with **28,317 frames** and **45,221 tool instances**, plus **35 sparsely annotated sequences** with **2,681 frames** and **4,098 tool instances** [2406.16039]. This directly addresses a capability the original CholecT50 did not provide: dense instance-level tool segmentation.

**CholecT50-SAP** repurposes the corpus for forward-looking decision support rather than retrospective recognition. It is constructed by grouping consecutive frames sharing the same action into clips, restricting attention to segments around Calot’s triangle dissection, duct and vessel clipping, and dissection from the liver bed, and collapsing the label space to
\[
\{\text{Aspiration, Coagulation, Dissection, Tissue Retraction, Vessel Clipping}\}.
\]
The resulting dataset contains **225 samples**, split into **35 training videos / 168 samples** and **15 testing videos / 57 samples** [2503.18296]. Here, CholecT50 becomes a substrate for next-action prediction and prompt-based planning with large language models.

**CholecT50-Caption-VQA**, one component of **CholeVidQA-32K**, segments CholecT50 videos by action continuity and then uses clinician-authored captions plus GPT-5.1 to generate question–answer pairs. This subset contains **1,544 video segments** and **18,368 QA pairs**, covering **Tool Perception**, **Action Perception**, **Anatomical Structure Perception**, **Surgical Scene Description**, and **Action Rationale Reasoning** [2603.29962]. In this formulation, CholecT50 no longer functions as a triplet classifier benchmark but as a foundation for temporally grounded surgical VQA.

Other papers repurpose CholecT50 labels as language prompts. **Surgical Imagen** uses approximately **101K image-text pairs** from CholecT50, with triplet-based prompts such as short linearized captions, to train a diffusion-based text-to-image generator [2407.09230]. **MML-SurgAdapt** integrates CholecT50’s **100 triplet labels** with **7 phase labels** from Cholec80 and **3 CVS labels** from Endoscapes2023 into a shared **110-label** CLIP-based multi-task space, using **Single Positive Multi-Label (SPML)** learning to cope with missing annotations [2507.05020]. **fine-CLIP** introduces a CholecT50-specific base-to-novel benchmark with **Unseen-Target** and **Unseen-Instrument-Verb** settings, using held-out triplet subsets to test compositional zero-shot transfer [2503.19670]. **DeepSeek**-based evaluation uses CholecT50 only for **Single Phrase QA**, reformulating images into closed-set questions over **5 instrument**, **8 target**, and **13 motion** phrases rather than evaluating standard triplet prediction [2503.23130].

Taken together, these adaptations show that CholecT50 has evolved from a triplet-recognition dataset into a general-purpose semantic substrate for laparoscopic cholecystectomy representation learning.

## 6. Limitations, critiques, and continuing role

Several limitations recur across the literature. First, the original dataset is **weakly supervised spatially**. The original paper states that CholecT50 provides frame-level presence labels rather than full spatial interaction annotation and does not distinguish multiple instances of the same triplet class within one frame [2109.03223]. This limitation is precisely what made the shift from recognition to detection so difficult.

Second, CholecT50 is strongly **long-tailed**. The CholecTriplet2021 analysis shows that rare instruments and targets perform poorly, that target recognition is the main bottleneck, and that some verbs, such as irrigate, are especially hard because they depend on temporal fluid dynamics [2204.04746]. The standardized benchmarking paper argues that this imbalance makes single train/test splits unstable and motivates the official 5-fold cross-validation protocol [2204.05235].

Third, later work has criticized the dataset’s annotation granularity and provenance. **ProstaTD** characterizes CholecT50 as having **“three critical limitations”**: it provides only class labels without spatial localization, lacks clear standardized temporal boundaries for actions, and originates from a **single institution**, which in that paper’s view restricts generalizability [2506.01130]. These claims are part of a comparative argument for a new robot-assisted prostatectomy dataset rather than a correction to the original CholecT50 papers, but they have become an important part of how CholecT50 is discussed in the broader triplet-detection literature.

Fourth, CholecT50 remains **procedure-specific**. It is confined to laparoscopic cholecystectomy. This specificity is valuable for controlled benchmarking, but later comparative work uses it as an example of a dataset where adaptation can work better precisely because it has **lower tool diversity** and a **more uniform video-level distribution** than more heterogeneous surgical corpora [2603.27341]. In that study, CholecT50 serves as a public external benchmark for instrument detection and a robustness check showing that fine-tuning can perform strongly even when zero-shot surgical perception remains unreliable [2603.27341].

Despite these constraints, CholecT50 continues to function as a reference point for the field. It underpins the public **CholecT45** subset, the **CholecTriplet2021** and **CholecTriplet2022** challenges, standardized **ivtmetrics** evaluation, and a large fraction of subsequent work on fine-grained surgical action understanding [2204.05235] [2204.04746]. It also remains the dataset against which new ideas in diffusion modeling, video-native pretraining, prompt-based multi-task learning, zero-shot transfer, vision-language reasoning, and surgical planning are still tested. This suggests that CholecT50’s enduring role is not only historical. It persists because its triplet formalization provides a compact but expressive interface between low-level surgical video and higher-level semantic reasoning.

Source: https://www.emergentmind.com/topics/cholect50