CholecT50: Fine-Grained Laparoscopic Triplet Dataset

• CholecT50 is a fine-grained laparoscopic cholecystectomy dataset that formalizes surgical actions as instrument-verb-target triplets.
• The dataset comprises 50 videos annotated at 1 frame per second with clinically curated multi-label triplets, facilitating precise tool–tissue interaction analysis.
• It serves as a canonical benchmark for research in triplet recognition, detection, multi-task learning, and vision-language evaluation in surgical workflows.

CholecT50 is a fine-grained laparoscopic cholecystectomy video dataset in which surgical activity is formalized as frame-level, multi-label triplets of the form $$\langle \text{instrument}, \text{verb}, \text{target} \rangle$$. It was introduced to move surgical workflow analysis beyond coarse labels such as phases or steps and toward direct modeling of tool–tissue interactions, and later work characterizes it as the most widely adopted dataset for surgical triplet research (Nwoye et al., 2021, Chen et al., 1 Jun 2025). Within the surgical computer-vision literature, CholecT50 has served both as a canonical benchmark for triplet recognition and as a substrate for challenge protocols, detection extensions, multi-task learning, vision-language evaluation, text-to-image generation, question answering, and action planning (Nwoye et al., 2022).

1. Origins, scope, and semantic ontology

CholecT50 was introduced alongside the Rendezvous model as an extension and refinement of the earlier CholecT40 dataset (Nwoye et al., 2021). It contains 50 videos of laparoscopic cholecystectomy, with the original paper describing the corpus as comprising 45 videos from Cholec80 and 5 videos from an in-house dataset, all from the same procedure type (Nwoye et al., 2021). The later benchmarking paper further clarifies that CholecT45 is the first public release of CholecT50 and contains the 45 of the 50 videos that were publicly available, while the remaining five videos corresponded to the held-out challenge test set (Nwoye et al., 2022).

The dataset’s central abstraction is the surgical action triplet,

$$\langle \text{instrument}, \text{verb}, \text{target} \rangle,$$

intended to capture the acting tool, the action being performed, and the anatomical or operative target (Nwoye et al., 2021). The component vocabularies reported for CholecT50 are 6 instrument classes, 10 verb classes, and 15 target classes, combined into 100 valid triplet classes (Nwoye et al., 2021, Nwoye et al., 2022). The benchmarking paper also reports 7 phase classes in the dataset statistics, reflecting the dataset’s overlap with broader laparoscopic cholecystectomy annotation ecosystems (Nwoye et al., 2022).

The triplet vocabulary was not obtained by taking the full combinatorial product of instruments, verbs, and targets. The original paper states that, although the raw component spaces would permit $6\times10\times15=900$ combinations and roughly 300 combinations were observed in the raw annotations, the final 100 triplet classes were clinically curated in two stages: semantically equivalent labels were grouped into super-classes, and candidate triplets were ranked by clinicians according to plausibility and utility (Nwoye et al., 2021). This curation makes CholecT50 more than a naïve label product: it is a clinically filtered ontology of laparoscopic tool–tissue interactions.

The dataset was explicitly motivated by the limitations of coarser workflow tasks. In the original framing, triplet recognition is the only workflow-analysis formulation that aims at “truly fine-grained and comprehensive information” about surgical activity (Nwoye et al., 2021). That claim explains why CholecT50 became a reference point not only for triplet recognition, but also for later work seeking semantically richer surgical representations.

2. Annotation regime and reported corpus statistics

CholecT50 is annotated at 1 frame per second, and its labels are frame-level and multi-label: a frame may contain multiple simultaneous action triplets (Nwoye et al., 2021, Nwoye et al., 2022). The original paper describes annotation as being performed by two surgeons using Surgery Workflow Toolbox-Annotate, with action intervals marked on the timeline and labeled by instrument, verb, and target (Nwoye et al., 2021). An action ended when the instrument left the frame or when either the verb or the target changed; out-of-frame actions were not annotated, and frames recorded outside the patient’s body were zeroed out (Nwoye et al., 2021).

A crucial characteristic of the original release is that it is a frame-level weak-label dataset rather than a densely localized interaction dataset. Each frame has binary presence labels for one or more triplet classes, but the original paper notes that CholecT50 does not distinguish multiple instances of the same triplet class within one frame (Nwoye et al., 2021). This weakly supervised spatial regime later became a major source of methodological difficulty for triplet detection work.

Reported corpus totals differ across papers.

Source	Reported scale	Notes
Rendezvous (Nwoye et al., 2021)	50 videos, 100,863 annotated frames, 161,005 triplet instances	Extension of CholecT40
Benchmarking protocol (Nwoye et al., 2022)	50 videos, 100.9K frames, 151.0K triplet instances, 13.0K bounding boxes	Also reports 7 phase classes
CholecTriplet2021 (Nwoye et al., 2022)	50 videos, 100.9K frames, 161K triplet instance labels	Challenge-scale summary

These discrepancies are part of the published record rather than a single reconciled canonical count. A plausible implication is that different papers are summarizing slightly different annotation exports or benchmark views of the same underlying corpus.

Beyond full triplets, the challenge literature emphasizes that CholecT50 also exposes component-wise labels for instruments, verbs, and targets, enabling component and association evaluation in addition to full-triplet scoring (Nwoye et al., 2022). At the same time, target recognition emerges repeatedly as the hardest component. The CholecTriplet2021 analysis reports that target recognition had the lowest overall performance, and identifies cystic artery, blood vessels, peritoneum, omentum, and gut as especially difficult targets (Nwoye et al., 2022). This establishes one of the dataset’s enduring empirical signatures: instruments are comparatively easy, verbs intermediate, and targets the main bottleneck.

3. Benchmark protocols and evaluation methodology

A substantial part of CholecT50’s significance comes from the effort to standardize how it is used. The paper “Data Splits and Metrics for Method Benchmarking on Surgical Action Triplet Datasets” defines three important split families for CholecT50 and recommends 5-fold cross-validation as the primary research protocol (Nwoye et al., 2022).

Protocol	Structure	Purpose
RDV split	35 train / 5 val / 10 test videos	Reproducibility with Rendezvous
CholecTriplet challenge split	45 public trainval videos / 5 hidden test videos	Challenge evaluation
Official 5-fold CV	5 folds over all 50 videos	Recommended research benchmark

The RDV split preserves the train/validation/test setup used in the original Rendezvous paper (Nwoye et al., 2021). The challenge split underlies the CholecTriplet2021 and CholecTriplet2022 competitions, where 45 public videos form the development set and 5 hidden videos are reserved for official testing (Nwoye et al., 2022, Nwoye et al., 2023). The official 5-fold cross-validation split was introduced to reduce bias from class imbalance and missing classes in single held-out test sets; the folds were constructed by first sorting videos by procedure duration as a proxy for difficulty and then distributing videos across folds so that difficulty was more balanced (Nwoye et al., 2022).

Evaluation is centered on Average Precision (AP). The benchmark papers define component metrics

$AP_I,\; AP_V,\; AP_T,$

association metrics

$AP_{IV},\; AP_{IT},$

and full-triplet recognition

$AP_{IVT},$

with $AP_{IVT}$ treated as the primary metric (Nwoye et al., 2022, Nwoye et al., 2022). Recognition AP is computed as the area under the precision–recall curve, and the standardized tooling computes both video-level and global variants, with the benchmark tables using video-level aggregation (Nwoye et al., 2022).

For methods that output only triplet scores, component predictions are derived by filtering the triplet outputs: the score of an instrument, verb, or target is obtained by taking the maximum triplet score over all triplets containing that component (Nwoye et al., 2021, Nwoye et al., 2022). This disentangling procedure became formalized in the released ivtmetrics library, which supports recognition, detection, and component extraction for CholecT50-style evaluation (Nwoye et al., 2022).

Challenge-style reporting introduces one further convention: only 94 valid triplet classes are scored, because 6 null triplet classes are excluded from ranking (Nwoye et al., 2022, Nwoye et al., 2022). This matters when comparing numbers across papers, since some works report over the nominal 100-class space and others over the challenge-effective 94-class space.

The 2022 detection challenge extends this protocol by adding spatial evaluation. It uses 45 full training videos, 5 short validation clips, and 5 full hidden test videos, with instrument-tip bounding boxes provided for validation and hidden test evaluation but not for the full training split (Nwoye et al., 2023). That design operationalizes triplet detection as weakly supervised localization and association rather than fully supervised detection training.

4. Recognition and detection performance landscape

CholecT50 has been used to chart the progression from early triplet recognition baselines to more structured association models, pretraining-based approaches, and detection systems. The CholecTriplet2021 challenge paper remains the clearest benchmark snapshot of the original recognition era: across 24 evaluated models, reported $AP_{IVT}$ ranged from 4.2% to 38.1%, with a mean triplet performance of $23.3 \pm 9.9\%$ (Nwoye et al., 2022). The official winner, Trequartista, reached 38.1% $AP_{IVT}$, while the strongest organizer baseline, Rendezvous, achieved 32.7%; an ensemble of top methods later pushed performance to 42.4% $$\langle \text{instrument}, \text{verb}, \text{target} \rangle,$$0 (Nwoye et al., 2022).

Under the later standardized official 5-fold cross-validation protocol, reproduced baselines are lower but more comparable across studies. The benchmarking paper reports Tripnet at $$\langle \text{instrument}, \text{verb}, \text{target} \rangle,$$1 $$\langle \text{instrument}, \text{verb}, \text{target} \rangle,$$2, Attention Tripnet at $$\langle \text{instrument}, \text{verb}, \text{target} \rangle,$$3, and Rendezvous at $$\langle \text{instrument}, \text{verb}, \text{target} \rangle,$$4, with Rendezvous also leading on $$\langle \text{instrument}, \text{verb}, \text{target} \rangle,$$5 and $$\langle \text{instrument}, \text{verb}, \text{target} \rangle,$$6 (Nwoye et al., 2022). This standardized CV view is important because it reduces the optimism of favorable single-split results and exposes nontrivial fold variance.

Later work improved substantially on this baseline regime. DiffTriplet recast triplet recognition as conditional denoising in a joint space of triplets and components, and on CholecT50 cross-validation reported $$\langle \text{instrument}, \text{verb}, \text{target} \rangle,$$7 $$\langle \text{instrument}, \text{verb}, \text{target} \rangle,$$8 using Rendezvous features and $$\langle \text{instrument}, \text{verb}, \text{target} \rangle,$$9 using SDSwin features (Liu et al., 2024). Its ablations show that joint-space learning and association guidance both contribute measurable gains over a diffusion model operating only in triplet space (Liu et al., 2024). UniSurg, a V-JEPA-based video-native foundation model evaluated with attentive probing on a frozen backbone, reported AP-I 91.55, AP-V 57.72, AP-T 48.18, AP-IV 40.39, AP-IT 43.47, and AP-IVT 39.54, leading all six metrics in its comparison table (Wu et al., 5 Feb 2026). SurgFM, pretrained on Surg-3M, reported $6\times10\times15=900$0 mAP for CholecT50 action recognition in a frame-wise protocol and outperformed Tripnet, Rendezvous, and Attention Tripnet in that setting (Che et al., 25 Mar 2025). On a separate fixed 40/5/5 CholecT50 split, MEJO-B is reported to outperform CoT by 1.6% in $6\times10\times15=900$1 and to achieve the best results across $6\times10\times15=900$2 (Zhang et al., 16 Sep 2025).

Detection remained much harder. The CholecTriplet2022 challenge extended the problem from recognition to triplet detection, requiring instrument-tip localization plus triplet assignment (Nwoye et al., 2023). In that leaderboard, the weakly supervised RDV-Det baseline achieved only 0.24 $6\times10\times15=900$3, while the strongest challenge method, ResNet-CAM-YOLOv5, reached 4.49 $6\times10\times15=900$4 (Nwoye et al., 2023). The follow-up MCIT-IG method, which combines instrument-aware target embeddings with a bipartite interaction graph, reported 60.1 $6\times10\times15=900$5, 66.6 $6\times10\times15=900$6, 7.32 $6\times10\times15=900$7, and 10.26 $6\times10\times15=900$8 on the private challenge test set, and the authors state that it would have ranked first on the CholecTriplet2022 leaderboard (Sharma et al., 2023). This gap between recognition and detection is a persistent characteristic of CholecT50: recognizing that a triplet is present is materially easier than localizing and associating the operative actor under weak supervision.

5. Derivative resources and repurposed task formulations

CholecT50 has generated a large family of derivative resources and alternative task formulations. Some extend the dataset itself; others repackage its labels into new modalities or learning problems.

Resource or protocol	Transformation of CholecT50	Resulting task
CholecT45 (Nwoye et al., 2022)	Public 45-video subset of CholecT50	Public benchmarking
CholecInstanceSeg (Alabi et al., 2024)	New instance masks and IDs on CholecT50/Cholec80 frames	Tool instance segmentation
CholecT50-SAP (Xu et al., 24 Mar 2025)	Consecutive same-action frames grouped into clips; 5-action label space	Surgical action planning
CholecT50-Caption-VQA (Li et al., 31 Mar 2026)	Action-consistent clips with clinician captions converted to QA	Video question answering
Surgical Imagen training set (Nwoye et al., 2024)	Triplets linearized into text prompts	Surgical text-to-image generation

CholecInstanceSeg is the most direct annotation extension. It reuses CholecT50 frames and adds semantic masks and instance IDs for tool instance segmentation. The CholecT50-derived portion comprises 15 fully annotated sequences with 28,317 frames and 45,221 tool instances, plus 35 sparsely annotated sequences with 2,681 frames and 4,098 tool instances (Alabi et al., 2024). This directly addresses a capability the original CholecT50 did not provide: dense instance-level tool segmentation.

CholecT50-SAP repurposes the corpus for forward-looking decision support rather than retrospective recognition. It is constructed by grouping consecutive frames sharing the same action into clips, restricting attention to segments around Calot’s triangle dissection, duct and vessel clipping, and dissection from the liver bed, and collapsing the label space to

$6\times10\times15=900$9

The resulting dataset contains 225 samples, split into 35 training videos / 168 samples and 15 testing videos / 57 samples (Xu et al., 24 Mar 2025). Here, CholecT50 becomes a substrate for next-action prediction and prompt-based planning with LLMs.

CholecT50-Caption-VQA, one component of CholeVidQA-32K, segments CholecT50 videos by action continuity and then uses clinician-authored captions plus GPT-5.1 to generate question–answer pairs. This subset contains 1,544 video segments and 18,368 QA pairs, covering Tool Perception, Action Perception, Anatomical Structure Perception, Surgical Scene Description, and Action Rationale Reasoning (Li et al., 31 Mar 2026). In this formulation, CholecT50 no longer functions as a triplet classifier benchmark but as a foundation for temporally grounded surgical VQA.

Other papers repurpose CholecT50 labels as language prompts. Surgical Imagen uses approximately 101K image-text pairs from CholecT50, with triplet-based prompts such as short linearized captions, to train a diffusion-based text-to-image generator (Nwoye et al., 2024). MML-SurgAdapt integrates CholecT50’s 100 triplet labels with 7 phase labels from Cholec80 and 3 CVS labels from Endoscapes2023 into a shared 110-label CLIP-based multi-task space, using Single Positive Multi-Label (SPML) learning to cope with missing annotations (Walimbe et al., 7 Jul 2025). fine-CLIP introduces a CholecT50-specific base-to-novel benchmark with Unseen-Target and Unseen-Instrument-Verb settings, using held-out triplet subsets to test compositional zero-shot transfer (Sharma et al., 25 Mar 2025). DeepSeek-based evaluation uses CholecT50 only for Single Phrase QA, reformulating images into closed-set questions over 5 instrument, 8 target, and 13 motion phrases rather than evaluating standard triplet prediction (Ma et al., 29 Mar 2025).

Taken together, these adaptations show that CholecT50 has evolved from a triplet-recognition dataset into a general-purpose semantic substrate for laparoscopic cholecystectomy representation learning.

6. Limitations, critiques, and continuing role

Several limitations recur across the literature. First, the original dataset is weakly supervised spatially. The original paper states that CholecT50 provides frame-level presence labels rather than full spatial interaction annotation and does not distinguish multiple instances of the same triplet class within one frame (Nwoye et al., 2021). This limitation is precisely what made the shift from recognition to detection so difficult.

Second, CholecT50 is strongly long-tailed. The CholecTriplet2021 analysis shows that rare instruments and targets perform poorly, that target recognition is the main bottleneck, and that some verbs, such as irrigate, are especially hard because they depend on temporal fluid dynamics (Nwoye et al., 2022). The standardized benchmarking paper argues that this imbalance makes single train/test splits unstable and motivates the official 5-fold cross-validation protocol (Nwoye et al., 2022).

Third, later work has criticized the dataset’s annotation granularity and provenance. ProstaTD characterizes CholecT50 as having “three critical limitations”: it provides only class labels without spatial localization, lacks clear standardized temporal boundaries for actions, and originates from a single institution, which in that paper’s view restricts generalizability (Chen et al., 1 Jun 2025). These claims are part of a comparative argument for a new robot-assisted prostatectomy dataset rather than a correction to the original CholecT50 papers, but they have become an important part of how CholecT50 is discussed in the broader triplet-detection literature.

Fourth, CholecT50 remains procedure-specific. It is confined to laparoscopic cholecystectomy. This specificity is valuable for controlled benchmarking, but later comparative work uses it as an example of a dataset where adaptation can work better precisely because it has lower tool diversity and a more uniform video-level distribution than more heterogeneous surgical corpora (Skobelev et al., 28 Mar 2026). In that study, CholecT50 serves as a public external benchmark for instrument detection and a robustness check showing that fine-tuning can perform strongly even when zero-shot surgical perception remains unreliable (Skobelev et al., 28 Mar 2026).

Despite these constraints, CholecT50 continues to function as a reference point for the field. It underpins the public CholecT45 subset, the CholecTriplet2021 and CholecTriplet2022 challenges, standardized ivtmetrics evaluation, and a large fraction of subsequent work on fine-grained surgical action understanding (Nwoye et al., 2022, Nwoye et al., 2022). It also remains the dataset against which new ideas in diffusion modeling, video-native pretraining, prompt-based multi-task learning, zero-shot transfer, vision-language reasoning, and surgical planning are still tested. This suggests that CholecT50’s enduring role is not only historical. It persists because its triplet formalization provides a compact but expressive interface between low-level surgical video and higher-level semantic reasoning.