---
title: 'EndoVis Challenge: Surgical Vision Benchmarks'
url: https://www.emergentmind.com/topics/endoscopic-vision-endovis-challenge
type: topic
---

# EndoVis Challenge: Surgical Vision Benchmarks

The Endoscopic Vision (EndoVis) Challenge is a multi-track, community-driven benchmarking initiative committed to advancing computer vision methodologies—particularly in the context of robot-assisted and minimally invasive surgery. Conceived to catalyze progress in surgical scene understanding, tool localization, workflow analysis, and robust perception, EndoVis now encompasses a spectrum of surgical tasks: instrument segmentation, depth estimation, domain adaptation, activity modeling, phase recognition, and 3D reconstruction, all on rigorously annotated multicenter datasets. EndoVis benchmarks are foundational for reproducible algorithmic assessment and for the development of reliable, clinically translatable AI assistance in endoscopy.

## 1. Historical Evolution and Motivation

The EndoVis Challenge originated at MICCAI in 2015 with a focus on ex vivo tissue; subsequent years saw the scope expand to realistic da Vinci instrument segmentation (2017), scene segmentation (2018), and dense depth estimation with structured light (SCARED, 2019) [2001.11190]. A recurring theme is the use of well-calibrated datasets annotated by a combination of robot kinematics, instrument CAD meshes, and surgical expert correction. Progressively, EndoVis diversified to include semantic segmentation of anatomy and instruments (CaDIS, cataract surgery, 2020) [2110.10965], workflow and phase recognition (PitVis, pituitary surgery, 2023) [2409.01184], action triplet modeling (CholecTriplet, 2021–22) [2302.06294], and context-aware benchmarks integrating surgical phase, keypoint, and instance segmentation (PhaKIR, cholecystectomy, 2024) [2507.16559]. This structured expansion reflects the community’s recognition that robust surgical AI must solve multi-faceted, temporally consistent, and contextually rich tasks, not isolated vision problems.

## 2. Core Tasks and Dataset Design

EndoVis tasks span the following principal categories:

- **Instrument Segmentation:** Binary or multi-class pixel-wise identification of instruments, complemented by anatomical and device classes in later years. Datasets include the da Vinci Xi robotic system (2017/2018) [2001.11190], Sinus-Surgery live/cadaver videos [2002.10675], and multicenter cholecystectomy sets [2507.16559]. Annotation combines kinematics-based automatic masks (earlier years), manual polygon labeling (anatomy, devices), and more recently, multi-class, multi-instance maps (instance segmentation).
- **Depth Estimation:** SCARED (2019) [2101.01133] provides rectified stereo pairs and structured-light ground-truth point clouds for porcine subjects, with dense metric accuracy computed in millimeters. EndoDepth (2024) extends this with systematic robustness assessment under 16 corruption types [2409.19930].
- **Workflow, Phase, and Action Triplet Recognition:** Challenges such as PitVis (pituitary workflow) [2409.01184] and CholecTriplet (laparoscopic action triplets) [2302.06294] leverage longitudinal video annotation, including per-second surgical step labels, tool usage, and fine-grained action ⟨Instrument, Verb, Target⟩ triplet detection.
- **Keypoint and Instance Localization:** The PhaKIR sub-challenge [2507.16559] defines simultaneous frame-wise phase classification, instrument keypoint estimation (variable keypoints and visibilities per instance), and multi-class instance segmentation on annotated human cholecystectomy videos.
- **Domain Adaptation and Robustness:** SurgVisDom (2020) [2102.13644] explores transfer from synthetic VR simulation clips to clinical-like porcine videos, with explicit domain adaptation tasks. SegSTRONG-C (2024) probes segmentation robustness to non-adversarial corruptions (bleeding, smoke, low illumination) [2407.11906].

Each dataset is released with carefully defined splits, annotation standards, and, in the case of newer tracks, bootstrapped confidence intervals and ranking stability plots.

## 3. Methodological Advances and Evaluation Protocols

EndoVis has served as the proving ground for a range of architectures:

- **Encoder-Decoder CNNs and Transformers:** Early successes (U-Net, DeepLab) [2001.11190, 2110.10965] have given way to transformer-based methods for both spatial and temporal encoding, e.g., Mask2Former (instance segmentation) [2507.16559], Swin-based spatio-temporal backbones (workflow) [2409.01184], and vision transformers for robust matching (EndoMatcher) [2508.05205].
- **Augmentation and Robustness:** Techniques include elastic/projective transforms, AutoAugment policies [2407.11906], simulation of surgical artifacts, and exposure/color shift modeling (EndoDepth) [2409.19930].
- **Weak Supervision and Multi-round Bootstrapping:** Instrument localization under weak supervision via video-level tool presence events (WS-YOLO) [2309.13404] and weakly-supervised action triplet detection using CAMs, multi-task heads, and pre-trained detectors (CholecTriplet2022) [2302.06294].
- **Multi-task and Context-driven Modeling:** Simultaneous modeling of phase, instrument localization, and activity context (PhaKIR [2507.16559], PitVis [2409.01184], CholecTriplet2022 [2302.06294]), cross-task feature sharing, and joint post-processing algorithms.
- **Temporal Consistency Enforcement:** EMA-based smoothing, harmonic filtering, and transformer sequence decoders are key in workflow and activity recognition [2409.01184, 2507.16559].

Metrics are chosen to match task goals: Intersection over Union (IoU), Dice Similarity Coefficient (DSC), Mean Average Precision (mAP at varying IoU thresholds), Hausdorff Distance for mask boundaries, Object Keypoint Similarity (OKS), surgical phase Balanced Accuracy, and, for robustness, composite mDERS (mean Depth Estimation Robustness Score) [2409.19930].

## 4. Landmark Results, Benchmarks, and Limitations

EndoVis challenges consistently report state-of-the-art performance figures and analyze failure modes:

- **Segmentation:** Top binary segmentation on da Vinci Xi (2017) exceeds 90% DSC [2002.10675]; multi-class scene segmentation sees a drop to ≈62% mIoU as anatomical classes and devices are introduced [2001.11190].
- **Semantic Segmentation (CaDIS):** Anatomy-only segmentation achieves mIoU 0.94; instrument-only 0.53; combined, 0.56 [2110.10965]. Explicitly modeled boundaries improve performance on thin, complex anatomy.
- **Depth Estimation (SCARED/EndoDepth):** Modern pipelines approach 2.95–3.60 mm MAE [2101.01133]; under corruption, robustness varies dramatically with mDERS ranging 0.23–0.31 for leading monocular methods [2409.19930].
- **Workflow Recognition (PitVis):** Spatio-temporal multi-task models raise step Macro-F1 from <15% to >60%, instrument Macro-F1 from ~34% to ~42% [2409.01184].
- **Multi-task Contextual Analysis (PhaKIR):** Transformer-powered instance segmentation achieves DSC 35.5%, mAP 36.05% on challenging cholecystectomy data; phase recognition F1 69.1%, Balanced Accuracy 84.2% [2507.16559].
- **Action Triplet Detection (CholecTriplet2022):** Top triplet detection mAP ≈35% and joint detection+localization mAP 4.5%; weak supervision via CAMs remains limited for high-precision box detection [2302.06294].
- **Robustness to Corruption (SegSTRONG-C):** U-Net + AutoAugment achieves DSC/NSD ≈0.79/0.67 under smoke, bleeding, low brightness [2407.11906]; domain-specific drops highlight the need for truly representative corruptions.

Limitations and failure cases are recurrent themes: class imbalance, anatomy/instrument boundary ambiguity, poor generalization to new centers/intuitions, lack of temporal modeling in instance segmentation, and susceptibility of DNNs to subtle scene corruptions.

## 5. Current Directions: Robustness, Generalization, and Real-time Constraints

Recent EndoVis tracks (EndoDepth, SegSTRONG-C, EndoMatcher) explicitly target universal robustness—how models degrade under plausible intraoperative artifacts, domain shifts, and multi-institutional data [2409.19930, 2407.11906, 2508.05205]. Recommendations include:

- **Training on Corrupted and Synthetic Data:** Systematic corruption simulation, photorealistic artifact rendering, and domain-randomized synthetic data are key for closing the lab-to-clinic gap.
- **Multi-domain and Multi-objective Pre-training:** EndoMatcher leverages a massive multi-domain dataset (Endo-Mix6, ~1.2M pairs across six domains) and a progressive multi-objective training scheme to balance feature representation and avoid negative transfer.
- **Real-time Pose, Tracking, and Reconstruction:** Efficient architectures (47 FPS/1K matches for EndoMatcher [2508.05205], 86 FPS for EndoWave [2510.23087]) enable deployment in intraoperative assistance loops.
- **Explicit Temporal Modeling:** Future EndoVis benchmarks advocate for joint spatial-temporal modeling in segmentation, keypoints, and workflow—potentially integrating robot kinematics, stereo signals, and multi-modal cues.

Across tasks, there is consensus that robust, adaptive, context-driven and temporally aware networks will underpin next-generation surgical AI.

## 6. Impact, Standards, and Community Practices

EndoVis is not only a technical contest but serves as the field standard for dataset design, annotation practice, unbiased benchmarking, and transparent algorithmic reporting. Adherence to guidelines such as Metrics Reloaded and BIAS [2507.16559] ensures replicable, fair, and robust comparative analyses. All major datasets (e.g., CholecT50, EndoNeRF, SCARED, PitVis, PhaKIR) are released under open licensing protocols with documented annotation, ethical approval, and, increasingly, multi-institutional provenance.

EndoVis Tools and Evaluation Protocols  
─────────────────────────────────────
Component                                        | Description                                      | Notable Implementation
--------------------------------------------------|--------------------------------------------------|------------------------
Dataset standardization                           | Multi-center, multi-class annotation, bootstrapping| PhaKIR [2507.16559], SCARED [2101.01133]
Metrics protocols                                | Composite scores (DSC, mAP, mDERS, OKS, BA, HD95) | EndoDepth [2409.19930], PhaKIR [2507.16559]
Open-source code, docker templates               | Facilitate community engagement, reproducibility   | CholecTriplet2022 [2302.06294]
Multi-track, temporally linked benchmarking      | Enables cross-task, downstream integration        | PitVis [2409.01184], SegSTRONG-C [2407.11906]

The EndoVis Challenge defines and catalyzes the technical frontier for scene understanding in surgical vision, encouraging convergence on standardized datasets, objective evaluation, robust multi-task algorithms, and clinically meaningful, reproducible results. 

## 7. Future Directions and Open Problems

Several avenues merit further exploration:

- **Temporal, Spatio-Contextual Integration:** Unified spatial-temporal and cross-task architectures (e.g., tool tracking and workflow with mutual priors).
- **Universal Robustness:** Simulation-in-the-loop, uncertainty-driven correction, and adaptive normalization for all major artifact types.
- **Multi-modal Fusion:** Incorporating kinematic signals, stereo/depth inputs, and audio for comprehensive real-time guidance.
- **Human-centric Annotation and Usage:** Expansion to human datasets, risk structure maps, explainable assistance, and federated multicenter data acquisition.
- **Benchmark Expansion and Standards:** Regular updates to datasets, inclusion of new surgical procedures, richer annotation targets (scene graphs, biophysical signals).

EndoVis remains the definitive community reference for benchmarking, algorithmic innovation, and deployment standards in computer vision for endoscopy, catalyzing progress from lab to operating theater.

Source: https://www.emergentmind.com/topics/endoscopic-vision-endovis-challenge