Surgical Workflow Analysis Overview

Updated 25 June 2026

Surgical workflow analysis is a computational approach that segments surgical phases, actions, and instrument interactions using multi-modal data and deep learning.
It employs spatial feature extraction and temporal aggregation techniques to accurately recognize phases, track instruments, and assess surgical skill.
Advanced models integrate unsupervised pretraining, few-shot adaptation, and hierarchical reasoning to enable real-time decision support and enhanced clinical safety.

Surgical workflow analysis is the computational study of temporal structure and functional organization in surgical procedures, aiming to automatically recognize, segment, and interpret a sequence of surgical activities, phases, and interactions in the operating room (OR) or within endoscopic video. Modern workflow analysis leverages multi-modal video, sensor, and instrument data to enable intraoperative decision support, skill assessment, safety monitoring, and future autonomous surgical systems.

1. Problem Formulation and Annotation Standards

Surgical workflow analysis encompasses sequence modeling for a hierarchy of tasks, including:

Phase/step recognition: Assigning each video frame or clip a label from a predefined taxonomy (e.g., “dissection,” “closure”).
Action/triplet recognition: Detecting fine-grained events as (instrument, verb, target) triplets, such as “clip (verb) cystic artery (target) with a clipper (instrument)” (Jeon et al., 18 Jan 2026).
Tool/instrument tracking: Identifying the presence and identity of surgical instruments.
Skill assessment: Quantitative or qualitative rating of surgeon proficiency based on kinematics, workflow, and errors (Wagner et al., 2021).

Annotation protocols typically operate at multiple hierarchical levels:

Steps: High-level procedural segments.
Tasks: Mid-level objectives.
Activities/triplets: Atomic interactions (action + instrument + target) (Zhao et al., 2022, Jeon et al., 18 Jan 2026).

Datasets such as HeiChole (Wagner et al., 2021), LLS48 (Jeon et al., 18 Jan 2026), RLLS12M (Zhao et al., 2022), Cholec80 (Czempiel et al., 2022), and OphNet (Hu et al., 2024) provide varying granularity, multi-institutional diversity, and annotation richness.

State-of-the-art workflow analysis leverages deep convolutional and transformer-based models, often operating in two- or three-stage pipelines:

Spatial backbone: Frame- or clip-level feature extraction (ResNet, Swin-Transformer, I3D, EfficientNet).
Temporal aggregation: Sequence modeling using RNNs (LSTM, GRU), multi-stage temporal convolutional networks (MS-TCN), transformers, or multi-resolution temporal transformers (MRTT) (Jeon et al., 18 Jan 2026, Czempiel et al., 2022).
Multi-modal fusion: Integration of intensity images, depth maps, tool kinematics, and scene graphs yields robust representations. For example:
- Unsupervised multi-modal pretraining with intensity and depth as “views” tightly coupled via cross-view clustering loss (using Sinkhorn-regularized optimal transport) enhances generalizability in low-label regimes (Jamal et al., 2022).
- Vision-LLMs enable prompt-based and text-driven adaptation for both discriminative (classification) and generative (captioning) tasks, reducing the need for large annotated sets by few-shot alignment to expert descriptions (Chen et al., 16 Jan 2025, Kondo, 19 May 2025, Gastager et al., 14 Mar 2025).

Prototype-based and contrastive methods (e.g., ProtoFlow, CurConMix+) leverage clustering and curriculum-guided hard negative sampling to create interpretable latent spaces and boost few-shot/fine-grained performance (Holm et al., 16 Dec 2025, Jeon et al., 18 Jan 2026). Dynamic scene graphs and relational modules further support structured reasoning at the object and event level.

3. Temporal Modeling and Handling of Ambiguity

Temporal context is indispensable for resolving:

Local ambiguities: Short occlusions or transient events resolved by 3D CNNs and clip-level models (Czempiel et al., 2022).
Global ambiguities: Long-range phase similarity, requiring RNNs (LSTM/GRU), temporal convolutions, or state-space models to enforce procedure order and history dependence (Ban et al., 2020).

Augmenting standard temporal models with propagated sufficient statistics (e.g., HMM filtering, cumulative sum likelihoods, Gabor features) further improves short-phase and transition detection by exposing the model to what has or hasn’t occurred (Ban et al., 2020).

Novel dual-pathway architectures, such as DSTED, explicitly decouple temporal stabilization (e.g., Reliable Memory Propagation leveraging reliable short-term memory) from discriminative enhancement (e.g., Uncertainty-Aware Prototype Retrieval for ambiguous transitions). These modules reduce prediction jitter and recover hard-to-predict boundary frames, with a learned gating mechanism to adaptively combine the streams based on prediction confidence (Chen et al., 22 Dec 2025).

4. Annotation Efficiency and Data-Efficiency Techniques

Barriers to workflow analysis include the scarcity and cost of expert-annotated datasets. Addressing this, multiple strategies are prominent:

Self-supervised and unsupervised pretraining: Temporal coherence pretraining, cross-view clustering with multi-modal data, and scene-graph autoencoding allow representation learning from large collections of unlabeled surgical video (Funke et al., 2018, Jamal et al., 2022, Holm et al., 16 Dec 2025).
Active learning: Deep Bayesian networks with uncertainty-based acquisition functions (e.g., entropy, variation ratio, mutual information) enable targeted annotation, reducing required labeled fractions by 30–40% while achieving or surpassing random selection performance. Monte Carlo dropout provides Bayesian uncertainty estimates, crucial for instrument detection and phase segmentation tasks (Bodenstedt et al., 2018).
Few-shot adaptation and prompt-tuning: Text-driven adaptation aligns small sets of image-text anchors within foundation models, enabling efficient transfer to new workflow taxonomies and annotation-poor tasks. This method supports both discriminative and generative end-tasks, such as triplet recognition and surgical scene captioning (Chen et al., 16 Jan 2025).

5. Advanced Pipeline Designs: End-to-End, Hierarchical, and Anticipative Models

End-to-End Video Learning and Normalization

Batch normalization (BN) introduces sample dependencies problematic in long-sequence or online video analysis—leading to model collapse in small batch or sequential settings, and cheating via information leakage. BN-free architectures (GroupNorm, LayerNorm, ConvNeXt) support long-context training, outperforming two-stage or Transformer-based pipelines when combined with carry-hidden training and adequate data augmentation (Rivoir et al., 2022).

Hierarchical and Relational Reasoning

Frameworks such as MURPHY and CurConMix+ target the full semantic hierarchy of surgical workflow, from action triplet to step. MURPHY uses relational graph convolutions and hierarchical cross-attention to exploit intra- and inter-relational annotation structures, yielding gains in mean average precision for step, task, and triplet recognition (Zhao et al., 2022). CurConMix+ leverages curriculum-guided contrastive and mixup learning, with adaptive multi-resolution temporal fusion, achieving SOTA in action triplet recognition and strong performance transfers to coarser annotation levels (Jeon et al., 18 Jan 2026).

Long-term Anticipation and Generative Decoding

Conventional anticipatory models are limited to short-term, single-event forecasts. Generative approaches such as SWAG formulate phase sequence prediction at variable horizons (up to 30 min) as a generative sequence modeling task, producing dense, repeatable predictions using both auto-regressive and single-pass decoders. Embedding class transition priors and regression-to-classification (R2C) methods further enhance anticipation accuracy, supporting intraoperative planning and context-aware alerting (Boels et al., 2024).

Collaborative stochastic-deterministic learning (CoStoDet-DDPM) co-trains a denoising diffusion model and a deterministic recognizer on shared spatio-temporal features; the diffusion branch regularizes feature learning by modeling procedural uncertainty during training, while inference remains fast and deterministic (Yang et al., 13 Mar 2025).

Interpretability

Prototype-based clustering, as in ProtoFlow and DSTED’s UPR, enables alignment of latent representations to recurring clinical sub-techniques or high-uncertainty transition frames, providing interpretable reasoning paths and visualizable “reason codes” for decision support (Holm et al., 16 Dec 2025, Chen et al., 22 Dec 2025).

6. Benchmark Datasets, Evaluation, and Analysis

The reproducibility and generalizability of workflow analysis algorithms depend on comprehensive benchmarks:

Dataset	Procedure Coverage	Video Hours	Labels/Hierarchy	Use Cases
HeiChole	Cholecystectomy	22	Phases, actions, skill	Generalizability
LLS48	Liver Sectionectomy	2.6 (clips)	Step, task, triplet	Hierarchical/HAT
RLLS12M	Liver Sectionectomy	588	Step, task, triplet	Relational
Cholec80	Cholecystectomy	80 videos	7 phases	Internal/Laparosc
OphNet	Ophthalmology	204.8	66 surgeries, 102 phases, 150 ops	Large-scale, multi-level

Metrics are typically mean Average Precision (mAP), per-class F1, Jaccard Index, phase boundary/edit score, and skill assessment MAE (Wagner et al., 2021, Jeon et al., 18 Jan 2026, Boels et al., 2024). Comprehensive evaluation includes both generalization across centers and low-label regimes.

7. Challenges, Limitations, and Future Research

Ongoing challenges and research trajectories include:

Generalizability: Many algorithms show performance degradation when moving from single- to multi-center datasets due to inter-institutional variability in video, workflow, and patient demographics (Wagner et al., 2021). Domain adaptation and multi-center data aggregation remain essential.
Label efficiency: Annotation bottlenecks persist despite progress in active/few-shot learning. Automated and semi-automated labeling tools, and leveraging of open-source video repositories, are active areas of research (Chen et al., 16 Jan 2025, Gastager et al., 14 Mar 2025, Hu et al., 2024).
Hierarchical and multi-task modeling: Unified models that jointly capture actions, phases, steps, and skills, and propagate context across these semantic levels, are under development (Zhao et al., 2022, Jeon et al., 18 Jan 2026).
Interpretability and trust: Prototype-discovery, scene-graph embeddings, and explicit relational reasoning provide direction in making deep workflow analysis models more transparent and clinically actionable (Holm et al., 16 Dec 2025, Chen et al., 22 Dec 2025).
Real-time and anticipatory analytics: Integration with OR systems demands low-latency, robust inference, and anticipation of upcoming phases or needs, with proven reduction in false alarms and meaningful intraoperative warning rates (Boels et al., 2024, Yang et al., 13 Mar 2025).
Expansion to richer modalities: Integration of kinematics, audio, and hemodynamic data with video for synergistic workflow understanding is nascent (Jamal et al., 2022, Guo et al., 2024).
Clinical translation: Current research has demonstrated strong technical advancements, but systematic studies are needed to quantify patient and operational outcomes from real-world deployments.

In summary, surgical workflow analysis has rapidly evolved from phase-by-phase video recognition to robust, multi-level, and interpretable systems capable of anticipating and contextualizing complex clinical events. State-of-the-art research continues to drive data-efficiency, multi-modal grounding, anticipation, and human-centric interpretability, with clinical integration now on the horizon.